cmd/cue: add hidden command to dump command structure for cuelang.org

[myitcv]

For cuelang.org we rather clumsily/forgetfully maintain a list of commands that need to have pages auto-generated for their cue help documentation:

https://github.com/cue-lang/cuelang.org/blob/d94054d9cd41f12b7e506d1eb3cabc46e3191b4f/content/docs/reference/command/commands.cue#L115-L151

We should not require a manual update on the cuelang.org side here - the cuelang.org docs should always follow what cmd/cue does/says/supports.

Instead we should have a hidden command for cmd/cue which dumps a structure of some sort for all the commands it supports.

The cuelang.org infrastructure can then auto-generate from the output of this hidden command for the given CUE version.

[myitcv]

cc @jpluscplusm

[mvdan]

Fat fingered attempt at typing a comment, apologies.

I suggest a hidden command which outputs a JSON list of structs, one per command. For example, cue list-commands.

Which information is needed for each command? I suggest:

• Full command string, e.g. cue mod tidy
• Hidden boolean
• Experimental boolean (not all hidden commands are experimental, such as this very one!)
• Short help text string (as shown by e.g. cue help or cue help mod)
• Long help text string (so that cuelang.org does not have to execute cue help $cmd for each of them afterwards)

[mvdan]

Alternatively, we could group these for-internal-use-only hidden commands like we did with cue exp for experimental commands, so e.g. cue internal list-commands. I might prefer this, to make it extra obvious and reduce the chances that we might later conflict with other top-level command names we might want to use.

[myitcv]

Or, just to say it out loud, we actually start from a declarative data structure (which we could also use for cuelang.org) and code generate some Go off the back of that.

[mvdan]

Yes - we can and should declare this structure as CUE - but I wanted to summarise it in plain English before I got into the details. Anyway, here goes:

[...#Command]

#Command: {
    // args is the list of arguments to invoke this command,
    // such as ["cue", "mod", "tidy"].
    args!: [...string]

    // runnable describes if a command can be executed on its own
    // or if it exists only to contain subcommands, such as `cue mod`.
    runnable!: bool

    // experimental is true when a command is in an experimental stage,
    // meaning that it may be changed, renamed, or removed at any time,
    // such as `cue exp` and its subcommands.
    experimental?: bool

    // internal is true when a command exists solely for use by the CUE project,
    // such as `cue internal` and its subcommands.
    internal?: bool

    // shortDescription is a single line description of the command,
    // used by `cue help` when listing available commands or subcommands.
    shortDescription!: string

    // helpText is the multi-line output of `cue help foo` or `cue foo --help`,
    // which may include available flags and subcommands.
    helpText!: string
}

I chose to omit a "hidden" boolean because "experimental" and "internal" give the reason why a command is hidden in a much clearer way.

From my reading of https://cuelang.org/docs/reference/command/, the website would render all commands except those marked as internal. And when rendering the page, it would primarily use experimental to add a warning, and helpText for the content.

In the future we can add more fields if useful, of course, but for now this seems like a good start.

[jpluscplusm]

The cuelang.org reference pages (e.g. cue vet) also (currently) need to track the help command that's invoked, separately from the command being described. It's (currently) included in the page title and trailing URL path element.

We can (and currently do) string-process our way around maintaining this link, as seen here: https://github.com/cue-lang/cuelang.org/blob/d94054d9cd41f12b7e506d1eb3cabc46e3191b4f/content/docs/reference/command/commands.cue#L26-L38. If we're emitting a structure from cmd/cue that's primarily for cuelang.org's benefit, I think it would be useful to minimise the amount of munging needed on the consumer side.

If we choose not to do this, and require the cuelang.org build to insert a help command as args[1], then we'll also need to include a field identifying commands such as cue help filetypes so that they're not post-processed in this way.

---

Separately, we have a special case for the cue command by itself, because its command<>help-text<>URL-path translation is unique.

The current setup of the reference pages was written with consistency in mind (hence using a cue help prefix for every command, and not --help, which didn't translate into URL paths or page titles nicely) but cue help by itself was a corner cases which we couldn't avoid at the time. We can, of course, change any aspects of these pages to suit the generation process, but if we end up changing any page URL paths then we should maintain an old-path-to-new-path mapping that can drive HTTP redirects for a period. I'm not sure which side of the cmd/cue/cuelang.org process should maintain this map.

[jpluscplusm]

In the future we can add more fields if useful, of course, but for now this seems like a good start.

I share the desire for the cuelang.org consumer side to not need to maintain the list of cmd/cue sub-commands itself, but I should point out that IMHO the real value to the reader in automating this will be when cuelang.org can include cross-links so that each cue-help-... page isn't an isolated wall of text.

We started to do this, tracking some in-text links for the cue help mod ... hierarchy (e.g. links at the bottom of https://cuelang.org/docs/reference/command/cue-help-mod/), but we stopped doing this manually when the idea of getting cmd/cue to emit this data was proposed.

I'd like to ask that the first cut of this cmd/cue feature includes a schema facility for related commands to be communicated. I'll happily then spend the time to trawl through the help texts and populate the per-command metadata in cue-lang/cue!

---

There's also the question of cuelang.org's tags. You'll notice that pages like https://cuelang.org/docs/reference/command/cue-help-mod-init/ display a modules tag lozenge near the top and bottom. And whilst this on-page indicator&link is useful for readers, the more useful aspect is that the tag results in these pages showing up in the modules search index: https://cuelang.org/search/?q=tag:modules.

Given the very slow rate at which non-experimental commands will ever be removed from cue/cmd, we can easily continue to maintain a manual page-slug-to-tag map, on the cuelang.org side, without material risk of drift; but I felt it worth mentioning here for completeness.

[mvdan]

Would you be able to write a CUE schema like I tried above, along with what changes to cmd/cue you would suggest to e.g. track links to other help pages inside the texts?

This doesn't have to be perfect day one either. We need to start with something that will take a couple of hours to implement and integrate. We can then improve on top of that over time, e.g. categorizing the commands or turning the help text "rich" so that the website can render links.

[jpluscplusm]

Would you be able to write a CUE schema like I tried above, along with what changes to cmd/cue you would suggest to e.g. track links to other help pages inside the texts?

Modifying @mvdan's schema slightly, I propose this:

commands: [C=#HelpTextCommand]: #Command & {helpCommand: C}

#Command: {
        // args is the list of arguments to invoke this command,
        // such as ["cue", "mod", "tidy"].
        args!: [...string]

        // runnable describes if a command can be executed on its own
        // or if it exists only to contain subcommands, such as `cue mod`.
        runnable!: bool

        // experimental is true when a command is in an experimental stage,
        // meaning that it may be changed, renamed, or removed at any time,
        // such as `cue exp` and its subcommands.
        experimental?: bool

        // internal is true when a command exists solely for use by the CUE project,
        // such as `cue internal` and its subcommands.
        internal?: bool

        // shortDescription is a single line description of the command,
        // used by `cue help` when listing available commands or subcommands.
        shortDescription!: string

        // helpText is the multi-line output of `cue help foo` or `cue foo --help`,
        // which may include available flags and subcommands.
        helpText!: string
        // helpCommand is the shell command which, when invoked, emits helpText.
        helpCommand!: string & #HelpTextCommand

        // relatedCommands is a list of keys in the commands struct that
        // represent other commands' help texts which are mentioned inside
        // helpText. It defaults to an empty list.
        relatedCommands: [...or(#HelpTextCommands)]
}

#HelpTextCommand: =~"^cue help[ a-zA-Z0-9]*$"
#HelpTextCommands: [for command, _ in commands {command}]

The intent behind #HelpTextCommand is to impose some restrictions on the help-text-emitting commands (possibly loosened later) such that we can definitely perform bidirectional s/<SPACE>/-/g and s/-/<SPACE>/g/ transformations on the commands whilst building cuelang.org, and use Hugo to perform build-time page existence checks.

#HelpTextCommands deals with the reality that, once we introduce the concept of the "related command", each related command is a lookup into something. It feels better to force early failure inside cue-lang/cue rather than permitting a successful update made here to then fail, later, when the change reaches cuelang.org.

The commands struct is only present to provide a target for #HelpTextCommands' comprehension.

I've left runnable as-is, but I don't think we'll need to use it for cuelang.org. I think its existing comment is potentially ambigouous, as it calls out grouping commands but doesn't indicate what value this field would have for info commands such as cue help filetypes.