disco: OverrideHostDiscoveryURL

[apparentlymart]

The disco.Disco type already supports forcing a static service discovery result for a particular hostname using Disco.ForceHostServices, which provides one way to override the services on a particular host.

However, using that method requires hard-coding the services and thus uses of this for non-development purposes effectively subvert the main motivation of the service discovery system: to have the service information for a particular hostname encoded in a central location so that it can be updated everywhere without notifying all clients.

Disco.OverrideHostDiscoveryURL provides a slightly different take on that that makes a different compromise: instead of specifying a fixed service discovery result, a caller can instead tell the Disco object to use a different URL when fetching the service discovery information for a host over the network. This means that the overridden definition is still maintained in a central place that can be updated when needed, and in particular the server that the overridden URL refers to could potentially produce a service discovery document that's somehow derived from the upstream one, such as by overriding the location of just one service while retaining the others.

However, this should only be done with URLs that the caller trusts, because this effectively allows redirecting all requests for a particular hostname to different servers that are under someone else's control, and any credentials configured for that hostname would still be sent to the new service endpoints presented in the overridden document.

Description

My intention with this change is to introduce a new building-block that could in future enable a more satisfactory solution for requests like the following:

• Allow configuring default hostname for localterraform.com terraform#34808
• Support providers using localterraform.com terraform#33312
• Set the default registry URL terraform#34862
• Impossible to use Private Modules Registry if it uses url subpath terraform#33541

This PR only introduces the lowest-level component of such a change, but I'm imagining this as the backend for a new Terraform CLI Configuration setting, extending the existing host block type which allows reconfiguring how service discovery is performed on a particular hostname.

For example:

host "localterraform.com" {
  service_discovery_url = "https://app.terraform.io/.well-known/terraform.json"
}

The effect of the above would be to make the hostname localterraform.com appear to have all of the same services that app.terraform.io has, which seems relevant to the first two issues I linked above.

The "set the default registry URL" use-case is perhaps a little more dubious because provider network mirrors already offer a way to second-source providers that originated in someone else's registry from a local mirror operated by an organization. But network mirrors are currently supported only for providers, whereas this new setting would offer a more general solution that works for all Terraform-native services on a particular hostname.

I designed this as an override for the discovery document URL as a whole, rather than just aliasing one hostname to another, because that then potentially allows someone to run a single local server acting as aliases for many different hostnames that they want to use resources from, if for example they are running Terraform in an "air-gapped" environment that cannot access these services normally and this internal server they are running contains a snapshot of content that was available on the real remote services at some point:

host "registry.terraform.io" {
  service_discovery_url = "http://terraform-mirror.example.com/public-registry/services.json"
}

host "app.terraform.io" {
  service_discovery_url = "http://terraform-mirror.example.com/private-registry/services.json"
}

host "third-party-registry.example.net" {
  service_discovery_url = "http://terraform-mirror.example.com/example-net-registry/services.json"
}

This design therefore imposes as few constraints as possible, giving operators more room to make this work within whatever technical constraints their environment might impose on them.

Testing plan

I have added a unit test case for the new API as part of this commit.

Actually making use of this in the ways I described above would require changes to Terraform itself, of course.

External links

• Remote Service Discovery describes the protocol that this PR is effectively tweaking.

  Specifically, it modifies Discovery Process so that the documented process of forming the discovery document URL is now only for forming the default discovery document URL, with the system preferring to use the overridden discovery URL if any is configured.

• The localterraform.com domain is documented in Generic Hostname - HCP Terraform and Terraform Enterprise, and is currently documented only for module registry usage.

  Support providers using localterraform.com terraform#33312 requested allowing this also for provider registry usage, which this PR could be a building-block for. However, I'd note that Terraform would treat localterraform.com/example/foo as an entirely separate provider to app.terraform.io/example/foo (or whichever Terraform Enterprise hostname someone is using) under this model because the service URL change is at too low a level of abstraction for the provider-resolution logic to understand them as synonymous, and so this probably isn't a complete solution for that situation.

• In Document the "host" block type in CLI configuration terraform#28309 and CLI config "host" block not documented terraform#20453 there was some discussion about whether and where to document this host block type in the CLI configuration. The hesitation was the earlier-mentioned concern that anyone using this would effectively be hard-coding specific service URLs that the hostname owner expects to be able to change at any time. Perhaps the form of host with service_discovery_url would be more palatable to document because it at least keeps the sprawl of duplicating another host's service discovery information to only one additional location.