Misleading error log on 429 errors from registry

[arnoldyahad]

Hey,

Recently we have been throttled by hashicorp registry (As we confirmed by talking with their support)
But it was impossible to understand from our error logs (even when enabling debug)

we are running terragrunt plan commands on high scale and this is the error log that we got:

13:14:42.342 INFO Terragrunt Cache server is listening on 127.0.0.1:38691
14513:14:42.342 INFO Start Terragrunt Cache server
14613:14:42.752 INFO Downloading Terraform configurations from git::ssh://[[email protected]](mailto:[email protected])/<organization>/<repo>.git?ref=lambda_v1.1.37 into ./.terragrunt-cache/uCvmXUGu5jL2E_fX-K1egZAxmTs/bkwVI7uwl97tkFlbhBGmEdngoJk
14713:14:43.657 INFO Caching terraform providers for ./.terragrunt-cache/uCvmXUGu5jL2E_fX-K1egZAxmTs/bkwVI7uwl97tkFlbhBGmEdngoJk/lambda
14813:14:44.355 ERROR terraform invocation failed in ./.terragrunt-cache/uCvmXUGu5jL2E_fX-K1egZAxmTs/bkwVI7uwl97tkFlbhBGmEdngoJk/lambda
14913:14:44.355 INFO Shutting down Terragrunt Cache server...
15013:14:44.355 INFO Terragrunt Cache server stopped
15113:14:44.355 ERROR 3 errors occurred:
152
153* unable to cache provider: registry.terraform.io/hashicorp/external v2.3.4, err: not found provider download url
154
155* unable to cache provider: registry.terraform.io/hashicorp/local v2.5.2, err: not found provider download url
156
157* unable to cache provider: registry.terraform.io/hashicorp/null v3.2.3, err: not found provider download url

the error here has not information about rate-limiting or the status code of 429, and makes it much harder to recognize and troubleshoot

we saw this error message due to this code:
https://github.com/gruntwork-io/terragrunt/blob/main/tf/cache/services/provider_cache.go#L218-L236

it doesnt say anything about being 429 and without hashicorp support we wouldn't understand this.

Steps To Reproduce

Get 429 from hashicorp (as their support states - something around 3000 requests in 5 minutes is enough to get 429 error)

also, have terragrunt use their caching mechanism:

export TERRAGRUNT_PROVIDER_CACHE=1

0.69.9 and 0.71.1 - its the same error message.

Expected behavior

We want a clear message that says - 429 Rate-limiting just like it would say if we would use terraform.

Versions

• Terragrunt version: 0.69.9 / 0.71.1 but you can also use latest as i see code haven't changed.
• OpenTofu/Terraform version: 1.10.1
• Environment details (Ubuntu 20.04, Windows 10, etc.): Ubuntu 20.04

[yhakbar]

I definitely agree that the error message could be better.

Requesting community contributions, as you've already done some root cause analysis, and it shouldn't be too hard for someone from the community to submit a fix.