Add support for Workload Identity Federation in Stackdriver output plugin

[DeanBrunt]

Is your feature request related to a problem? Please describe.
We're currently deploying new infrastructure in AWS but want to be able to keep our logs backend and dashboard as Stackdriver/Google Cloud Logging.
This is possible with the Stackdriver exporter but requires explicit passing of the long lived private key which is not ideal.

Given that Workload Identity Federation exists between GCP and AWS (and other providers), it would be great to leverage this instead to avoid the need to manage long lived private keys for the cross-cloud authentication.
The Bigquery exporter plugin already supports this.

Describe the solution you'd like
Ability to opt into using Workload Identity Federation with the Stackdriver exporter so that I can send logs from AWS to Stackdriver without needing to manage long lived private keys.

Describe alternatives you've considered

• Fluent Bit -> Fluentd -> Stackdriver
  • Not possible as the Google Auth Ruby Gem also doesn't support WI federation (notwithstanding this PR)
• Passing private keys to fluent bit directly
  • This works fine, but lacks the security benefits of WI federation.
• Fluent Bit -> OpenTelemetry Collector -> Stackdriver
  • This is the option I'll likely go with for now to leverage the power of Fluent Bit's log agent and k8s log processing config that I know works, whilst being able to offload via Otel collector to Stackdriver (Go auth library supports WI federation).

[gfrankliu]

There is a workaround by running the gke-addon-sidecar as the metadata server. That sidecar supports the Workload Identity Federation, and re-expose as a metadata server on localhost. In the stackdriver output config, you can then point metadata_server to it. Sidecar image can be found at gcr.io/gke-multi-cloud-release/gke-addon-sidecar

[DeanBrunt]

There is a workaround by running the gke-addon-sidecar as the metadata server. That sidecar supports the Workload Identity Federation, and re-expose as a metadata server on localhost. In the stackdriver output config, you can then point metadata_server to it. Sidecar image can be found at gcr.io/gke-multi-cloud-release/gke-addon-sidecar

Ah interesting, thanks for this, this is potentially a neat workaround.
Do you know if there's some docs for the configuration surface of this sidecar?
I'm struggling to find anything when searching for it.

[JeffLuoo]

There is an example of using the sidecar in our another repo: https://github.com/GoogleCloudPlatform/anthos-samples/blob/main/aws-logging-monitoring/logging/forwarder.yaml#L94-L115

If you find this useful and we can add this to the documentation of Fluent Bit.

@DeanBrunt

Edit: The way the fleet workload identity works is to use a Kubernetes service account (KSA) to authenticate but you mentioned you want to use the AWS tokens? Could you provide the documentations of the federation between GCP and AWS?

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.