Cannot connect to AWS EKS with SSO

[bentatham]

I am having issues getting headlamp to connect to my AWS EKS cluster.

It works fine in OpenLens and kubectl. Not sure how headlamp connects, but perhaps it needs additional related AWS libraries loaded if using something in AWS SDK directly?

We are using AWS Identity Center (formerly AWS SSO) to authenticate to the cluster, so maybe that's the issue? The UI just says Bad Gaterway and the logs have this. No idea which certificate it can't verify?

09:43:59.460 › server process stderr: 2024/02/14 09:43:59 http: proxy error: tls: failed to verify certificate: x509: certificate signed by unknown authority

09:44:08.935 › server process stderr: 2024/02/14 09:44:08 Error: failed to get context: key not found

09:44:12.280 › server process stderr: 2024/02/14 09:44:12 http: proxy error: tls: failed to verify certificate: x509: certificate signed by unknown authority

09:44:12.286 › server process stderr: 2024/02/14 09:44:12 http: proxy error: tls: failed to verify certificate: x509: certificate signed by unknown authority

[elrondvega]

From my experience this seems to happen because the CA cert being used to connect to the API endpoint is the one being mounted in with the service account in the container at /var/run/secrets/kubernetes.io/serviceaccount. If your cluster is signed by a different CA then the one used for your api endpoint then you'll see this. In my case the cluster CA was self-signed and the api endpoint was a letsencrypt signed.

[joaquimrocha]

cc/ @yolossn .

[kdeyko]

I have a similar issue with EKS (but without SSO): there are "Bad Gateway" messages in the UI.
Surprisingly, if I open the app via open /Applications/Headlamp.app in the terminal, it connects just fine 🤔

Just in case: version 0.23.1 on macOS.

[lindblombr]

Can confirm what @kdeyko says. I have a similar issue with a custom OIDC binary we invoke from exec. If we open the app through point-and-click (macOS), we just get bad gateway and lots of errors in the dev console connecting to localhost. If I open up in a terminal via /Applications/Headlamp.app/Contents/MacOS/Headlamp, everything works perfectly.

[lindblombr]

This looks like it can be related to #1885

[illume]

I hope a fix for this is in the latest release.

Is anyone able to confirm if the latest release works for them or not?

[kdeyko]

I still see the issue on v0.24.0.

[adamstirk-ct]

The issue also still exists for me on v0.24.0

[rajeeshckr]

I still see the issue on 0.24.0

[EsDmitrii]

The same using Helm chart
Can't add my custom CA to the chart or rebuild image with it

[dev-2-4-h]

I wanted to use Headlamp, but the Authentication using a Service Account is not working. I use EKS (with SSO) and kubectl or other tools, and they work on my desktop, not so headlamp.
It is sad =-) that the dev team is not interested in solving this issue, as I see Headlamp as the best tool for the job.

[joaquimrocha]

Hi @dev-2-4-h . We do want to fix this issue and we thought it had been fixed, but apparently it's still happening. The issue is marked as a blocker and will be our focus for the coming weeks. So please allows us a bit more time and patience, as we want to solve this problem for everybody.

[joaquimrocha]

BTW @dev-2-4-h , can you help us debug this? If you run headlamp from the command line, will it be able to use the exec in kubeconfig? This is an issue we had and we thought we had fixed but apparently it's still happening.

[dev-2-4-h]

@joaquimrocha -Thanks for your replay and I love to help. However, I have no clue where to start.
I have not really looked into the source code. ;-(
If you can provide a short guide of the things you need? Logs etc - and how to get them?
My Kubctrl runs on an Ubuntu 24.04 VM - as I use a Mac with M2 processor as host, the Ubuntu is ARM64 - maybe his is the first indicator. I can also offer to do a 1to1 via Slack or other messengers if that helps.

[joaquimrocha]

@dev-2-4-h , no worries. With the info we got now I think we can pursue this issue. I will ping you and the others when we have a test build with a fix, so you can try and let us know if it's working for you. Thank you!

[illume]

@yolossn I think these fixes help with this issue too?

• frontend: Fix refreshToken #2269
• app: Improve shell env capture for better cross-platform support #2271

[yolossn]

@yolossn I think these fixes help with this issue too?

• frontend: Fix refreshToken #2269
• app: Improve shell env capture for better cross-platform support #2271

Yes, Also

backend: Make sure context name is DNS compatible #2270

[illume]

Folks, there are a number of fixes for this now in the latest release. So I’m going to close this now, but please comment if there’s still an issue for you.

[mattiaperi]

Hi, just to let you know that, unfortunately, I'm still experiencing the issue (Version 0.25.1 (0.25.1)). Clusters connected via AWS SSO, "Bad Gateway". The same cluster connected outside SSO, no problem.

[illume]

Thanks @mattiaperi I'll reopen this.

[acelinkio]

Following up on this. In previous versions I was able to leverage the workaround by launching headlamp from the terminal via /Applications/Headlamp.app/Contents/MacOS/Headlamp. However now, I just see a bunch of getting credentials errors exec: executable aws failed with exit code 254.

I cannot connect to even my local clusters that do not use SSO. I have to close and relaunch Headlamp via open /Applications/Headlamp.app/Contents/MacOS/Headlamp from the terminal in order to connect to my k3s cluster. Nothing works from the dock anymore

meanwhile k9s/openlens/kubectl all work without any issues.

[Ca-moes]

When running with @acelinkio solution: open /Applications/Headlamp.app/Contents/MacOS/Headlamp there was an error log on the terminal that HeadLamp was not able to find the following file ~/Library/Application\ Support/Headlamp/kubeconfigs/config.

I ran cp ~/.kube/config ~/Library/Application\ Support/Headlamp/kubeconfigs/config, restarted HeadLamp as was able to connect to my EKS clusters. even running Headlamp from the MacOS Launchpad instead of the CLI works.

I'm not sure if this was what fixed it for me (also logged in to GKE and Omni clusters, that were also logging errors on the console, even if I didn't access them) but for now it seems to be working.

---

EDIT: On a different day, started HeadLamp and got the same error described on this Issue, of not being able to authenticate. This time I logged in to Omni, GCloud and AWS, to login to all of my clusters, and after that I was able to login normally.