self-hosting: enabling TLS causes the client to always receive '400 Bad Request'

[ajc]

I've been trying to setup a self-hosted version of the charm service using the package from your APT/DEB repo on an Ubuntu 22.04.1 LTS system (I've tried on both amd64 and arm64 systems).

% charm -v
charm version v0.12.4 (099f4f1)

I have a Let's Encrypt-provided TLS certificate, and I've set it up via systemd with these environment variables:

CHARM_SERVER_DATA_DIR=/some/path/charm/data
CHARM_SERVER_HOST=myhostname
CHARM_SERVER_USE_TLS=true
CHARM_SERVER_TLS_KEY_FILE=/etc/letsencrypt/live/myhostname/privkey.pem
CHARM_SERVER_TLS_CERT_FILE=/etc/letsencrypt/live/myhostname/fullchain.pem

Starting it, I see this in the systemd journal:

Jan 07 17:20:24 irmin charm[84564]: 2023/01/07 17:20:24 Opening SQLite db: /some/path/charm/data/db/charm_sqlite.db
Jan 07 17:20:24 irmin charm[84564]: 2023/01/07 17:20:24 Starting SSH server on :35353
Jan 07 17:20:24 irmin charm[84564]: 2023/01/07 17:20:24 Starting HTTPS server on: :35354
Jan 07 17:20:24 irmin charm[84564]: 2023/01/07 17:20:24 Starting HTTPS health server on: :35356

I can see that the certificate served up is valid, using certigo:

% certigo connect myhostname:35354
** TLS Connection **
Version: TLS 1.3
Cipher Suite: CHACHA20_POLY1305_SHA256 cipher

** CERTIFICATE 1 **
Valid: 2023-01-07 02:57 UTC to 2023-04-07 02:57 UTC
Subject:
        CN=myhostname
Issuer:
        C=US, O=Let's Encrypt, CN=R3
DNS Names:
        myhostname

** CERTIFICATE 2 **
Valid: 2020-09-04 00:00 UTC to 2025-09-15 16:00 UTC
Subject:
        C=US, O=Let's Encrypt, CN=R3
Issuer:
        C=US, O=Internet Security Research Group, CN=ISRG
        Root X1

** CERTIFICATE 3 **
Valid: 2021-01-20 19:14 UTC to 2024-09-30 18:14 UTC
Subject:
        C=US, O=Internet Security Research Group, CN=ISRG
        Root X1
Issuer:
        O=Digital Signature Trust Co., CN=DST Root CA X3

Checked OCSP status for certificate, got:
        Good (last update: 07 Jan 23 04:02 UTC)

Found 1 valid certificate chain(s):
[0] CN=myhostname
        => CN=R3
        => CN=ISRG Root X1 [self-signed]

When I run charm from another host, I see this in the server's journal:

Jan 07 17:20:31 irmin charm[84564]: 2023/01/07 17:20:31 ssh id
Jan 07 17:20:31 irmin charm[84564]: 2023/01/07 17:20:31 ID for user <UUID redacted>
Jan 07 17:20:31 irmin charm[84564]: 2023/01/07 17:20:31 ssh api-auth
Jan 07 17:20:31 irmin charm[84564]: 2023/01/07 17:20:31 JWT for user <UUID redacted>

But back on the other host, I just see

   Charm

  Uh oh, there’s been an error: server error: 400 Bad Request

If I change CHARM_SERVER_USE_TLS to false and restart the service, then running charm from another host seems to work just fine (I see the menu to Link a machine, Manage linked keys, Set username, Backup or Exit.)

What am I doing wrong when I have TLS turned on?

Cheers,
Cos.

[marcellmars]

I encountered the same issue. Here's a snippet of code that resolved it for me. This was my first time working with this codebase, and I didn't have enough time to figure out and follow the existing patterns.

This is just a quick fix that worked for me.
It requires setting the env variable:

• export CHARM_USE_TLS=true

I can imagine keeping the env name CHARM_SERVER_USE_TLS would work too. Or CHARM_CLIENT_USE_TLS. I might be off by one here but naming this is not very high on my TODO list today :)

If anyone could direct me to the proper place for this fix, I'd be more than willing to make it into a PR. Otherwise, I hope it can help someone more familiar with the codebase understand why both of us in this thread have encountered the same problem. :)

modified   client/auth.go
@@ -29,7 +29,6 @@ func (cc *Client) Auth() (*charm.Auth, error) {
 		if err != nil {
 			return nil, charm.ErrAuthFailed{Err: err}
 		}
-		cc.httpScheme = auth.HTTPScheme
 		p := &jwt.Parser{}
 		token, _, err := p.ParseUnverified(auth.JWT, &jwt.RegisteredClaims{})
 		if err != nil {
modified   client/client.go
@@ -34,6 +34,7 @@ type Config struct {
 	KeyType     string `env:"CHARM_KEY_TYPE" envDefault:"ed25519"`
 	DataDir     string `env:"CHARM_DATA_DIR" envDefault:""`
 	IdentityKey string `env:"CHARM_IDENTITY_KEY" envDefault:""`
+	UseTLS      bool   `env:"CHARM_USE_TLS" envDefault:"false"`
 }
 
 // Client is the Charm client.
@@ -110,6 +111,10 @@ func NewClient(cfg *Config) (*Client, error) {
 		Auth:            []ssh.AuthMethod{pkam},
 		HostKeyCallback: ssh.InsecureIgnoreHostKey(), // nolint
 	}
+
+     cc.httpScheme = "http"
+	if cfg.UseTLS {
+		cc.httpScheme = "https"
+	}
 	return cc, nil
 }

[mbentley]

I am not sure if this will help anyone else but in my case, I was setting up my own self-hosted charm server with TLS and I was finding that I was also getting 400 Bad Request errors from charm and skate. I did not have a reverse proxy set up as I am just running charm from inside a docker container but am using a macvlan interface for my network so it just appears like it is running on it's own dedicated machine/IP. The documentation makes it seem like CHARM_SERVER_PUBLIC_URL would only be needed when using a reverse proxy but this doesn't appear to be the case when using TLS.

CHARM_SERVER_PUBLIC_URL="https://charm.example.com:35354"
CHARM_SERVER_HOST="charm.example.com"
CHARM_SERVER_TLS_CERT_FILE="/certs/tls.crt"
CHARM_SERVER_TLS_KEY_FILE="/certs/tls.key"
CHARM_SERVER_USE_TLS=true