Client ends HTTP TLS session with RST

[marcindulak]

The client ends HTTP TLS session with RST, instead of TCP FIN/FIN/ACK.

First, let me say that I'm not sure this a problem with caddy. I reproduce the problem using curl 8.9.0 and GNU Wget 1.24.5 as clients, against caddy v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk= as the server. I don't see the problem against nginx/1.27.0 as the server. The host is an M1 MacOS arm64.

To reproduce

1. Use the following Caddyfile, to start caddy container

   localhost:443 {
        tls internal {
                insecure_secrets_log /srv/SSLKEYLOGFILE
        }
        handle / {
                respond "Hello, HTTP version {http.request.proto}!"
        }
   }
   

2. Start the container

   docker run --detach --rm -v $PWD/Caddyfile:/etc/caddy/Caddyfile -v $PWD:/srv --name caddy caddy:2.8.4-alpine
   

3. Install tools into the container and start tcpdump

   docker exec -it caddy sh -c "apk add procps vim curl iproute2 tcpdump wireshark-common tshark wget"
   docker exec -it caddy sh -c "tcpdump -i lo -w /srv/test.pcap '(tcp or udp) and port 443'"
   

4. In other terminal, make HTTPS request, and afterwards stop the tcpdump with ctrl+c

   docker exec -it caddy sh -c "curl -vI --http1.1 https://localhost"
   

5. Decrypt the traffic

   docker exec -it caddy sh -c "editcap --inject-secrets tls,SSLKEYLOGFILE test.pcap test.pcapng"
   

Result

Here is the packet sequence from caddy as the server, read with docker exec -it caddy sh -c "tshark -r test.pcapng"
Note that last RST packet

    1   0.000000    127.0.0.1 → 127.0.0.1    TCP 74 53212 → 443 [SYN] Seq=0 Win=65495 Len=0 MSS=65495 SACK_PERM TSval=2032565199 TSecr=0 WS=128
    2   0.000027    127.0.0.1 → 127.0.0.1    TCP 74 443 → 53212 [SYN, ACK] Seq=0 Ack=1 Win=65483 Len=0 MSS=65495 SACK_PERM TSval=2032565199 TSecr=2032565199 WS=128
    3   0.000036    127.0.0.1 → 127.0.0.1    TCP 66 53212 → 443 [ACK] Seq=1 Ack=1 Win=65536 Len=0 TSval=2032565200 TSecr=2032565199
    4   0.003347    127.0.0.1 → 127.0.0.1    TLSv1 583 Client Hello (SNI=localhost)
    5   0.003367    127.0.0.1 → 127.0.0.1    TCP 66 443 → 53212 [ACK] Seq=1 Ack=518 Win=65024 Len=0 TSval=2032565203 TSecr=2032565203
    6   0.005629    127.0.0.1 → 127.0.0.1    TLSv1.3 1495 Server Hello, Change Cipher Spec, Encrypted Extensions, Certificate, Certificate Verify, Finished, New Session Ticket
    7   0.005635    127.0.0.1 → 127.0.0.1    TCP 66 53212 → 443 [ACK] Seq=518 Ack=1430 Win=64256 Len=0 TSval=2032565205 TSecr=2032565205
    8   0.016725    127.0.0.1 → 127.0.0.1    TLSv1.3 130 Change Cipher Spec, Finished
    9   0.017028    127.0.0.1 → 127.0.0.1    HTTP 161 HEAD / HTTP/1.1 
   10   0.017039    127.0.0.1 → 127.0.0.1    TCP 66 443 → 53212 [ACK] Seq=1430 Ack=677 Win=65536 Len=0 TSval=2032565217 TSecr=2032565216
   11   0.017148    127.0.0.1 → 127.0.0.1    HTTP 252 HTTP/1.1 200 OK 
   12   0.017359    127.0.0.1 → 127.0.0.1    TLSv1.3 90 Alert (Level: Warning, Description: Close Notify)
   13   0.017599    127.0.0.1 → 127.0.0.1    TLSv1.3 90 Alert (Level: Warning, Description: Close Notify)
   14   0.017613    127.0.0.1 → 127.0.0.1    TCP 66 443 → 53212 [FIN, ACK] Seq=1640 Ack=701 Win=65536 Len=0 TSval=2032565217 TSecr=2032565217
   15   0.018616    127.0.0.1 → 127.0.0.1    TCP 66 53212 → 443 [RST, ACK] Seq=701 Ack=1641 Win=65536 Len=0 TSval=2032565218 TSecr=2032565217

Here is the corresponding packet sequence from nginx as the server.

    1   0.000000   172.30.0.5 → 172.30.0.3   TCP 74 45948 → 443 [SYN] Seq=0 Win=32120 Len=0 MSS=1460 SACK_PERM TSval=2534430028 TSecr=0 WS=4
    2   0.000089   172.30.0.3 → 172.30.0.5   TCP 74 443 → 45948 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1460 SACK_PERM TSval=3424397642 TSecr=2534430028 WS=128
    3   0.000102   172.30.0.5 → 172.30.0.3   TCP 66 45948 → 443 [ACK] Seq=1 Ack=1 Win=32120 Len=0 TSval=2534430028 TSecr=3424397642
    4   0.005571   172.30.0.5 → 172.30.0.3   TLSv1 464 Client Hello (SNI=server-nginx)
    5   0.005696   172.30.0.3 → 172.30.0.5   TCP 66 443 → 45948 [ACK] Seq=1 Ack=399 Win=64768 Len=0 TSval=3424397647 TSecr=2534430033
    6   0.008106   172.30.0.3 → 172.30.0.5   TLSv1.3 1701 Server Hello, Change Cipher Spec, Encrypted Extensions, Certificate, Certificate Verify, Finished
    7   0.008126   172.30.0.5 → 172.30.0.3   TCP 66 45948 → 443 [ACK] Seq=399 Ack=1636 Win=31312 Len=0 TSval=2534430036 TSecr=3424397650
    8   0.008422   172.30.0.5 → 172.30.0.3   TLSv1.3 72 Change Cipher Spec
    9   0.009052   172.30.0.5 → 172.30.0.3   TLSv1.3 140 Finished
   10   0.009118   172.30.0.3 → 172.30.0.5   TCP 66 443 → 45948 [ACK] Seq=1636 Ack=479 Win=64768 Len=0 TSval=3424397651 TSecr=2534430036
   11   0.009286   172.30.0.3 → 172.30.0.5   TLSv1.3 369 New Session Ticket
   12   0.009327   172.30.0.3 → 172.30.0.5   TLSv1.3 369 New Session Ticket
   13   0.010075   172.30.0.5 → 172.30.0.3   HTTP 168 HEAD / HTTP/1.1 
   14   0.010429   172.30.0.3 → 172.30.0.5   HTTP 326 HTTP/1.1 200 OK 
   15   0.010849   172.30.0.5 → 172.30.0.3   TLSv1.3 90 Alert (Level: Warning, Description: Close Notify)
   16   0.010894   172.30.0.5 → 172.30.0.3   TCP 66 45948 → 443 [FIN, ACK] Seq=605 Ack=2502 Win=31320 Len=0 TSval=2534430039 TSecr=3424397652
   17   0.011040   172.30.0.3 → 172.30.0.5   TCP 66 443 → 45948 [FIN, ACK] Seq=2502 Ack=606 Win=64768 Len=0 TSval=3424397653 TSecr=2534430039
   18   0.011045   172.30.0.5 → 172.30.0.3   TCP 66 45948 → 443 [ACK] Seq=606 Ack=2503 Win=31320 Len=0 TSval=2534430039 TSecr=3424397653

The problem with ending RST is also present for HTTP2.

The pcapng files are attached.

Contain RST, curl/wget request to caddy
http2.pcapng.gz
http1.1.pcapng.gz
http1.1wget.pcapng.gz

Without RST, curl request to nginx
http1.1nginx.pcapng.gz

[marcindulak]

The expected FIN->FIN->ACK sequence appears to occur when the docker container is started inside of a virtual machine, for example in a ubuntu/jammy64 vagrant box.

RST occurs when the docker container is started directly on the host, both Ubuntu 20.04 amd64 and MacOS M1 arm64 host.

I see a similar RST behavior for gRPC client https://github.com/marcindulak/grpc-client-tcp-rst, for cleartext HTTP2 (h2c).

[francislavoie]

Caddy itself isn't doing anything this low level at the TCP layer. You'll probably want to report this on https://github.com/golang/go if you think it's an issue. I don't understand why you think it is an issue though.

[mholt]

So the difference in behavior is whether the container is started on the host or in a VM? And with the VM setup, the connection is routed into the VM and into the container?

I do agree though, that we don't have any TCP-specific code in our code base, and this is probably something to report to Go instead, if there's a bug.

Thanks! We can reopen if this is wrong.

[marcindulak]

Yes there is a difference between starting the container on a host, or in a vm on the same host.

With an installation of docker inside of a VM, using the steps 1-5, I get no RST. The connection is made from the host into the container over ssh, and the steps 1-5 performed in the VM.

After seeing this, as another attempt, on the host, killed all open browsers and disconnected the network, still got RST from containers started on the host.

RST is not expected, unless someone says "yes, we are doing RST explicitly instead of FIN" for resource usage optimization or other purposes.

I'm looking for a smaller example, but until now what I found is caddy and gRPC (both Golang (RST sometimes) and Python (RST probably always)).