Jetty server: connection closed while response is sent before having received the full chunked request

[jfyuen]

Jetty version(s)
Jetty server 12.0.16: org.eclipse.jetty:jetty-server:12.0.16"

Jetty Environment
core + ee10

Java version/vendor (use: java -version)

openjdk version "17.0.12" 2024-07-16 LTS
OpenJDK Runtime Environment Corretto-17.0.12.7.1 (build 17.0.12+7-LTS)
OpenJDK 64-Bit Server VM Corretto-17.0.12.7.1 (build 17.0.12+7-LTS, mixed mode, sharing)

OS type/version
Alma linux 9.5 (kernel 5.14.0-503.16.1.el9_5.x86_64), also reproducible on Macos 14.7 and Alma linux 8

Description
We recently upgraded a server from jetty 10 to jetty12. Upon testing, we noticed some requests failed from the server on localhost (so no network involved).
The http client (at the time) is Apache Http Client 4 4.5.14, and we had random disconnection with the following error:

org.apache.http.NoHttpResponseException: localhost:5555 failed to respond
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)

Upon further investigation, and a tcpdump, we came to the conclusion that the Jetty 12 server is responding before consuming the whole request, because it is reading a chunked request, see excerpt from wireshark below:

Jetty 12 server is then closing the connection, and the next request from the client fails with the above exception.

The same issue kind of also occured with Jetty 10, but the server doesn't behave the same and sends a Connection: Close header instead of just closing the connection, so the client was notified and could re-open a new one.

How to reproduce?

I managed to reproduce the issue with the following test and the JettyClient, as well as the Apache Client (both provided):

• jetty client: org.eclipse.jetty:jetty-client:12.0.16
• jettyserver: org.eclipse.jetty:jetty-server:12.0.16
• apache http client4: org.apache.httpcomponents:httpclient:4.5.14 (also reproduced with Apache http client 5).

package com.example;

import java.io.IOException;
import java.io.OutputStream;
import java.nio.ByteBuffer;
import java.nio.charset.StandardCharsets;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutionException;

import org.apache.commons.io.IOUtils;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.entity.ContentType;
import org.apache.http.entity.EntityTemplate;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;
import org.eclipse.jetty.client.HttpClient;
import org.eclipse.jetty.client.OutputStreamRequestContent;
import org.eclipse.jetty.io.Content;
import org.eclipse.jetty.server.Handler;
import org.eclipse.jetty.server.Request;
import org.eclipse.jetty.server.Response;
import org.eclipse.jetty.server.Server;
import org.eclipse.jetty.server.ServerConnector;
import org.eclipse.jetty.util.Callback;
import org.junit.jupiter.api.Test;

class JettyServerWithClientTest {
    private static Server createServer() {
        Server server = new Server(5555);
        server.setHandler(new Handler.Abstract() {
            @Override
            public boolean handle(Request request, Response response, Callback callback) throws IOException {
                response.setStatus(200);
                response.getHeaders().add("Content-Type", "application/json;charset=utf-8");
                String content = "{}";
                ByteBuffer buffer = ByteBuffer.wrap(content.getBytes(StandardCharsets.UTF_8));
//                IOUtils.consume(Content.Source.asInputStream(request)); // Consuming the request makes the  client works
                response.write(true, buffer, callback);
                return true;
            }
        });

        return server;
    }

    @Test
    public void testJettyClient() throws Exception {
        var port = startServer();
        var url = "http://localhost:" + port + "/";

        try (HttpClient httpClient = new HttpClient()) {
            httpClient.start();
            for (int i = 0; i < 1000000; i++) {
                OutputStreamRequestContent content = new OutputStreamRequestContent();
                httpClient.POST(url)
                        .body(content)
                        .send(result -> {
                        });
                try (OutputStream output = content.getOutputStream()) {
                    output.write("{}".getBytes());
                }
            }
        }
    }

    private static Integer startServer() throws InterruptedException, ExecutionException {
        CompletableFuture<Integer> future = new CompletableFuture<>();
        var t = new Thread(() -> {
            try {
                var server = createServer();
                server.start();
                var port = ((ServerConnector) server.getConnectors()[0]).getLocalPort();
                future.complete(port);
                server.join();
            } catch (Exception e) {
                throw new RuntimeException(e);
            }

        });
        t.start();
        return future.get();
    }


    @Test
    public void testApacheHttpClient4() throws Exception {
        var port = startServer();

        var httpClient = HttpClients.createDefault();
        var url = "http://localhost:" + port + "/";

        for (int i = 0; i < 1000000; i++) {
            var post = new HttpPost(url);
            var entity = new EntityTemplate(outputStream -> {
                outputStream.write("{}".getBytes());
                outputStream.flush();
            });
            entity.setContentType(ContentType.APPLICATION_JSON.toString());
            post.setEntity(entity);
            var resp = httpClient.execute(post);
            try {
                assert resp.getStatusLine().getStatusCode() == 200;
                assert "{}".equals(IOUtils.toString(resp.getEntity().getContent(), StandardCharsets.UTF_8));
            } finally {
                EntityUtils.consume(resp.getEntity());
            }
        }
    }
}

Using the JettyClient, I get the following error:

java.nio.channels.AsynchronousCloseException
	at org.eclipse.jetty.client.transport.internal.HttpConnectionOverHTTP.close(HttpConnectionOverHTTP.java:285)
	at org.eclipse.jetty.client.transport.internal.HttpReceiverOverHTTP.earlyEOF(HttpReceiverOverHTTP.java:539)
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1753)
	at org.eclipse.jetty.client.transport.internal.HttpReceiverOverHTTP.parse(HttpReceiverOverHTTP.java:319)
	at org.eclipse.jetty.client.transport.internal.HttpReceiverOverHTTP.parseAndFill(HttpReceiverOverHTTP.java:248)
	at org.eclipse.jetty.client.transport.internal.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:77)
	at org.eclipse.jetty.client.transport.internal.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:97)
	at org.eclipse.jetty.client.transport.internal.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:250)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99)
	at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:480)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:443)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:201)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:311)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:979)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1209)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1164)
	at java.base/java.lang.Thread.run(Thread.java:840)

and with tcpdump:

The issue is "resolved" by consuming the request, see commented line on the test:

IOUtils.consume(Content.Source.asInputStream(request));

Any idea of what might be the cause? Do I need to always fully consume the request before sending a response? I

[joakime]

This is standard HTTP/1.1 behavior.

If your endpoint doesn't consume the request body content, then you just broke HTTP/1.1 persistent connection behavior.

If you use HTTP/2, then this behavior (of not reading the request body) is fully supported by the protocol.

In past versions of Jetty, based on the Servlet spec, even in the core level of Jetty, the request would attempted to consume and remaining unconsumed content (within limits) to satisfy Servlet behaviors.

This attempt at consuming unconsumed request content has been a security issue for a while now. (how much do we read? under what conditions do we read it? does the response influence this behavior? only error status codes? or all status codes? etc. a long list of questions about this behavior over the years)

We've seen bad actors send multi gigabyte POST requests with the server responding quickly that it doesn't want that, but the servlet behaviors around unconsumed request body content meant jetty had to waste time consuming that multi gigabyte POST request body. Some bad actors would intentionally even send it slowly to not trigger idle timeout and just waste system resources.

So when we removed Servlet from our core behaviors in Jetty 12, the oddball servlet behaviors followed, including the attempt to deal with unconsumed request body content. Leaving that behavior to the endpoint to determine what to do.

[jfyuen]

Thanks a lot for you detailed answer, very informative.
I actually had the same behavior with a HttpServlet, but tried to have a minimal repro-case.

Does that mean that even if we respond an "error code" (like 4XX, 5XX), the request also needs to be fully consumed to comply with HTTP/1.1? Because I guess we can observe the same behavior while the client reuses a connection after receiving such a code if the request was not fully consumed.

Or is it safer to manually add a Connection: Close header to the response in that case to ensure that the http client will close the connection and open a new one for the next request?