Random UnexpectedConnectionClosureException with akka-http client

[mdedetrich]

Akka-http Version: 10.0.10
Scala Version: 2.11.11/2.12.3

We are experiencing issues where we randomly get a

akka.http.impl.engine.client.OutgoingConnectionBlueprint$UnexpectedConnectionClosureException: The http server closed the connection unexpectedly before delivering responses for 1 outstanding requests

when running a test suite that generates a lot of connections/load

I have attempted debugging with

    <logger name="akka.http.impl.engine.client.PoolGateway" level="DEBUG" additivity="false">
        <appender-ref ref="STDOUT"/>
    </logger>

And after inspecting the PoolGateway all of the connections are correctly being acquired/released (so its not an issue with a http request discardEntityBytes()/Unmarshal not being called)

Http-pipelining is also disabled (here is the application.conf we are using). Note for the akka-http client, http-pipelining should be disabled by default (according to the default conf here https://doc.akka.io/docs/akka-http/current/scala/http/configuration.html).

akka {
  loglevel = "DEBUG"
  loggers = ["akka.event.slf4j.Slf4jLogger"]
  logging-filter = "akka.event.slf4j.Slf4jLoggingFilter"
  http {
    host-connection-pool {
      // This is to speed up our tests
      max-connections = 32
      max-open-requests = 256
    }
  }
}

I highly doubt its an issue with the server we are calling since we used a different http client before (http4s) and the same test suite ran without issues (also the error happens with different hosts running different http stacks). Unfortunately this is company code so I can't post it here, however we have a lightbend subscription at our company if it helps diagnosing the problem.

Is there any low hanging fruit which can help in diagnosing/fixing this issue?

[mdedetrich]

I also believe that #768 would help (at least temporarily) in resolving this

[mdedetrich]

Close as duplicate

[jrudolph]

ftr: Now filed as #1460