Fix akka HTTP client POST request racing conditions

[t83714]

Fix akka HTTP client POST request racing conditions

Although it's rare, users might see the following error, when registry request send POST requests to OPA for auth decision:

akka.http.impl.engine.client.OutgoingConnectionBlueprint$UnexpectedConnectionClosureException: The http server closed the connection unexpectedly before delivering responses for 1 outstanding requests 

According to this issue, Akka will try to reuse the same connection for requests until server sends "Connection: close" header.

There might be a racing condition where akka http client sends out request when server's "Connection: close" header has left the server but still on its way to the client.

When that happens, if it's a HTTP GET request, akka client will auto retry it.

However, for POST requests, akka won't retry them (as POST requests can't be considered as idempotent) and akka.http.impl.engine.client.OutgoingConnectionBlueprint$UnexpectedConnectionClosureException error will be thrown.

This error will not happen very often but does create troubles for test cases in pipelines.

Solution

Manually retry POST request

[t83714]

closed via PR: #3252