Server statistics in Akka HTTP

[akka-ci]

Issue by RichardBradley
Monday Mar 30, 2015 at 10:44 GMT
Originally opened as akka/akka#17095

---

Spray 1.3 had "Server Statistics" that reported on various connection-level stats that are difficult or inefficient to monitor in userland:

case class Stats(
  uptime: FiniteDuration,
  totalRequests: Long,
  openRequests: Long,
  maxOpenRequests: Long,
  totalConnections: Long,
  openConnections: Long,
  maxOpenConnections: Long,
  requestTimeouts: Long)

Please could we add something similar to Akka HTTP 1.0?

[akka-ci]

Comment by ktoso
Monday Mar 30, 2015 at 15:00 GMT

---

// cc @sirthias @jrudolph

[akka-ci]

Comment by sirthias
Monday Mar 30, 2015 at 19:21 GMT

---

Back in 2013, in the first PR for what is now Akka IO (akka/akka#1030), I was proposing to port over the simple metrics code we used to have in spray-io back then:

@viktorklang added a note on 16 Jan 2013
I don't think we should bake in stats into this

@sirthias added a note on 16 Jan 2013
This is a simple facility providing introspection onto how many connections the selector actors are currently handling as well as how many connections they have handled so far. Isn't this something valuable?

@viktorklang added a note on 16 Jan 2013
No, we don't explicitly put monitoring code in, it will be woven in (everyone has their own preference of stats collection).

@sirthias added a note on 16 Jan 2013
Ok, no problem. Will be ripped out.

So, my understanding is that, if things haven't changed, it's a general policy to not include any kind of stats/metrics code within the Akka modules themselves.

[akka-ci]

Comment by RichardBradley
Tuesday Mar 31, 2015 at 09:00 GMT

---

OK, fair enough, but if that's the plan, then there need to be extension points where I can weave in my own preferred stats library.
How can I collect these stats in Akka HTTP?

I suppose it'll be something like a custom Http.bindAndstartHandlingWith to count connections, and a custom Route.handlerFlow to count requests, but it would be nice to have a worked example in the docs if this is now up to the end users.

I'll submit a docs PR if I get this working.

[akka-ci]

Comment by RichardBradley
Tuesday Apr 07, 2015 at 20:17 GMT

---

Hmm -- there's quite a bit of finessing around atomicity / race conditions in the StatsSupport class in Spray (100+ lines). I suppose I need to copy and paste all of that into my app to do this in userland?

Is this really something we want people to be copy+pasting from the Akka docs?

I'll try doing that and see how that docs PR might look.

[akka-ci]

Comment by RichardBradley
Wednesday Apr 08, 2015 at 09:04 GMT

---

I’ve reimplemented the Spray HTTP stats in “userland” in my app, and a docs PR to Akka would look something like the following.
@viktorklang / @sirthias, please could you let me know what you think?

I think there are a couple of issues which make this unsuitable for a “recipe” in the docs for users to copy and paste into their apps:

1. The length of the code involved
2. The “leaked” open connections issue – can you see a way to fix this?

If you’re sure this belongs in the docs and not in the framework, I can convert this draft into a PR to the docs RST files:

Proposed new documentation section starts here:

Server HTTP statistics

To allow for users to connect to their own preferred statistics libraries, Akka HTTP does not have any built-in statistics monitoring.
You can capture the basic HTTP metrics using code like the following:

Add the following class to your application:

import akka.http.model.{HttpRequest, HttpResponse}
import akka.stream.stage.{Context, Directive, PushStage, TerminationDirective}
import java.util.concurrent.TimeUnit
import java.util.concurrent.atomic.AtomicLong
import scala.annotation.tailrec
import scala.concurrent.duration._

/**
 * This is based on StatsSupport from Spray 1.3.1
 *
 * See https://github.com/spray/spray/blob/269ce885d3412e555237bb328aae89457f57c660/spray-can/src/main/scala/spray/can/server/StatsSupport.scala
 * See https://github.com/akka/akka/issues/17095
 */
private object StatsSupport {

  class StatsHolder {
    private val startTimeMillis = System.currentTimeMillis()
    // FIXME: Spray used "PaddedAtomicLong" here -- is that important?
    private val requestStarts = new AtomicLong
    private val responseStarts = new AtomicLong
    private val maxOpenRequests = new AtomicLong
    private val connectionsOpened = new AtomicLong
    private val connectionsClosed = new AtomicLong
    private val maxOpenConnections = new AtomicLong

    private def onConnectionStart(): Unit = {
      connectionsOpened.incrementAndGet()
      adjustMaxOpenConnections()
    }

    private def onConnectionEnd(): Unit = {
      connectionsClosed.incrementAndGet()
    }

    private def onRequestStart(): Unit = {
      requestStarts.incrementAndGet()
      adjustMaxOpenRequests()
    }

    private def onResponseStart(): Unit = {
      responseStarts.incrementAndGet()
    }

    @tailrec
    private def adjustMaxOpenConnections(): Unit = {
      val co = connectionsOpened.get
      val cc = connectionsClosed.get
      val moc = maxOpenConnections.get
      val currentMoc = co - cc
      if (currentMoc > moc)
        if (!maxOpenConnections.compareAndSet(moc, currentMoc)) adjustMaxOpenConnections()
    }

    @tailrec
    private def adjustMaxOpenRequests(): Unit = {
      val rqs = requestStarts.get
      val rss = responseStarts.get
      val mor = maxOpenRequests.get

      // FIXME: if a connection was aborted after we saw a request and before we
      // saw a response, then we will "leak" an apparently open request here...
      val currentMor = rqs - rss
      if (currentMor > mor)
        if (!maxOpenRequests.compareAndSet(mor, currentMor)) adjustMaxOpenRequests()
    }

    def toStats = Stats(
      uptime = FiniteDuration(System.currentTimeMillis() - startTimeMillis, TimeUnit.MILLISECONDS),
      totalRequests = requestStarts.get,
      openRequests = requestStarts.get - responseStarts.get,
      maxOpenRequests = maxOpenRequests.get,
      totalConnections = connectionsOpened.get,
      openConnections = connectionsOpened.get - connectionsClosed.get,
      maxOpenConnections = maxOpenConnections.get)

    def clear(): Unit = {
      requestStarts.set(0L)
      responseStarts.set(0L)
      maxOpenRequests.set(0L)
      connectionsOpened.set(0L)
      connectionsClosed.set(0L)
      maxOpenConnections.set(0L)
    }

    /**
     * Create a PushStage which should be inserted into the connection flow
     * before the sealed route.
     *
     * This is also used to watch the connections.
     */
    def watchRequests(): PushStage[HttpRequest, HttpRequest] = {

      onConnectionStart()

      new PushStage[HttpRequest, HttpRequest] {

        override def onPush(
          request: HttpRequest,
          ctx: Context[HttpRequest]
        ): Directive = {
          onRequestStart()

          ctx.push(request)
        }

        override def onUpstreamFailure(
          cause: Throwable,
          ctx: Context[HttpRequest]
        ): TerminationDirective = {
          onConnectionEnd()
          ctx.fail(cause)
        }

        override def onUpstreamFinish(
          ctx: Context[HttpRequest]
        ): TerminationDirective = {
          onConnectionEnd()
          ctx.finish()
        }
      }
    }

    /**
     * Create a PushStage which should be inserted into the connection flow
     * after the sealed route.
     *
     * Connections are not counted here.
     */
    def watchResponses(): PushStage[HttpResponse, HttpResponse] = {

      new PushStage[HttpResponse, HttpResponse] {

        override def onPush(
          Response: HttpResponse,
          ctx: Context[HttpResponse]
        ): Directive = {
          onResponseStart()

          ctx.push(Response)
        }
      }
    }
  }
}

/**
 * Note that 'requestTimeouts' is missing v.s. Spray 1.3
 *
 * Note that 'openRequests' may drift upwards over time due to aborted
 * connections!
 */
case class Stats(
  uptime: FiniteDuration,
  totalRequests: Long,
  openRequests: Long,
  maxOpenRequests: Long,
  totalConnections: Long,
  openConnections: Long,
  maxOpenConnections: Long)

Change the server “Http. bindAndstartHandlingWith” call in your server bootstrapping code to look like:

val serverFlow = Http().bind(interface = host, port = port)

val route: Route = ...
val sealedRoute: Flow[HttpRequest, HttpResponse, Unit] =
  route2HandlerFlow(route)(routingSetup)
val stats = new StatsHolder

val serverBindingFuture = serverFlow.to(Sink.foreach { conn =>

  conn.flow.transform(() => stats.watchRequests())
    .join(sealedRoute.transform(() => stats.watchResponses()))
    .run()

}).run()

Now you can call stats.toStats to get the current HTTP statistics for your server.

[akka-ci]

Comment by RichardBradley
Thursday Apr 16, 2015 at 08:42 GMT

---

Hi @viktorklang / @sirthias -- please could you take a look at this when you have a moment?

I have two open questions:

• My implementation of the stats based on Spray's version has a "leak" (see discussion above) - can you see a sensible fix?
• Would this (see draft above) make a sensible addition to the Akka docs? I will prepare a docs PR if you think so. However, I think it might make more sense to do this in Akka itself, as discussed above.

Thanks,

Rich

[akka-ci]

Comment by viktorklang
Monday Apr 20, 2015 at 19:30 GMT

---

Hi @RichardBradley, I'm unfortunately out of cycles for the moment, @rkuhn, could you have a look when you have a cycle or two to spare?

[akka-ci]

Comment by rkuhn
Tuesday Apr 21, 2015 at 08:50 GMT

---

We are very busy working towards HTTP 1.0-RC1 this week, will revisit this topic afterwards. In any case thanks for sharing your work!

[akka-ci]

Comment by RichardBradley
Tuesday Apr 21, 2015 at 09:54 GMT

---

OK, thanks, that's fine. I had previously thought this might be part of HTTP 1.0, to give feature parity with Spray 1.3, but I now agree it's fairly low priority. Take a look when you have time.

[gauravkumar37]

Do we have any solution for monitoring Akka HTTP? Any way to track metrics?

[ktoso]

Please use the akka-user mailing list for general questions, tickets are reserved for actionable code related things @gauravkumar37.

Lightbend monitoring tracks advanced things about Akka, we'll eventually add HTTP features to it (including tracing).
And you can use Kamon.io too.

[adamvoss]

Link in OP is broken, here is a fresh one to Server Statistics in the spray documentation.

[schrepfler]

I'm probably late to the party but, why don't we unify the metric names to be the same as the ones in finagle, it will definitely go long way in organisations which have standardised components and metrics based on the finagle stack?

[jrudolph]

@schrepfler Not sure what you mean, akka-http currently doesn't expose any metrics itself. Are you talking about the kamon exposed metrics?

[schrepfler]

That's what I'm saying, it would be really good if akka-http has a standard way to expose metrics so that customers have a choice to map the stats onto whatever they want. Unifying metrics brings a lot of value to stacks which are based on same core technology (like finagle) so whatever the service is, we always get the same stats so it's easy to drag and drop services and have generic dashboards out of the box.