Serve metrics via client_golang

[siggy]

Switch to exporting metrics using the Prometheus client_golang library,
to help ensure we conform to Prometheus metrics best practices. This
change also decreases scrape latency and memory usage.

Summary of changes:

• generate metrics values out-of-band from scrape
• remove duplicate time series from /metrics
• add histogram _sum and _count metrics
• ensure consistent labels per metric
• deterministic label ordering per timeseries
• HELP and TYPE metadata support

Signed-off-by: Andrew Seigner [email protected]

misc testing

main

local

$ time curl -s -o /dev/null http://localhost:4191/metrics

real	0m3.282s
user	0m0.294s
sys	0m0.203s

this branch

local

$ time curl -s -o /dev/null http://localhost:4191/metrics

real	0m1.227s
user	0m0.147s
sys	0m0.189s

resource usage in AKS, with 40 agents

$ kubectl --context aks-load-test -n fauxmetheus top pods
NAME                           CPU(cores)   MEMORY(bytes)
fauxmetheus-5d5f4d4568-d4ww5   2060m        2521Mi

successful concurrent scrapes in AKS

$ curl -s localhost:4191/metrics|grep promhttp_metric_handler_requests
# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 49
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 5578
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0

[adleong]

Very nice!

[adleong]

should this be removed? I believe the client_id label is only used for inbound metrics.

[siggy]

the client_golang library will fail with a runtime error if we remove this, as labels should never be optional within a single metric.

we're caught between conforming to prom best practices (and the emerging openmetrics spec) and reproducing the proxy's output. eventually i'd expect the proxy to conform to openmetrics, part of that work is documented in linkerd/linkerd2#4102.