Support prometheus-operator

[eh-am]

Prometheus operator doesn't use annotations, but uses ServiceMonitors/PodMonitors instead. It would be nice to have first class support for it.

[kolesnikovae]

I think we need our own operator with own CRDs: we already mentioned this idea during internal discussions and I believe the time has come.

And that's going to be a separate component (not part of the server) as I see it (optionally, with its own repo and helm chart).

[eh-am]

Not really, I am talking about having a flag such as serviceMonitor.enabled that generates a yaml like https://github.com/prometheus-operator/prometheus-operator/blob/9c0db5656f04e005de6a0413fd8eb8f11ec99757/example/thanos/prometheus-servicemonitor.yaml

[kolesnikovae]

Oh, I see - just to enable pyroscope server metrics scraping. Totally makes sense! What I was thinking about is kinda separate topic - perhaps we should revisit it again later.

[eh-am]

Something simple like this should do the work (using templates of course):

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: pyroscope
  namespace: monitoring
  labels:
     app.kubernetes.io/instance: pyroscope
     app.kubernetes.io/name: pyroscope
spec:
  selector:
    matchLabels:
      app.kubernetes.io/instance: pyroscope
      app.kubernetes.io/name: pyroscope
  endpoints:
  - port: http

I may create a PR later.