-
Notifications
You must be signed in to change notification settings - Fork 3.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Prometheus] Reduce number of series for per-exchange/per-queue metrics #11217
Comments
We discuss this every now and then and overall I think we agree - we expose too many metrics and the high-cardinality metrics shouldn't be there at all probably. As always, it's a matter of time and effort required to clean this up. Perhaps you'd be interested in contributing this? As a start, I'd say the exact list of changes would be nice. Your overall description is good, but I'm thinking more about a TODO list:
This would allow us to discuss individual changes if needed and would guide the implmentation of these changes. |
As far as I'm concerned, the only metric label that causes issues here is Alternatively, metrics that make sense without this label (see below) could be exposed under another name without the In some cases (like people creating temporary queues with UUIDs in the name or something) other metrics could cause high cardinality as well, but in that case, those metrics can be easily dropped in Prometheus using (Documentation for reference: https://www.rabbitmq.com/docs/prometheus#detailed-endpoint) Metrics including
Metrics including
|
Thanks. As I said - I think it's a valid request and it concurs our own experience. Therefore, I will leave this open but note that we are not actively working on this at this time. Contributions would be very much appreciated. It's mostly about deleting stuff, so shouldn't be too hard. :) |
If I was an Erlang dev I'd be happy to contribute code, but getting used to Erlang and setting up a dev environment to test the changes is too much for me right now I think. Maybe I'll get annoyed enough at the lack of metrics I'll take a look at it some day 😅 |
Coming up with a list of specific actionable changes (like in #11217 (comment) but I assume the list is not necessarily complete) is considered to be a partial contribution by our team. Those seven is a good starting point. Well, at least the first four, I am less certain about removing entire metrics as it is always particularly hard to remove features/data. But for 4.x, that's certainly a good idea. |
My suggestion to make this a non-breaking change would be to add these metrics under new names (as all of them except
this doesn't cause name collisions when exporting all metrics and in the 4.x release, all per-channel metrics could be removed. |
I can take care of this issue. I'll be waiting to hear core dev team opinion on whether the strategy suggested on #11217 (comment) should be taken. |
It sounds like a good approach to me |
#11559 needs another around: the tests were failing often and were not excluded from mixed version runs that they cannot pass (with the assertions they have against a 3.13.x node). |
Is your feature request related to a problem? Please describe.
As of now, a lot of useful per-exchange/per-queue-metrics (for example the number of unroutable messages, number of delivered messages, etc.) are only included in the
channel_exchange_metrics
/channel_queue_metrics
/channel_queue_exchange_metrics
, which can't be easily used with Prometheus because of their high cardinality.Practically, all per-channel-metrics (including
channel_exchange_metrics
andchannel_queue_metrics
andchannel_queue_exchange_metrics
) can't be used with Prometheus, as they include the channel ID as a label value which is something Prometheus explicitly warns against, because it causes a lot of active time series, which are, by default, limited to around 200.000 and causes high memory usage.Combined with the lack of aggregation features in
metrics_relabel_config
in Prometheus, this forces users to disable useful metrics like messages delivered per queue (as included inchannel_queue_metrics
), messages published/confirmed/unroutable per exchange (as included inchannel_exchange_metrics
) or messages published from exchange to queue (as included inchannel_queue_exchange_metrics
).Describe the solution you'd like
Ideally, there would be groups group for queue, exchange and queue-exchange metrics, not including the channel id as a label value.
Queues even have the group
queue_metrics
already, which, however, does not include metrics like the number of delivered/acked messages.These new groups should just expose the (relevant) metrics of
channel_exchange_metrics
,channel_queue_metrics
andchannel_queue_exchange_metrics
but aggregated only by queue/exchange instead of queue+channel/exchange+channel).Describe alternatives you've considered
No response
Additional context
No response
The text was updated successfully, but these errors were encountered: