You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
One of my tenants has been increasing the number of streams, which has led to a noticeable increase in the loki_ingester_memory_streams metric. and This increase has also impacted some other tenants, causing them to receive 429 (Too Many Requests) errors.
All these tenants are coming from a single log shipper, so they share the same ingestion pipeline.
To mitigate the issue, I increased the max_global_streams_per_user limit, which has partially fixed the problem.
However, my concern is that the limit is calculated as follows:
(global limit / number of ingesters) * replication factor
My question is:
Is this calculation applied globally across all tenants in Loki’s ingester, or is it specific to each individual tenant across all ingesters?
In other words, does this calculation consider the global limit for all tenants, or does it treat each tenant separately when distributing the limit across ingesters?
i saw this config before
# Maximum number of active streams per user, per ingester. 0 to disable.
# CLI flag: -ingester.max-streams-per-user
[max_streams_per_user: <int> | default = 0]
# Maximum number of active streams per user, across the cluster. 0 to disable.
# When the global limit is enabled, each ingester is configured with a dynamic
# local limit based on the replication factor and the current number of healthy
# ingesters, and is kept updated whenever the number of ingesters change.
# CLI flag: -ingester.max-global-streams-per-user
[max_global_streams_per_user: <int> | default = 5000]
but i do not understand it. I would appreciate it ifyou could help me figure this out.
What I expected:
I expected that the tenant that increases the streams would be the one limited, and not other tenants. Ideally, the increase in streams for one tenant should only impact that tenant, rather than affecting all tenants using the same log shipper.
The text was updated successfully, but these errors were encountered:
One of my tenants has been increasing the number of streams, which has led to a noticeable increase in the loki_ingester_memory_streams metric. and This increase has also impacted some other tenants, causing them to receive 429 (Too Many Requests) errors.
All these tenants are coming from a single log shipper, so they share the same ingestion pipeline.
To mitigate the issue, I increased the
max_global_streams_per_user
limit, which has partially fixed the problem.However, my concern is that the limit is calculated as follows:
My question is:
Is this calculation applied globally across all tenants in Loki’s ingester, or is it specific to each individual tenant across all ingesters?
In other words, does this calculation consider the global limit for all tenants, or does it treat each tenant separately when distributing the limit across ingesters?
i saw this config before
but i do not understand it. I would appreciate it ifyou could help me figure this out.
What I expected:
I expected that the tenant that increases the streams would be the one limited, and not other tenants. Ideally, the increase in streams for one tenant should only impact that tenant, rather than affecting all tenants using the same log shipper.
The text was updated successfully, but these errors were encountered: