Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Understanding the Calculation of Stream Limits in Loki Ingesters #16386

Open
LinTechSo opened this issue Feb 20, 2025 · 0 comments
Open

Understanding the Calculation of Stream Limits in Loki Ingesters #16386

LinTechSo opened this issue Feb 20, 2025 · 0 comments

Comments

@LinTechSo
Copy link
Contributor

LinTechSo commented Feb 20, 2025

One of my tenants has been increasing the number of streams, which has led to a noticeable increase in the loki_ingester_memory_streams metric. and This increase has also impacted some other tenants, causing them to receive 429 (Too Many Requests) errors.

All these tenants are coming from a single log shipper, so they share the same ingestion pipeline.

To mitigate the issue, I increased the max_global_streams_per_user limit, which has partially fixed the problem.
However, my concern is that the limit is calculated as follows:

(global limit / number of ingesters) * replication factor

My question is:
Is this calculation applied globally across all tenants in Loki’s ingester, or is it specific to each individual tenant across all ingesters?

In other words, does this calculation consider the global limit for all tenants, or does it treat each tenant separately when distributing the limit across ingesters?

i saw this config before

# Maximum number of active streams per user, per ingester. 0 to disable.
# CLI flag: -ingester.max-streams-per-user
[max_streams_per_user: <int> | default = 0]

# Maximum number of active streams per user, across the cluster. 0 to disable.
# When the global limit is enabled, each ingester is configured with a dynamic
# local limit based on the replication factor and the current number of healthy
# ingesters, and is kept updated whenever the number of ingesters change.
# CLI flag: -ingester.max-global-streams-per-user
[max_global_streams_per_user: <int> | default = 5000]

but i do not understand it. I would appreciate it ifyou could help me figure this out.

What I expected:
I expected that the tenant that increases the streams would be the one limited, and not other tenants. Ideally, the increase in streams for one tenant should only impact that tenant, rather than affecting all tenants using the same log shipper.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant