Measuring Actual Memory Requirements for a Growing Service

[sevenjay]

Our service is continuously growing, and we will need to increase memory. According to this article https://qdrant.tech/articles/memory-consumption/ on memory consumption, there is a significant gap between the memory usage displayed and the actual memory required until the system becomes unusable. Is there any other method to measure the actual memory requirements to ensure we have sufficient memory available, rather than experimenting until the system becomes unusable every time?

[timvisee]

Not really, calculating memory requires is an extremely hard and sometimes impossible problem.

Could you tell a bit about your current configuration. If you're not using memory mapping at all, the current memory usage should give good insight in required memory.

If you are using memory mapping, this becomes a bit harder. You can still check how much memory is currently consumed, physically and by your memory maps. But the question is what percentage of physical memory you'd want to have. This choice totally depends on your query times, and it's something you'd have to benchmark yourself. You'd likely be able to go as low as 10% physical memory, at the cost of search times.

I hope that helps pointing you in the right direction somewhat.