Can someone explain to me, in detail, on what llama.cpp allocates memory? #9936
Replies: 1 comment 8 replies
-
This is the model weights. The total size should be very close to the size of the model file on disk in most cases.
This is the model context. Parameters such as
This is a small buffer used to store the results of the computations, usually it is too small to worry about.
This is the buffer where the intermediate results of computations and other temporary data used during inference is stored. Parameters such as |
Beta Was this translation helpful? Give feedback.
-
|
The backend may need to allocate some additional memory, but it shouldn't grow significantly between evaluations. If you think that may be a bug, open an bug report with all the necessary details to reproduce the issue. |
Beta Was this translation helpful? Give feedback.
-
I don't know what is considered to be normal or a significant growth. The following table shows the memory increase post model initialization per evaluation that I can observe with a
Edit: further investigation shows that this seems to be the cost of the computation graph, meaning that this is normal? If so how can I calculate the maximum amount of memory the computation graph will consume for a given context size? |
Beta Was this translation helpful? Give feedback.
-
I can replicate this behavior during CPU-only inference and to me it seems likely to be a bug, because I'm getting these "false positive" error messages (with no server crashes and with correct responses) like the ones below, even when the model avoids using the GPU according to Interestingly, the available GPU memory is being measured correctly even when inference is done entirely in the CPU, because reducing context window size (here from 164K to 32K tokens) to reduce KV cache size so that it would fit GPU memory (if GPU was used) gets rid of those "false positive" error messages: |
Beta Was this translation helpful? Give feedback.
-
|
The bug might have remained undiscovered because few users do truly CPU-only inference on GPU-enabled machines / in GPU-enabled containers, given that those who attempt it in such conditions would just just set So the conditions required for replication are pretty strict:
|
Beta Was this translation helpful? Give feedback.
-
|
@tsurai this is probably caused because the CUDA/HIP backend converts some of the attention matrices to F16 for performance, and these matrices become bigger as the used context increases. We should consider not doing that anymore and take the performance hit. This is especially bad with ROCm because it does not support the VMM allocator, so instead of growing a buffer it keeps allocating new buffers. Please open an issue to keep track of this. @mirekphd please open an issue with instructions to reproduce this. Both of these issues could be also resolved by using flash attention. |
Beta Was this translation helpful? Give feedback.
-
Hello everyone,
I recently asked a question about llama.cpp memory usage. I recieved some helpful answers, one of which suggested looking at the outputs I get when I start llama-server.
From looking at the output of starting llama-server, I see that KV buffer size is around 1.3 GB. This led me to a question: what is the KV buffer? It reminded me of key-query-value in attention, but I'm not sure if this is it.
I will now paste my output log below. I would really appreciate it if someone could go over it block-by-block (or line-by-line if necessary) to explain the memory allocations of llama.cpp and what they relate to in the LLM.
Thank you in advance!
Beta Was this translation helpful? Give feedback.
All reactions