More GPUs don't always mean more usable memory
A workload hit an out-of-memory error. Doubling the GPUs would not have fixed it. Here is why, and what did.
More GPUs don't always mean more usable GPU memory. We ran into a clean example of this recently: an inference workload hit an out-of-memory error. The interesting part was not how much memory was in use. It was what was using it, and whether that memory could be spread across GPUs at all.
Each GPU is a shelf
Think of each GPU as a shelf with, say, about 32 GB of space. A large language model needs that space for three things:
- Weights
The model itself.
- KV cache
Memory that grows as the context gets longer.
- Working
Temporary memory for the calculations made while the model runs.
What tensor parallelism changes
Tensor parallelism (TP) splits the model across GPUs. With TP4 the model is distributed over four GPUs, with TP8 over eight. Each GPU holds a smaller piece of the weights, and memory that can be distributed, like the KV cache, scales across more GPUs too.
But not every calculation is distributed this way.
The flag that decides the split
This is where a flag like --mem-fraction-staticmatters. It tells SGLang how much of each GPU's shelf to reserve for the model and the KV cache. At 0.85, roughly 85% of each GPU is reserved. At 0.4, roughly 40%.
The key detail is that this is per GPU. It is not 40% of the combined memory of all four or eight GPUs. Whatever is left over on each card is what temporary memory has to fit into.
What actually ran out
In our case, the evaluation needed a temporary logits tensor of about 1.9 GB. On that code path the tensor is not split across GPUs, so every GPU needs roughly 1.9 GB of free space on its own.
| --mem-fraction-static | Free memory per GPU |
|---|---|
| 0.85 reserved (Out of memory) | ~1.7 GB |
| 0.4 reserved (Fits) | ~19 GB |
Moving from TP4 to TP8 would not have helped. More GPUs add total capacity, but they do not add temporary memory per GPU, and that was the thing running out.
Where more GPUs do help
The picture changes with long-context inference. The KV cache grows with context length, and it can be distributed, so TP8 can meaningfully increase how much context you can serve.
Inference optimization isn't just “use more GPUs.”
You need to know what is consuming memory, where it lives, and how it scales. That is where the real optimization happens, and it is the kind of detail our campaigns are built to find on your workload instead of a generic one.