If batching is where throughput comes from, memory decides how big the batch can get.
GPU memory gets split two ways. The model weights load once. Then every concurrent user needs their own KV cache: the keys and values for every token in their conversation, so the model never has to recompute them.
KV per token is simple arithmetic
2 × layers × KV heads × head size × bytes. For Llama 3.1 8B in FP16:
2 × 32 × 8 × 128 × 2 = 128 KiB per token.
On one 80 GB H100
Scene 4 in the lab lets you change each of these:
- Weights take 16 GB, and I allow ~2 GB for runtime overhead, leaving ~62 GB.
- At 8k tokens of context, each user's KV cache is about 1.07 GB.
- So about 57 users fit at once. Store the KV cache in FP8 and it is about 115.
- Switch to Llama 3.1 70B with FP8 weights and only ~8 GB is left: room for about 2 users at 8k. That is why bigger models get split across GPUs.
When memory runs out, the server stops admitting new requests, and under pressure it preempts running ones and recomputes them later. Users feel that as a stalled answer.
The levers
A shorter maximum context, an FP8 KV cache, prefix caching so a shared system prompt is stored once, or more memory per replica.