MGMilind Gunjan / gunjanlabs

Lab

LLM inference, animated

Five live simulations of one GPU serving an LLM. Start with the whole GPU under live traffic, where every idea happens at once, then the four close-ups. Pause and step through any of them. The series explains each scene.

Illustrative model built with AI assistance; assumptions and numbers checked by me. Not a benchmark.

Start here · The whole GPU, live: one H100 serving Llama 3.1 8B

Requests arrive, wait for KV memory, have their prompt read in chunks, then decode one token per step. Every control below changes one of the five ideas. Watch which numbers move, and which one doesn't: GPU utilization.

Queue
0 waiting

KV memory: 62 GB left after the weights, drawn as 240 cells

Shared prefix, stored onceOne request's own KV (paler while its prompt is read)Free

Time per step, last 80 steps

Decode onlyStep also carried a prefill chunk

Bars top out at 100 ms. Every decoding user waits one whole step for their next token, so a tall bar is a stall for all of them.

–Time to first token (median)
–Inter-token latency (median)
–Throughput, all users
–KV memory used
–Users running
0Preemptions
–Prompt tokens from cache
–GPU “utilization”
t = 0.0 s
Try this, in order
  1. Leave the defaults. The red bars are steps that also read a prompt chunk: every decoding user stalls for them.
  2. Raise arrivals to 15 per second. The GPU can't read prompts as fast as they arrive: the queue and time to first token climb into seconds, KV fills and requests get preempted, while GPU “utilization” still reads 100%.
  3. Keep 15 per second and tick the shared prefix. Each prompt needs about 75% less prefill and much less KV: time to first token drops from seconds to tens of milliseconds and the queue drains.
  4. Untick it, set arrivals to about 10 and the answer to about 3,000 tokens. KV memory fills, new requests wait and running ones get preempted.
  5. Tick FP8 KV cache. Each token's KV halves, so about twice the users fit.
  6. Change the token budget. A bigger budget starts long prompts sooner but makes the red stalls taller.
  7. Tick disaggregation. The stalls vanish because prompts are read on another pool; time to first token now includes the KV transfer.

An illustrative model, not a benchmark: one H100 (80 GB, 3.35 TB/s, ~600 TFLOPS achieved for prefill), Llama 3.1 8B with FP16 weights (16 GB) and 128 KiB of KV per token (64 KiB in FP8). Step time = (weights + every decoding user's KV) ÷ bandwidth + prefill math. A request reserves its prompt's KV when admitted and grows by one token per step; when memory runs out the newest request is preempted and recomputed later. With disaggregation the prefill pool is assumed big enough.

Takeaway: one GPU, five ideas. Decode re-reads the weights every step, batching shares that read, each user's KV grows in the same memory, prefill is a compute burst that stalls decode unless it's chunked or moved, and cost is GPU-hours over tokens. Utilization reads 100% throughout, so watch queue depth, KV usage, first-token time and inter-token latency instead.

1 · One request: queue, prefill, then one token at a time

Prefill reads the whole prompt in one parallel burst and fills the KV cache. Decode then produces one token per step, reading all the weights again each time and adding one KV entry.

QueuePrefillDecode loopDone
Prompt tokens
Model layers
KV cache
Output stream
–Time to first token
–Time per output token
0KV entries held

Timings are illustrative: one prefill burst for the 12-token prompt, then a fixed time per decode step.

Takeaway: prefill is one compute-heavy burst that sets time to first token. Decode re-reads every weight for each token, so it's memory-bound and sets time per output token. The KV cache grows by one entry per token, which is what eventually limits how many users fit.

2 · Static vs. continuous batching on the same 6 GPU slots

The same queue of requests, with different answer lengths, runs on both sides. Static batching waits for the longest answer before refilling. Continuous batching refills a slot the step after it frees up.

Static batching

Continuous batching

0Requests finished
0%Slot utilization
0Requests finished
0%Slot utilization
step 0

Takeaway: decode reads the weights once per step no matter how many users are in the batch, so every idle slot is wasted bandwidth. Continuous batching refills slots every step, which is why it's the default in vLLM, TensorRT-LLM and SGLang.

3 · Speculative decoding: a small model guesses, the big model checks

A draft model proposes 4 tokens. The big model verifies all 4 in one pass, which costs about the same as one normal step. Accepted guesses are kept. At the first wrong guess the big model substitutes its own token, so the output is identical to running the big model alone.

Draft proposes
Output so far
0Big-model passes
0Tokens produced
–Tokens per pass (normal = 1.0)

Raise the acceptance rate to see why a draft model that matches your traffic well gets more tokens out of every big-model pass.

Takeaway: decode leaves compute idle. Speculative decoding spends it verifying several cheap guesses per expensive step. Same output, fewer big-model steps, and the gain depends on the acceptance rate.

4 · GPU memory: weights first, then KV cache decides how many users fit

Pick a GPU, a model and a context length. The weights load once; every concurrent user needs their own KV cache. Whatever memory is left sets the maximum batch.

WeightsRuntime overhead (~2 GB)KV cache in useFree
–Max concurrent users
–KV cache per user
–Fits on one GPU?

KV per token = 2 × layers × KV heads × head size × bytes: ~128 KiB for Llama 3.1 8B and ~320 KiB for 70B, with an FP16 cache. Estimates, not benchmarks.

Takeaway: model size sets the memory, which sets GPUs per replica. Memory left after weights sets how many users one replica holds. That's why FP8, a shorter max context or a bigger GPU raises concurrency.