MGMilind Gunjan / gunjanlabs

LLM inference, explained · Part 2 of 5 · Nov 22, 2026 · 1 min read

Prefill vs. decode

Every LLM request has two phases, and they are bottlenecked by completely different things.

Every LLM request has two phases, and they are bottlenecked by completely different things. Scene 1 in the lab walks one request through both.

Prefill reads your whole prompt at once. All the tokens go through the model in parallel, so it is a compute-heavy burst. It decides time to first token (TTFT).

Decode then writes the answer one token at a time. For every single token, the GPU re-reads all of the model's weights from memory. That makes decode memory-bandwidth bound, and it decides inter-token latency (ITL): how fast the words stream.

Rough numbers

From my model of Llama 3.1 8B on one H100:

  • Reading 16 GB of weights at 3.35 TB/s takes about 4.8 ms. That is the floor for every decode step, even for a single user.
  • Prefilling a 2,000-token prompt takes about 50 ms of compute at the throughput I assumed (~600 TFLOPS achieved).

Why it matters

The fixes are different. Long prompts and slow first tokens point at prefill: prefix caching, chunking, more compute. Slow streaming points at decode: batching, quantization, speculative decoding, more memory bandwidth.

If a team reports "latency" as one number, ask which one.

Next: why batching is the biggest throughput lever in decode.

Illustrative model built with AI assistance; assumptions and numbers checked by me. Not a benchmark. See this in the lab →

Next: Continuous batching →