The Two Jobs of the Model: Prefill and Decode

A large language model serving a request is not running one computation. It is running two — and they could hardly be more different. The first reads the prompt; the second writes the answer. One is a compute monster, the other a memory glutton. Cerebras frames this as the two jobs of the model, and it is the single most useful lens for understanding modern inference hardware. It also produces two completely different signatures at the power rail.

One request, two computations

When you send a prompt to a language model and watch the reply stream back, two distinct things happen in sequence. First the model has to read and understand everything you wrote — the system prompt, the conversation history, your question. Then, only once it has digested all of that, it begins to write, producing the answer one word-piece at a time. These two activities are not two settings of the same dial. They are different algorithms running on the same weights, and they place opposite demands on the silicon. The industry calls them prefill and decode.

The reason this distinction matters — and the reason Cerebras and others keep returning to it — is that almost every important property of an inference system follows from it. How fast the first word appears, how fast the rest stream, how many users a chip can serve at once, how much memory bandwidth versus raw arithmetic a chip should have, even whether a single chip should handle a request at all: all of it traces back to the split between prefill and decode.

Prefill: reading the prompt

Prefill is the model reading your input. Crucially, every token in the prompt is known up front, so the model can process all of them at once. It computes the key and value vectors for every input position in parallel and runs the prompt through every layer as a set of large, dense matrix multiplications. There is no waiting on a previous step, because nothing in the prompt depends on a token the model has not yet seen. The whole prompt is laid out and crunched in one parallel sweep.

That makes prefill compute-bound. It is limited by how fast the chip can do floating-point math, not by how fast it can move data. The tensor cores run near saturation; the arithmetic intensity — the number of math operations performed per byte read from memory — is high, on the order of hundreds of operations per byte. The attention computation grows with the square of the prompt length, so a prompt four times longer costs roughly sixteen times the attention work. In wall-clock terms, prefilling a long prompt of several thousand tokens takes on the order of a few hundred milliseconds on a current high-end GPU. This single number is essentially your time to first token: the model cannot say anything until prefill is done.

Prefill also produces an artifact the next phase depends on entirely: the KV cache. As it processes the prompt, the model stores the key and value vectors for every layer and every position. Those stored vectors are what let the model, during generation, attend back over everything it has already seen without recomputing it. Prefill, in other words, is not only reading the prompt — it is building the working memory the rest of the request will run on.

Decode: writing the answer

Decode is the model writing its reply, and it is fundamentally serial. Each new token depends on the token before it, because the model feeds its own last output back in as the next input. You cannot generate the tenth word until you have generated the ninth. So instead of one big parallel sweep, decode is a long sequence of tiny steps — one forward pass through the entire network per output token.

The arithmetic in each of those steps is small. The model is multiplying its weight matrices by a single new vector, not by a whole batch of prompt tokens — a matrix–vector product rather than a matrix–matrix product. But to do it, the chip must read the entire model's weights out of memory, plus the entire KV cache, on every single step. The math is trivial; the data movement is enormous. That makes decode memory-bandwidth-bound: it is limited not by how fast the chip can compute but by how fast it can stream weights and cache out of high-bandwidth memory. Arithmetic intensity collapses to roughly one operation per byte, and the expensive tensor cores sit mostly idle, waiting on memory.

Because each token requires reloading the model, decode speed is measured in tokens per second — typically tens to low hundreds per second per request on a high-end GPU — and the time per output token defines the streaming experience the user actually feels. Decode is almost always the bottleneck: a request spends a few hundred milliseconds in prefill and then potentially many seconds in decode, generating a long answer one memory-bound step at a time. When people complain that a model "feels slow," they are almost always describing decode.

The KV cache is the hand-off

The KV cache is the thread that ties the two phases together — and a major reason decode is so memory-hungry. Prefill builds it; decode reads all of it on every step and then appends one new entry for the token it just produced. So the cache starts at the size of the prompt and grows linearly with the length of the answer. Across many concurrent users generating long responses, the aggregate KV cache can swell to several times the size of the model's own weights. Every decode step has to stream that growing structure out of memory, which is exactly why the phase is bandwidth-limited and why memory capacity, not compute, is what usually caps how many requests a chip can serve at once.

Why the two jobs split the hardware

Plot the two phases on a roofline — performance against arithmetic intensity — and they land on opposite sides of the ridge. Prefill sits in the compute-bound region, where the answer is "buy more FLOPS." Decode sits deep in the memory-bound region, where the answer is "buy more bandwidth." No single, fixed ratio of compute to bandwidth is optimal for both. A chip sized for prefill wastes bandwidth during decode; a chip sized for decode wastes compute during prefill.

This is why modern inference is increasingly disaggregated: prefill and decode are run on different machines, tuned for their respective bottlenecks. A request flows through a compute-rich accelerator for prefill, the resulting KV cache is handed to a bandwidth-rich accelerator for decode, and tokens stream back from there. It is also the architectural argument behind wafer-scale parts like the Cerebras WSE, whose enormous on-wafer SRAM bandwidth is aimed squarely at the memory-bound decode phase that conventional GPUs struggle to feed. Once you see inference as two jobs rather than one, splitting the hardware to match stops looking exotic and starts looking inevitable.

Two jobs, two power signatures

Here is where the split becomes a power-integrity problem — and where it concerns us at Anasim. The two jobs do not only differ in throughput. They draw current in two completely different shapes, and the power delivery network has to absorb both.

Prefill, with its saturated tensor cores, presents a sustained high-current plateau. The rails are loaded near their maximum for the full duration of the prefill, and the dominant concerns are thermal and static — average power, IR drop across the grid, and a deep but relatively steady supply sag. This regime looks electrically like training: demanding, but quasi-steady and characterizable.

Decode is the opposite. Average power is lower, because the tensor cores are mostly idle waiting on memory — but the current is bursty and cadenced, one compute pulse per generated token interleaved with memory-bound stalls. That structure is rich in exactly the thing power networks fear: rapid current swings, or high di/dt. Each token is a small load step; the stream of tokens is a periodic excitation. If the token cadence or its harmonics line up with a package–chip resonance of the PDN, the droop does not just repeat — it can build, the constructive interference we have elsewhere called a rogue wave.

And between the two jobs sits the single largest transient of the request: the prefill-to-decode transition. The chip drops from compute-saturated to memory-bound almost instantaneously — a large, fast step-down in current that the PDN must catch without letting the supply overshoot or ring. At sub-volt rails, where 30 to 50 millivolts of droop is enough to clip frequency or violate timing, this hand-off is a recurring stress event, repeated for every request, millions of times a day across a fleet.

What it means for inference-silicon design

The two-jobs framing carries a design consequence that static power analysis cannot express. The relevant question is not "what is the single worst-case current this chip can draw?" It is "what does the current waveform look like across a realistic prefill, the transition, and a long decode — and how does this PDN respond to that waveform?" A grid that comfortably passes worst-case static IR-drop sign-off can still fail under the specific resonant excitation that a decode token-stream produces, or undershoot on the prefill-to-decode step.

Disaggregation sharpens the point rather than dissolving it. A prefill-optimized part lives near a sustained current ceiling and should be designed for thermal limits and steady droop; a decode-optimized part lives in a world of fast, repetitive load steps and should be designed for transient response and resonance control. The two have different PDN design centers — which is one more reason the workloads are being pulled onto different silicon, and one more reason each needs to be simulated against its own realistic current profile rather than a shared synthetic worst case.

That is precisely the analysis PDNLab is built for: driving a full-stack, board-to-die power-delivery model with the actual prefill and decode current signatures — the plateau, the transition, the token-cadenced pulses — and reporting where and when the supply droops as a spatiotemporal surface, not a scalar. Two jobs, two signatures, one power network that has to survive both. Understanding the split is the first step; modeling its electrical consequence is the next.