What Enters the Compute Units: The Matrix Math of an LLM

A large language model's computation reduces almost entirely to matrix multiplication, and a frontier accelerator's compute units do one thing: multiply tiles of two matrices and accumulate the result. This article is specific about what those two matrices are — the activations and the weights — how a transformer layer decomposes into a fixed set of such multiplications, and how the operands change across training, prefill, and decode.

An LLM is a stack of matrix multiplications

A large language model computes by repeating one structure many times. Input tokens are converted to vectors by an embedding table, and that sequence of vectors then passes through a stack of identical transformer layers — typically tens to over a hundred of them. Each layer performs two blocks of work, attention and a feed-forward network, and produces a refined sequence of vectors for the next layer. After the final layer, a last matrix multiply against the vocabulary produces the logits from which the next token is drawn.

Almost all of the arithmetic in that process is matrix multiplication. The normalization, the softmax inside attention, and the activation function inside the feed-forward network are real operations, but they are cheap, and they run on the vector units rather than the matrix engine. The matrix engine — the systolic array or tensor core — does one thing: it multiplies tiles of two matrices and accumulates the result. The question that determines everything about how an accelerator is loaded is therefore narrow and concrete: what are those two matrices?

The two operands: activations and weights

Every matrix multiply in an LLM combines two kinds of data, and they are fundamentally different.

Activations are the evolving numerical representation of the input. They begin as the embedded tokens and are transformed, layer by layer, into the model's internal state. Activations are dynamic: they depend entirely on the specific input, they change on every forward pass, and their size scales with the number of tokens being processed.

Weights are the learned parameters of the model. They are fixed once the model is trained, identical for every input, and they constitute the overwhelming majority of the data — hundreds of billions of values. During inference the weights are read but never changed; during training they are also updated.

With one exception, every matrix multiply in a transformer multiplies an activation matrix by a weight matrix. The exception is inside attention, where two activation matrices are multiplied together. The operands are typically held in a low-precision format — BF16, FP16, or increasingly FP8 — while the running sum inside the unit is accumulated in higher precision, usually FP32, to preserve numerical accuracy. What enters the compute units, then, is a stream of low-precision tiles of activation and weight matrices; what leaves is a stream of higher-precision partial sums.

One transformer layer, as operands

A single layer decomposes into a small, fixed set of matrix multiplications. The table below lists them for a representative large model with model dimension d = 12,288, feed-forward dimension d_ff = 49,152, processing a sequence of S tokens. Activation operands are shown in green, weight operands in amber.

Matrix multiply	What it computes	Operand A	Operand B	Output
Q, K, V projection	queries, keys, values	X [S×d]	WQKV [d×3d]	Q,K,V [S×3d]
Attention scores (per head)	token-to-token affinity	Q [S×dh]	KT [dh×S]	scores [S×S]
Attention × value (per head)	weighted sum of values	P [S×S]	V [S×dh]	O [S×dh]
Output projection	recombine heads	O [S×d]	WO [d×d]	Y [S×d]
FFN up-projection	expand to hidden dim	X [S×d]	W1 [d×d_ff]	H [S×d_ff]
FFN down-projection	contract to model dim	H [S×d_ff]	W2 [d_ff×d]	Y [S×d]

■ activation (dynamic, input-dependent)  ·  ■ weight (learned, fixed at inference)

Four of the six matrix multiplies pair an activation with a weight; the two inside attention pair two activations. Between them sit the cheap vector operations — RMSNorm before each block, the softmax that turns scores into the probabilities P, and the activation function inside the feed-forward network — none of which occupy the matrix engine. This six-matmul pattern, repeated once per layer and once per layer for every layer in the stack, is the entire compute load of an LLM.

The magnitudes

The weight operands dwarf the activation operands. In the model above, a single feed-forward weight matrix holds 12,288 × 49,152 ≈ 600 million parameters; one complete layer holds roughly 1.8 billion; the full model holds on the order of 175 billion. By contrast, the activation matrix for a 2,048-token prompt is 2,048 × 12,288 ≈ 25 million values. At two bytes each, that single feed-forward weight matrix is about 1.2 GB, against roughly 50 MB for the activations it multiplies.

This asymmetry is the central fact of LLM hardware. The arithmetic is dominated by the activation-times-weight matmuls, and the data movement is dominated by streaming the weights. A useful approximation is that a forward pass costs about 2N floating-point operations per token, where N is the parameter count — roughly 350 billion operations per token for a 175-billion-parameter model, all of it executed by the matrix engine as the multiplications in the table.

What changes between prefill and decode

The same matrix multiplies run during inference, but the shape of the activation operand changes, and with it the character of the computation. During prefill, the model processes all prompt tokens at once: the activation operand X is a full [S×d] matrix, and every matmul is a large matrix-by-matrix product (a GEMM) that keeps the compute units saturated. During decode, the model generates one token at a time: the activation operand collapses to a single row, [1×d], and the matmuls become matrix-by-vector products (GEMV). The weights, however, are unchanged — the entire [d×d_ff] matrix must still be read from memory to produce one token. Decode therefore moves the same weight bytes for a fraction of the arithmetic, which is why it is bound by memory bandwidth rather than compute. We treat this split in detail in the two jobs of the model.

Attention adds a second input during decode. To compute the scores for a new token, the model multiplies its single query against the keys of every preceding token, and then the resulting weights against every preceding value. Those accumulated keys and values are the KV cache, an activation tensor that grows by one entry per generated token and is read in full on every step. For the model above, the cache for a single 2,048-token sequence is on the order of 10 GB across all layers, and at scale it can exceed the size of the weights.

What training adds

Training runs the same forward matmuls and then adds two more categories of work. The backward pass propagates gradients in reverse, which requires two further matrix multiplies per weight matrix — one to compute the gradient of the activations and one to compute the gradient of the weights — so that the operands entering the units now include error signals as well as activations and weights. The backward pass roughly doubles the forward arithmetic. The optimizer step then updates each weight from its gradient; that update is elementwise, not a matrix multiply, and runs on the vector units. Training also processes large batches, so the activation operand carries a batch dimension in addition to the sequence dimension, enlarging every activation matrix accordingly.

How the operands reach the compute units

A compute unit never receives a whole matrix. A matrix multiply of [S×d] by [d×d_ff] is far larger than any array, so the compiler decomposes it into tiles sized to the hardware — for example, blocks of 128×128 — and feeds them in sequence. In the common weight-stationary arrangement, a tile of the weight matrix is loaded into the array and held, and tiles of the activation matrix are streamed through it, each operand reused across the array as it passes; the partial sums accumulate and are written out, and the next pair of tiles follows. The mechanics of that tile-level dataflow are described in what a systolic array is. What the compute units see, moment to moment, is not a model or a layer but an unbroken stream of low-precision operand tiles.

From operands to electrical behavior

That operand stream is also the chip's electrical load. Each tile that enters the array drives thousands of multiply-accumulate cells, and the rate and structure of the stream — set by the matmul shapes, the numeric precision, and the tiling — determine the current the power grid must supply. Lower precision packs more operations into each tile and raises current density; a sustained GEMM during prefill or training presents a high, steady draw; the matrix-by-vector pattern of decode presents a bursty, memory-paced one. The data flowing into the compute units and the current flowing out of the power grid are two descriptions of the same event, which is why the workload and the power delivery network must ultimately be analyzed together.