What Is a Systolic Array?

A systolic array is a grid of identical processing elements through which data is pumped one step per clock cycle, with each element performing a single multiply-accumulate as the data passes. It is the matrix engine at the heart of AI accelerators — TPUs, tensor cores, and NPUs — because it turns dense matrix multiplication into a highly regular, memory-efficient operation. This article explains how it works, with diagrams.

The structure: a grid of processing elements

A systolic array is a regular grid of identical processing elements (PEs), each connected only to its immediate neighbors. There is no global bus and no shared memory inside the array; a PE communicates only with the cells directly adjacent to it. Computation proceeds by passing data from one PE to the next on every clock cycle — the rhythmic, pulse-like dataflow from which the architecture takes its name (Kung and Leiserson, 1978; "systole" is the contraction phase of the heartbeat).

The processing element

Every PE performs the same primitive operation: a single multiply-accumulate. In the common weight-stationary arrangement, each PE holds one weight value for the duration of the computation. On each cycle it multiplies the activation arriving from its left by that stored weight, adds the product to the partial sum arriving from above, forwards the activation unchanged to the PE on its right, and forwards the updated partial sum to the PE below. That is the entire behavior — and because every PE does exactly this, the array is built by tiling one simple cell.

Dataflow: data pumped through in rhythm

The power of the structure is in how data moves. The weights are loaded once and held in place. The activations stream in from the left edge and propagate rightward through the array. The partial sums begin at zero along the top edge and accumulate as they descend, emerging from the bottom edge as the finished output columns. Each value is fed in once and then reused by every PE it passes through.

Because a value advances exactly one PE per cycle, the set of currently active cells forms a diagonal wavefront that sweeps across the array as the computation proceeds. Inputs are deliberately staggered in time at the edges so that each activation and its corresponding partial sum meet in the right PE at the right cycle.

Mapping a matrix multiply

For a matrix product C = A × W, the weight matrix W is mapped onto the array one element per PE, and the rows of A are streamed in from the left. As the activations flow through and the partial sums accumulate down the columns, each output element c_ij is computed as the dot product of a row of A with a column of W, assembled across the cells it traverses. After an initial fill latency while the first wavefront propagates in, the array produces a full row of results on every cycle: high latency to first result, but high sustained throughput thereafter.

Why it is efficient

The systolic array exists to solve a memory-bandwidth problem. A naive matrix multiply re-reads operands from memory for every multiply-accumulate, and at the scale of modern models that memory traffic, not the arithmetic, is the bottleneck. The array reads each operand from memory once and then reuses it across an entire row or column of PEs as it streams through. This converts a memory-bandwidth-bound operation into a compute-bound one and raises arithmetic intensity — the ratio of operations performed to bytes moved — by orders of magnitude, with almost no control logic, since the only instruction is "multiply-accumulate and pass along."

Dataflow variants

The example above is weight-stationary, the arrangement used in the Google TPU. Other mappings hold a different operand in place: output-stationary designs keep each accumulating result in one PE while both inputs stream through; row- or input-stationary designs hold the activations. The trade-offs differ — which data is reused locally, how partial sums are reduced, how operands are staged — but the underlying structure is the same: a mesh of identical multiply-accumulate cells through which data is pumped in lockstep.

Where systolic arrays are used

The systolic array is the matrix engine at the center of modern AI silicon. A Google TPU matrix unit is a 128×128 array — 16,384 multiply-accumulate units operating together. NVIDIA tensor cores and dedicated neural processing units apply the same principle. Because these arrays are large, regular, and clocked together, their electrical behavior is as distinctive as their computational behavior: thousands of identical cells switching in lockstep present a coherent, simultaneous load to the power grid, which we examine in simultaneous switching in AI compute.