It is natural to assume an AI chip’s limit is how fast it can do math. For modern AI workloads — large language models in particular — that is almost never the bottleneck. The limit is memory: how fast and how much data the chip can move and hold. Compute scaled far faster than memory did, and inference mostly consists of moving data rather than computing on it, so the chip’s enormous arithmetic units sit idle waiting for weights and cache to arrive. This is the “memory wall,” and it shows up in five places at once: on-chip SRAM, cache, the KV cache, DRAM, and HBM.
The root cause: a forced trade-off
You can build memory that is fast or memory that is big — not both. So every chip uses a hierarchy, and each tier is a compromise between bandwidth, capacity, and latency.
| Tier | Bandwidth | Capacity | Where |
|---|---|---|---|
| On-chip SRAM (cache / scratchpad) | enormous (on-die) | tiny — MBs to tens of MB | on the die |
| HBM (stacked DRAM) | high — ~TB/s | moderate — tens to hundreds of GB | beside the die, on the interposer |
| DRAM (DDR) | lower — 100s of GB/s | large — TBs | off-package |
| Storage (NVMe) | low | huge | far off-package |
Why large models are memory-bound, not compute-bound
This is the part that surprises people. Four facts stack up.
1. The models don’t fit on-chip. A 70-billion-parameter model in 16-bit precision is about 140 GB just for the weights; frontier models run to terabytes. On-chip SRAM holds megabytes. So the weights must live in HBM or DRAM and be streamed to the compute units continuously.
2. Generation re-reads the entire model for every token. When a model produces text one token at a time (autoregressive decode, at small batch size), it has to read all of the weights from memory to compute one token — doing only a couple of operations per weight. The ratio of math to data moved — the arithmetic intensity (FLOPs per byte) — is tiny, so the chip operates far below its compute ceiling. The token rate is governed not by FLOPS but by:
A 140 GB model on 2 TB/s of memory needs ~70 ms just to read its weights once — a hard ceiling of ~14 tokens/sec, no matter how fast the math units are. Bandwidth literally caps the speed. This is the roofline picture:
3. The KV cache — the bottleneck unique to transformers. To generate each new token, an attention model needs the Key and Value vectors of every previous token. Recomputing them each step would be wasteful, so they are stored — the KV cache. Its size grows linearly with context length and with batch size (the number of concurrent sequences/users). With long contexts and many users, the KV cache can grow larger than the model weights themselves, and it hits both limits at once: it must be read every decode step (more bandwidth pressure) and stored (more capacity pressure).
4. Prefill and decode have opposite bottlenecks. Processing the prompt (prefill) batches all its tokens together, giving high arithmetic intensity — it is compute-bound. Generating the answer (decode) happens one token at a time — it is memory-bandwidth-bound. A single chip is therefore limited by different resources at different moments in the same request.
Why each tier is its own bottleneck
- On-chip SRAM / cache — the fastest memory there is, but tiny, and crucially its density has stopped scaling: at the leading nodes, SRAM barely shrinks generation to generation. You cannot simply add more on-chip memory; every extra megabyte costs compute area. That is a hard ceiling on the “keep it on the die” strategy.
- HBM — the workhorse compromise (high bandwidth plus useful capacity), but expensive, supply-constrained, and power-hungry, and still far slower than on-chip SRAM. HBM bandwidth is the single number that most determines decode speed today, which is why accelerators keep adding stacks.
- DRAM — large and cheap, but off-package, so high latency and limited bandwidth. Good for capacity overflow (offloading the KV cache), too slow to feed the compute units directly.
- The KV cache — not a memory type but the workload that breaks the others: it makes both capacity and bandwidth scale with how long and how many your conversations are.
The consequences ripple outward
Capacity forces the system topology. A model plus its KV cache that won’t fit in one chip’s memory must be split across many chips (tensor or pipeline parallelism). That turns the interconnect between chips into yet another bottleneck and adds latency. Memory capacity, in other words, decides how many chips a model needs and how they are wired together.
It is also an energy bottleneck. Moving a bit of data from external memory costs on the order of hundreds of times the energy of the arithmetic performed on it — an off-chip memory access is hundreds of picojoules; a single operation is a fraction of one. So “memory-bound” is also “data movement dominates the power budget.”
The connection to power. Because data movement dominates both time and energy, feeding and shuttling memory traffic is a large share of the current an AI chip actually draws — and those transfers are bursty and synchronized, which is exactly the kind of fast current swing that stresses power delivery. The memory bottleneck and the power-delivery problem are two views of the same fact: at this scale, moving data is the expensive part.
The bottom line
AI chips are memory-bound because the models are far larger than fast memory, and inference mostly moves data rather than computing on it. Token speed is gated by bandwidth; model and context size are gated by capacity; and the KV cache makes both scale with how long and how busy the workload is. Compute is rarely the thing you run out of first. Memory bandwidth, memory capacity, and the energy of moving data are — and designing around that, rather than around peak FLOPS, is what separates a fast AI system from a slow one.