notebook № 046 plates · 20 min to read, longer to play
CPU vs GPU vs TPU, and how they work together
On three bets and one roof
Three chips made of the same silicon, each betting its area on a different idea of what “work” looks like. This notebook races them, puts them under one roofline, and shows why a real system needs all three talking to each other.
plate I · area
Three bets on the same silicon
A chip designer has a fixed budget: so many square millimetres, so many watts. Every transistor spent on one thing is a transistor not spent on another. Look at where the area goes and you can read a chip’s whole philosophy.
The CPU bets that the next instruction matters most. It spends lavishly on control (branch predictors, out-of-order machinery) and on caches, so that one thread almost never waits. The GPU bets on crowds: strip each core down to a simple lane, put thousands on the die, and hide any one thread’s waiting behind all the others. The TPU bets that the work is matrix multiplication, almost always. It spends most of its area on two giant multiply-accumulate grids and lets a compiler plan everything else in advance.
Switch between them and hover the regions. The floorplans are stylised, nobody’s real die, but the proportions tell the true story.
plate II · the race
Same job, three runners
“Which chip is fastest?” has no answer until you say at what. So let’s race them. Four workloads, the same three chips, and the winner changes.
Watch the little grid next to each lane: those are the chip’s workers, and the bright ones are busy. That is the whole lesson in one picture. A GPU with one busy lane out of thousands is a very expensive, rather slow CPU. A CPU with every core busy on a matrix multiply is doing its best with a hundredth of the multipliers.
The interesting runs are the ones where the “slow” chip wins. A long chain of dependent steps can’t be split, and a branchy parser fights every lane that wants to march in step. There, the CPU’s years of cleverness about a single thread are exactly what you want.
Run the dependent chain, then the big matmul. Same three chips, opposite podiums. What changed is the shape of the work, not the hardware.
plate III · one roof each
The roofline
The race hides a single number that explains most of it: arithmetic intensity, the operations a workload performs for every byte it pulls from memory. Adding two arrays does one addition per twelve bytes. Multiplying two big matrices reuses every byte thousands of times.
Every chip then has two limits. It can’t compute faster than its peak, the flat roof. And it can’t compute faster than memory feeds it: bandwidth × intensity, the slanted roof. Where the two meet is the ridge point. Left of it, the chip is waiting on memory; right of it, on arithmetic. Drag across the chart, or pick a workload.
Two things jump out. First, at low intensity the expensive accelerators are barely ahead: everyone is waiting on memory, and the winner is whoever has the widest pipe. Second, the accelerators’ ridge points sit far to the right. To use a GPU or TPU fully, a workload must do hundreds of operations per byte. Most code doesn’t.
Compare LLM decode with batched decode. Same model, same weights. Serving one user reads every weight to do one multiply-add with it; serving 64 users at once reads it once and uses it 64 times. That one slide to the right is most of the economics of serving language models, and a large part of what an AI bill is actually paying for.
Find the intensity where the CPU stops being memory-bound, then the one where the GPU does. The gap between those two ridges is the zone where an accelerator is expensive and half-asleep.
plate IV · cooperation
One training step, three chips
An accelerator never works alone. It has no file system, no network stack, no idea what a JPEG is. Somewhere there is a host CPU reading training data from disk, decoding and augmenting it, copying it across a link into the accelerator’s memory, and launching the work. The GPU or TPU then runs the forward and backward pass, and waits for the next batch.
That last word is the expensive one. Start with everything happening one after another and a single data-loader process, and watch the accelerator lane: the rose stretches are the world’s most expensive chip doing nothing. Then turn on prefetching, add loader workers, try a bigger batch.
Overlap is the big win: the CPU prepares batch n + 1 while the accelerator computes batch n, so the slowest stage sets the pace instead of the sum of all three. Bigger batches help too, because the fixed cost per step (launching kernels, synchronising, updating the optimiser) gets spread over more samples. They also need more accelerator memory, and past a point they change how the model learns, which is why batch size is a modelling decision as much as a performance one.
When I look at an AI infrastructure bill, this is the plate I keep in my head. Paying for an accelerator by the hour and running it at a third of its capacity because the data pipeline can’t keep up is one of the most common, and most fixable, ways to waste money in the whole field.
plate V · transport
The cost of moving
Plate IV had a link in the middle, and links are where heterogeneous systems leak time. Data on the chip moves at tens of terabytes a second. Across to the HBM beside it, a few. Over PCIe to the host CPU, a few dozen gigabytes: a hundred times slower than the accelerator’s own memory. Across the network to another machine, slower again.
Pick a payload and look at the right-hand column: how many big matrix multiplies a top accelerator could have finished in the time it took to move it. Shipping a model’s weights over PCIe costs seconds; that is why weights are loaded once and kept resident, and why “send it to the GPU, get the answer back” only pays when the work is large compared with the trip.
It is also why the biggest systems are wired so carefully. TPU pods and GPU clusters spend enormous engineering on chip-to-chip links, because once a model is split across hundreds of chips, the links become the computer.
Find the smallest payload where the CPU ⇄ accelerator trip takes longer than one big matmul. Anything smaller than that, you’d be better off computing where the data already lives.
plate VI · judgement
Which one, when
By now the answer to “which is best?” should feel like a set of questions rather than a name. Is the work a chain or a crowd? Is it matrix multiplication? How much does it branch? How big is it, and what does the software already look like?
The chooser asks those five questions and shows its reasoning, pro and con, for each chip. It is deliberately simple: real decisions also weigh price, availability, power and what your team already knows. But the reasoning is the right reasoning.
And notice the verdict always keeps the CPU in the picture. Modern machines are heterogeneous by design: a CPU orchestrates, accelerators take the dense parallel work, and a lot of engineering goes into keeping the links between them busy. Somebody then has to generate machine code for all three from one program, which is the job of the compiler, and the subject of the AOT compiler notebook. To go deeper on each chip, open the CPU, GPU and TPU notebooks.
Questions people ask
- Which is faster: a CPU, a GPU or a TPU?
It depends on the work. On branchy code with little parallelism the CPU wins. On large, regular workloads such as image filters or matrix maths, a GPU or TPU can be tens to hundreds of times faster. Plate II races all three on four workloads.
- What is the roofline model?
A chart of how fast a chip can run a piece of code, given how many operations the code does per byte it reads from memory. Below a certain arithmetic intensity the code is limited by memory bandwidth; above it, by compute. It tells you which upgrade will actually help.
- Can a GPU or TPU replace the CPU?
No. Accelerators run kernels; something still has to load the data, run the operating system, launch the work and handle everything that is not matrix maths. Every GPU and TPU system has a CPU host keeping it fed, and a slow host leaves an expensive accelerator idle.
Related material
elsewhere in the notebooks
further reading
- Making Deep Learning Go Brrrr From First Principles Horace He Compute, memory bandwidth and overhead: the three things your model is waiting on, and how to tell which. Plates III and IV in essay form.
- How to Scale Your Model Google DeepMind A whole book on rooflines, TPUs and GPUs at scale. The best free resource on how big models meet real hardware.
- Roofline model Wikipedia The model behind plate III, with the original Williams, Waterman and Patterson paper linked.
- CUDA C++ Programming Guide NVIDIA Where the host-and-device split of plate IV is spelled out from the programmer’s side.
- Miri My GPU-first programming language: one language, compiled for the CPU and the GPU, because a modern program lives on both.