Slavik Shynkarenko

notebook № 247 plates · 22 min to read, longer to play ·

From your kernel to the silicon

On machine code that won’t sit still

part 6 of 6 in How chips compute · by

A CPU binary from 1998 still runs today. A GPU binary from a few years ago may not. This notebook follows one kernel from source to silicon: the PTX and SPIR-V that vendors put in the middle, the compiler hiding in your driver, the registers it hands out, Apple’s unified road, why games stutter, and the map of translators that holds it all together.

I

plate I · compatibility

Machine code that won’t sit still

A program compiled for a Pentium in 1998 still runs on the laptop I’m writing this on. Intel and AMD have rebuilt everything inside their chips since, but the instruction set at the edge is a promise they keep. Decades of software depend on it, so CPU makers pay for that promise in silicon and never stop paying.

GPU vendors don’t make it. NVIDIA’s real machine code, called SASS, changes from one architecture to the next: new instructions, new encodings, new scheduling rules. A kernel compiled for one generation may simply refuse to load on the next.

So CUDA packs two kinds of code into one file. A fatbinary holds finished machine code (a cubin) for each GPU generation you asked for, plus PTX, a portable assembly language for an imaginary NVIDIA GPU. When the program starts, the driver looks for a cubin that fits the chip. If there isn’t one, it compiles the PTX itself, there and then, and caches the result for next time.

The rules fit in two sentences. A cubin built for compute capability 8.0 runs on any 8.x chip with a minor version at least as high, and never on 9.0. PTX for 8.0 can be compiled for anything from 8.0 up.

Embed only the sm_80 cubin and pick the H100. Then switch the compute_80 PTX back on. That one switch is why old CUDA programs keep running on new GPUs.

fig. 24.1~/fatbinarychoose what to embed, then pick a GPU
The driver’s choice, per NVIDIA’s compatibility rules: the best-fitting cubin if there is one, otherwise the newest PTX it can compile, otherwise the launch fails. GPUs labelled by compute capability.
II

plate II · the layers

One kernel, three languages

Here is the smallest useful CUDA kernel, SAXPY: y = a·x + y, one element per thread. nvcc turns it into PTX, and ptxas, or later the driver, turns the PTX into SASS. Tap a line in any of the three to see what it became.

PTX looks like assembly, but it’s assembly for a machine that doesn’t exist. It has as many registers as it wants, each typed: %r for 32-bit integers, %f for floats, %rd for 64-bit addresses, %p for true-or-false. nvcc writes it close to the one-name-one-value form from the compiler notebook, with nearly every register assigned once.

fig. 24.2~/three-levelstap a line
PTX in the style nvcc writes, SASS in the style of an Ampere GPU. Both are trimmed and simplified: register numbers and constant-bank offsets are illustrative.

SASS is the real thing, for one generation. Look at what changed on the way down. The kernel’s parameters and blockDim aren’t loaded at all: they sit in a constant bank, c[0x0][…], and instructions read them in place. The bounds check became a predicated EXIT. The address arithmetic folded into IMAD.WIDE. And seventeen virtual registers became six real ones, which matters more than it sounds.

Tap the bounds check, if (i < n). In PTX it’s a compare and a branch. In SASS the branch has gone, and the threads that fail the test simply exit.

III

plate III · register allocation

Registers decide the crowd

Turning seventeen virtual registers into six real ones is register allocation, the same job the compiler notebook’s back end does with a CPU’s sixteen. On a GPU it decides something bigger than speed.

Every resident warp keeps its registers in the SM’s register file for its whole life. That is what makes switching between warps free. But the file is finite: 65,536 32-bit registers on a recent NVIDIA SM. Each extra register per thread costs 32 per warp, so a hungry kernel leaves room for fewer warps, and fewer warps means less to switch to while one of them waits for memory.

Ask for more than 255, the most a thread can address, and the compiler has to spill: park values in local memory, which is ordinary device memory with a cache in front. Fast kernels live in a narrow band, with enough registers to avoid spilling and few enough to keep the SM crowded.

With 128-thread blocks, slide from 32 to 64 registers: occupancy halves. Then pick 1,024-thread blocks and go to 255. The launch fails, which is a real CUDA error.

fig. 24.3~/register-fileregisters per thread · block size
One Hopper-class SM: 65,536 registers, at most 64 resident warps and 32 blocks. Registers are handed out per warp in steps of 256. Shared memory, the other limit, is left out.
IV

plate IV · spir-v

Words all the way down

CUDA is one vendor’s road. Nearly everyone else meets in SPIR-V, a binary intermediate language published by the Khronos Group. Vulkan takes its shaders as SPIR-V, OpenCL and OpenGL 4.6 accept it, and every desktop and mobile GPU vendor ships a driver that compiles it. Write in GLSL or HLSL, compile to SPIR-V once, and any of those drivers will take it from there.

fig. 24.4~/module.spvtap an instruction
A complete SPIR-V 1.0 compute shader that does nothing, 64 threads at a time: every word of it. The text on the left is what a disassembler such as spirv-dis prints.

It’s refreshingly literal. A module is a stream of 32-bit words. The first is a magic number, 0x07230203, then a version, the tool that made it, and a bound on the ids inside. After that, every instruction opens with one word that packs its own length and its opcode, so a tool can step over anything it doesn’t understand.

The first instruction is the interesting one. OpCapability Shader declares what the module needs from the device, and features like 16-bit floats, 64-bit integers or subgroup operations each have a capability of their own. A driver can turn a module down before running a single instruction of it.

Tap OpEntryPoint: five words, with “main” spread over two of them because the name needs a terminating zero. Then switch to bytes and find the magic number written backwards.

V

plate V · metal

Apple’s road

Apple went its own way. Metal shaders are written in the Metal Shading Language, a dialect of C++, and compiled ahead of time to AIR, an LLVM-based intermediate form packed into a .metallib file. When the app creates a pipeline, the driver compiles that AIR for the GPU in that particular Mac or iPhone. Apps that can’t afford the wait ship binary archives with the last step already done.

The bigger difference is where the data lives. A discrete GPU keeps its own memory on the far side of a PCIe link, so inputs get copied over and results copied back. On Apple silicon the CPU and the GPU share one memory, and a buffer the CPU just wrote is already where the GPU reads.

That isn’t automatically a win. The discrete card in the simulation has faster memory and far more arithmetic. Unified memory wins when the copy is the expensive part, which is whenever the kernel does little work for each byte it touches.

At 1 FLOP per byte the copy is almost all of the discrete card’s time. Raise the work until it wins anyway. Then switch on “data already on the GPU”, which is what well-written CUDA programs fight for.

fig. 24.5~/copy-or-sharedata size · work per byte
A simulation with rounded figures: a discrete card at about 1 TB/s and 80 TFLOP/s behind a PCIe 4.0 ×16 link at about 25 GB/s, against a unified-memory chip at about 0.4 TB/s and 15 TFLOP/s. The kernel reads and writes the data once.
VI

plate VI · just in time

Compiled while you play

Every road so far ends in the same place: the driver compiling for the exact chip, at run time. For a compute job that is a short pause at start-up. For a game it is a problem, because a frame at 60 fps lasts 16.7 milliseconds, and compiling a single shader pipeline can take many times that.

Games often meet a pipeline for the first time in the middle of play: the first explosion, the first rain, the first menu that fades in. The driver compiles, the frame waits, and you feel it as a hitch. The fixes move the cost rather than remove it. Drivers keep a cache on disk, so the second encounter is quick. Vulkan pipeline caches and DirectX 12 pipeline libraries let a game keep its own. And a game can compile everything behind the loading screen, trading a wait you expect for hitches you don’t.

Steam goes one step further for some games: players with the same GPU and driver share shader caches, so the work is done before you press play. The compile still happens, just earlier, and some of it on somebody else’s machine.

Play once with both switches off and count the hitches. Play again with a warm cache. Then precompile at load and see where the time went.

fig. 24.6~/frame-timesplay, then flip the switches
A simulation of ten seconds of play at 60 fps. Each new effect needs a pipeline the first time it appears. Compile times are invented but in a realistic range.
VII

plate VII · the network

Many roads, a few hubs

Put it all together and GPU software looks less like a stack and more like a rail network. A handful of hubs carry most of the traffic: PTX for NVIDIA, SPIR-V for nearly everyone else, DXIL for DirectX, AIR for Apple. Translators run between them. SPIRV-Cross turns SPIR-V back into Metal or HLSL source, MoltenVK runs Vulkan on top of Metal, and the browsers’ WebGPU engines, Tint in Chrome and naga in Firefox, translate WGSL for whichever API the machine has.

fig. 24.7~/routespick a language and a chip, or tap a cell
The main documented road for each pair, simplified. Most pairs have more than one. A dash means no road I know of that a vendor or a major project supports. Miri’s row (◇) shows its design, a back end per target, rather than a list of routes.

The map keeps changing. In 2024 Microsoft announced that DirectX will adopt SPIR-V as its interchange format from Shader Model 7, which would leave one portable hub for nearly everything that isn’t CUDA or Metal. AMD never had a PTX of its own. HIP compiles ahead of time, through LLVM, straight to the machine code of each AMD chip.

This map is why I designed Miri, my own language, the way I did. The code a programmer writes should stay put while the layers underneath keep moving. So Miri keeps one source surface and its own mid-level IR, MIR, and adds a back end per target behind it. Each back end lowers MIR to a different middle layer: a portable shading language, SPIR-V, a tile-level IR such as NVIDIA’s Tile IR. From there the vendors’ own toolchains carry it to the chip. It’s the same network seen from the language designer’s side. Aim at the hubs, and let the people who build the chips run the last mile. The front half of that pipeline is in how Miri’s compiler works.

So when someone says a GPU “runs CUDA” or “runs Vulkan”, it means less than it sounds. No GPU runs your code. It runs what its driver made of it, a moment ago, on your machine. How you write that code in the first place, as threads, tiles or whole arrays, is the subject of the GPU programming models notebook.

Pick GLSL and Apple, and count the hops: GLSL, SPIR-V, Metal source, AIR, the chip. Then pick MSL and Apple. Then pick Miri, which draws a design instead of a single road.

Questions people ask

What is PTX in CUDA?

PTX is NVIDIA’s virtual instruction set: an assembly language for an idealised GPU with unlimited typed registers. The CUDA compiler emits it, and then either ptxas turns it into SASS, the real machine code of one GPU generation, ahead of time, or the driver compiles it when the program first runs on a GPU it has no SASS for.

What is SPIR-V used for?

SPIR-V is a binary intermediate language from the Khronos Group. Vulkan, OpenCL and OpenGL 4.6 accept shaders and kernels as SPIR-V, so a compiler like glslang or DXC can produce one file that any vendor’s driver then compiles for its own hardware. Translators such as SPIRV-Cross also use it as a hub between shading languages.

Why do games stutter when compiling shaders?

The driver has to turn each shader pipeline into machine code for your exact GPU, and many games only ask for a pipeline the first time an effect appears. That compile can take far longer than a 16.7 ms frame, so the frame waits. Driver caches, pipeline caches and compiling during the loading screen move the work out of gameplay.

Related material

elsewhere in the notebooks

№ 23 Four ways to program a GPU On how much to spell out № 02 What a GPU is, and how it works On the patience of crowds № 05 The architecture of an ahead-of-time compiler On turning words into motion № 01 What a CPU is, and how it works On the clock that never stops

further reading

← the Chips series · all notebooks