What are you paying for?

A language model reads text in tokens, chunks about three-quarters of an English word long (the LLM notebook shows how text is cut up). Model APIs bill by the token, quoted per million tokens, and they keep two prices: one for the tokens you send in, one for the tokens the model writes back.

The output price is almost always the higher one, usually three to eight times the input price. There’s a mechanical reason. The model reads the whole prompt in one parallel pass, but it writes its answer one token at a time, with a full trip through the network for every token. Writing keeps the GPU busy far longer than reading.

So a request’s cost is two multiplications and a sum: input tokens × input price, plus output tokens × output price. Multiply by requests per day and the small numbers turn into a budget line. Many providers also sell the same model at about half price as a batch, if you can wait hours rather than seconds for the answer.

Set a long prompt and a one-line reply. Then flip it round. Which arrangement costs more, and by how much?

This surprises almost everyone the first time they look. The model is billed for everything it reads, and what you type is the smallest piece of that. Behind a twelve-word question sits a whole bundle the application sends along with it.

The system prompt holds the instructions that make a general model behave like your product. Tool definitions describe every function the model may call, and they go out in full with every request. The conversation so far rides along too, because the model remembers nothing between calls. A search step often pastes in retrieved documents so the model can answer from them.

Reasoning models add one more line. Before answering they write a hidden chain of thought, and those reasoning tokens are billed at the output price, even though you never see them. The FinOps Foundation’s guide to AI costs makes the same point: billed tokens and typed tokens are different numbers.

Turn everything on and read the ratio. Then switch off only the tool definitions. Was that the piece you expected?

An agent is a loop: the model reads, calls a tool, reads the result, calls another tool (the agent notebook builds one). Each turn is a fresh request, and because the model remembers nothing, each request re-sends everything so far: the instructions, every earlier message, every tool result.

So the bill grows with the number of turns squared. Turn thirty pays to read turns one to twenty-nine again. That’s why a coding agent’s bill is mostly input, and why a loop left running overnight can produce a bill nobody believes in the morning.

It can. Providers offer prompt caching: when a request starts with exactly the same tokens as a recent one, that prefix is read from a cache and billed as a separate line item, at a fraction of the input price, from about half down to a tenth depending on the provider. Writing a prefix into the cache can cost a little extra. On a loop like this one, the cache line ends up holding most of the tokens.

Play to turn 30 with caching off, note the total, then switch caching on. Then make each tool result twice as big.

Tokens aren’t the only meter. Run an open-weights model on your own GPUs, or train one, and the bill is in GPU hours: you pay for the machine for every hour it’s switched on, whether it’s answering a thousand requests or none, which is why self-hosting only pays at high, steady volume. Many providers sell their own models this way too, as provisioned throughput: a reserved slice of capacity at an hourly or monthly price, much like the cloud commitments sold for servers.

On this meter, the price of a token isn’t on any price list. It falls out of one ratio: what the hour cost, divided by the tokens you served in it. A busy GPU makes cheap tokens. An idle one makes very expensive tokens, or rather, no tokens at a fixed price.

Traffic has a shape, though. It peaks in office hours and sags at night, and capacity has to cover the peak or requests get throttled. Size for the peak and you pay for empty hours. Size for the average and the busiest hours spill over, onto a pay-per-token endpoint or into an error. Which way to lean is a business call.

Add GPUs until nothing spills, then read the cost per million tokens. Now halve the traffic.

The third meter counts people. Coding assistants, chat tools and the AI built into office software are mostly sold per seat: a flat fee per user per month, however much they use it. Finance loves seats, because the bill is known in advance. The trouble is that AI use is very uneven.

Plot it for a hundred people and you get a long tail. Most people use a little. A handful use a lot, often more than their seat costs the vendor in tokens. A flat seat makes the light users pay for the heavy ones, and every seat nobody opens is pure cost.

That’s why AI pricing keeps drifting towards a seat with usage included, plus overage. The seat covers the median user, and the tail pays for what it uses. For the buyer, that means the seat bill alone no longer tells you what the tool costs: the number to watch becomes cost per active user.

Try “flat seat” at 40% adoption, then “seat + usage” at 90%. Which one would you rather explain to finance?