notebook № 215 plates · 13 min to read, longer to play ·
Rent the tokens or run the GPUs?
On meters, utilisation and the costs behind the GPU
part 5 of 6 in What AI costs · by Slavik Shynkarenko
An API charges for every token and nothing for the quiet hours. A GPU charges for every hour whether anyone is asking or not. Which one is cheaper depends less on the price list than on how busy you can keep the hardware, and on the costs that never appear on the GPU bill.
plate I · two pricing models
Two meters for the same tokens
There are two ways to pay for a model’s answers. A per-token API charges for exactly what you send and receive. Quiet hours cost nothing. A self-hosted model runs on GPUs you rent by the hour, sized for your busiest moment, and the meter runs whether anyone is asking or not.
So the same traffic can cost very different amounts depending on its shape. Steady traffic around the clock keeps the GPUs busy and makes every hour count. Traffic that peaks at lunchtime and dies at night pays for a peak-sized fleet through all the empty hours.
Neither meter is cheaper in general. The question is how much of the rented time you can actually fill.
Switch between steady, office-hours and spiky traffic. In which shape do the GPUs win the day?
plate II · workload placement
Where the lines cross
Plot monthly cost against monthly volume and the two options draw different shapes. The API is a straight line: twice the tokens, twice the bill. Self-hosting starts with a floor, the smallest fleet you can run reliably, and then climbs in steps as each GPU fills up.
Where the lines cross is the break-even volume. Below it, you’re paying for GPUs to wait. Above it, every extra token on your own hardware is cheaper than buying it. At low volume the API wins by a wide margin. At very high volume self-hosting usually does.
Both inputs move. GPU prices vary by provider and commitment, and API prices for a given capability keep falling, as notebook 19 showed. A break-even calculated once is out of date within a year.
Find the break-even at $1 per million tokens. Then halve the API price, the way prices have tended to fall, and see how far the crossing moves.
plate III · GPU efficiency
Utilisation is the price
The effective price of a self-hosted token is simple division: what the GPU costs per hour, over the tokens it actually serves in that hour. Halve the utilisation and you double the price per token. A GPU that looks cheap on a spreadsheet at full load can cost several times the API price at the loads real services see.
The other lever is batching. A serving engine with continuous batching runs many requests through the GPU at once, and total throughput climbs steeply with concurrency. The catch is that each user’s answer streams more slowly, so latency sets a ceiling on how hard you can batch.
That’s also why a GPU you run is so different from a GPU you read about: the hardware is only as cheap as the scheduler that keeps it full.
Set utilisation to 25% and push concurrency to the top. Can batching alone reach the API price? Then raise utilisation until it does, and check each user’s tokens per second.
plate IV · total cost of ownership
What the GPU bill leaves out
An API price includes things a GPU price doesn’t: the engineers who keep the model serving, spare capacity in another zone, the upgrade to next quarter’s model, the security reviews. Self-host and they become your costs, and they don’t shrink with volume.
Add them honestly and the comparison changes. On-call engineers alone can outweigh the GPUs at moderate volume. Redundancy across zones adds idle hardware by design. And every model upgrade means new evaluations, new tuning and sometimes new hardware.
There are good reasons to self-host that aren’t about price: data that can’t leave your network, a fine-tuned model, latency you control. Make that case on its merits, and put the people back into the GPU quote.
Start with the GPU bill alone, then add each hidden cost. When does the API become the cheaper option? Then let the API price fall by half.
plate V · licensing and SaaS
Seats, usage or both
Most AI spend goes on tools rather than your own models: coding assistants, chat products, AI features inside SaaS you already pay for. They face the same choice between a fixed meter and a per-use one, as seats or usage.
Seats are easy to budget and generous to heavy users. Usage pricing follows real consumption, so light users cost little. Which is cheaper depends on how usage is spread across people, and in most teams it’s spread very unevenly, as notebook 18 showed.
Hybrid contracts sit in between: a seat with included credits, and usage above it. Know your usage curve before the renewal, not after it.
Compare the three contracts at today’s usage. Then turn usage intensity up, the way it grows once agents arrive. Which contract wins now?
Questions people ask
- When is self-hosting an LLM cheaper than using an API?
When volume is high and steady enough to keep GPUs busy most of the time. GPUs are paid by the hour, so the cost per token falls as utilisation rises. With low or spiky traffic, much of the paid GPU time is idle and a per-token API is cheaper. Engineering, on-call and redundancy costs push the break-even higher still.
- How do you calculate the cost per token of a self-hosted model?
Divide the GPU cost per hour by the tokens it actually serves per hour. That is the GPU’s throughput, which rises with batching, times its utilisation. A GPU at $3 an hour serving 1,500 tokens a second at full load costs about $0.56 per million tokens. At 25% utilisation it costs four times as much.
- Should AI tools be bought per seat or by usage?
Per-seat pricing is predictable and cheap for heavy users, but overpays for light ones. Usage pricing tracks actual consumption and suits uneven adoption, but is harder to budget. Many contracts mix the two: a seat with included credits and usage above it. The right choice depends on how usage is spread across people.
Related material
elsewhere in the notebooks
further reading
- FinOps for AI Overview FinOps Foundation The working-group paper on AI cost drivers, including the trade-offs between managed APIs, provisioned capacity and self-hosted models.
- FinOps Certified: AI Value FinOps Foundation The certification whose third level, on workload placement, licensing and SaaS decisions, this notebook follows.
- vLLM open-source project A widely used serving engine whose continuous batching is what lets one GPU serve many requests at once, the lever in plate III.