series6 notebooks · 31 plates · about 80 minutes of reading
What AI really costs
AI spend behaves differently from cloud spend. It’s priced per token instead of per hour, and it follows usage instead of capacity, so the bill moves with every prompt someone ships. Making that spend transparent is part of my job, and the questions these notebooks answer are the ones I hear most.
Six notebooks, in reading order, covering roughly the ground of FinOps Certified: AI Value. They borrow the vocabulary of the cloud series, so if allocation or unit economics are new words, start with Cloud FinOps.
The reading path
-
How AI is billed
AI is billed through a few meters. Model APIs charge per million tokens, with output dearer than input and cache reads cheaper. Self-hosted models and training are billed by the GPU hour, and provisioned capacity by the hour whether you use it or not. AI built into tools is billed per seat, often with usage on top. Priced by the tokenWhat you typed, what you paid forThe agent rereads its deskPaying for the clockPer head, not per useFour bills that don’t agree -
AI cost allocation
AI cost allocation assigns model and GPU spend to the teams, features and customers that caused it. Provider invoices only show keys and models, so teams use separate keys or projects, route calls through a gateway that tags every request, split shared GPU clusters by tokens or GPU time, and keep training separate from inference. One key, five teamsThe gateway sees every callSharing a GPU clusterTraining once, serving foreverThe heavy tail -
AI cost forecasting
Forecast AI spend by building it from drivers: users, requests per user, tokens per request and price per token, each with a low and high value. Expect a fat-tailed range rather than a single number, since prices fall while usage grows. Budget per unit, such as cost per active user, and add anomaly alerts and per-key limits. Estimate before the first callFat tails, not bell curvesCheaper tokens, bigger billsThe agent that ran all nightBudget the unit, not the total -
Cutting LLM costs
Cut LLM costs by sending fewer tokens and paying less for each one. Trim system prompts, history and retrieved context. Cache stable prompt prefixes. Route easy requests to smaller models, send work that can wait to batch APIs at about half price, and cap output length. Measure answer quality before and after every change. Where the tokens goPay once for the same prefixThe right model for each requestNot everything needs an answer nowStack the levers, watch quality -
API vs self-hosting
Self-hosting an LLM beats a per-token API only at high, steady volume. GPUs bill by the hour whether busy or idle, so the effective price per token depends on utilisation. Below the break-even volume, or with spiky traffic, the API is cheaper. Include on-call engineers, redundancy and model upgrades in the comparison. Two meters for the same tokensWhere the lines crossUtilisation is the priceWhat the GPU bill leaves outSeats, usage or both -
AI unit economics
AI unit economics measures the cost of each successful outcome, not each model call: a resolved ticket, an accepted code change, a correct document. Divide total AI cost, including retries and human follow-up, by outcomes that meet a quality bar, then compare it with the value each outcome creates, such as time saved or revenue. Cost per call is the wrong unitWhere you put the barCost against valueTime to valueA portfolio, not a project
Questions people ask
- How is AI usage billed?
Mostly three ways: per token for API models, per GPU hour for models you host or fine-tune yourself, and per seat for assistants and coding tools. Input and output tokens carry different prices, and cached input costs less again. The first notebook in the series takes each meter apart.
- Is it cheaper to self-host an LLM than to use an API?
Only above a break-even volume. A self-hosted model costs its GPU hours whether you use them or not, while an API charges for what you send. The self-hosting notebook lets you move the volume and watch where the two lines cross.
- Where should I start cutting LLM costs?
With the prompt: shorter system prompts, fewer tool definitions and cached context. Then route easy requests to smaller models and move work that can wait to batch APIs. The notebook on cutting LLM costs puts a price on each step.