notebook № 185 plates · 13 min to read, longer to play ·
Whose tokens are these?
On keys, tags and tokens with owners
part 2 of 6 in What AI costs · by Slavik Shynkarenko
An AI invoice arrives as one line per model, from a provider that has no idea which team, feature or customer sent each request. Allocating it takes new plumbing: keys per team, a gateway that tags every call, rules for the GPUs everyone shares, and a way to tell a one-off training run from the inference that serves customers every day.
plate I · keys and projects
One key, five teams
Most AI adoption starts the same way: someone creates an API key, it ends up in a shared secret store, and within a quarter five teams are using it. The invoice then says exactly one useful thing: how much the key cost.
The provider can’t split what it can’t see. It knows the key, the model and the token counts, as notebook 10 showed. It doesn’t know which feature sent the request, or which customer it served. So the first allocation step is the cheapest one: give each team or application one key, or one provider project. The provider’s own usage reports then split by owner.
That gets you to the team. Getting to the feature, the customer or the experiment needs metadata on every request, which no key can carry. For that, the requests have to pass through something that writes things down.
Switch from one shared key to a key per team, then to a gateway. Which questions can you answer at each step?
plate II · LLM gateways
The gateway sees every call
An LLM gateway (also called an AI gateway or model proxy) sits between your applications and every model provider. Each request passes through it with a few headers (team, feature, customer, environment) and the gateway logs them next to the model, the token counts and the price.
That log is the AI equivalent of the billing export: one row per call, ready to group any way you like. Cloud tags live on resources. AI tags live on requests. Once the log exists, the gateway can also enforce budgets, cache answers and route to cheaper models, which is why it tends to become the control point for AI spend.
The log is still an estimate until it’s reconciled. The gateway prices tokens with list rates, and the invoice applies discounts, credits and the provider’s own token counting. Allocate from the log, then scale it to match the invoice, and keep the difference visible.
Group the same calls by team, then by feature, then by model. Which view would a product manager want?
plate III · shared inference
Sharing a GPU cluster
Self-hosted models move the problem from invoices to infrastructure. A cluster of GPUs serves a shared model for many teams, billed by the hour whether busy or idle, the way Kubernetes clusters are.
The obvious split, by number of requests, is wrong for language models. One team sends short questions. Another sends 30-page contracts. Work on a GPU scales with tokens, and long prompts cost more than their share of requests suggests. Splitting by requests makes the short-prompt teams pay for the long-prompt ones.
The fairer bases are tokens processed, or measured GPU time per request where the serving stack reports it. And then there’s idle capacity: GPUs held for peaks or kept warm for latency. Someone has to own the idle, or it vanishes into everyone’s rate.
Split by requests, then by tokens. Watch what happens to the contracts team.
plate IV · training and inference
Training once, serving forever
AI spend has a lifecycle that cloud spend mostly doesn’t. Training or fine-tuning is a burst: a few days of many GPUs that produce a model. Inference is the steady cost of using that model, every day, for as long as it serves. Batch jobs sit in between. The FinOps Foundation’s AI guidance recommends tagging the three separately, because they answer different questions.
A fine-tune that cost $90k in March and serves three products for a year shouldn’t make March look like a disaster or the products look free. Like an upfront commitment, it can be amortised over its useful life and shared between the products that use it. Training behaves like an investment. Inference behaves like cost of goods sold.
That split matters beyond reports. Inference serving customers lands in gross margin. Training and experiments often count as research and development, and in some organisations can be capitalised. Which bucket a GPU hour lands in is a decision, so make it once and write it down.
Switch from “expense when run” to “amortise over life”, then stretch the useful life to 18 months.
plate V · per-user allocation
The heavy tail
Allocate AI cost all the way down to users, and a shape appears that cloud infrastructure rarely shows. Usage is wildly uneven: a small share of users drives a large share of tokens, through long conversations, big documents and agents left running.
Under a flat subscription, that tail decides the margin. The typical user may cost well under a dollar a month, while the top one percent can cost more than they pay. Seat-based AI products from the big vendors face the same maths, which is why so many plans come with fair-use limits and premium request quotas.
Per-user allocation needs the gateway from plate II with a user or customer tag on every call. With it, product teams can set caps, price heavy usage, or route the heaviest workloads to cheaper models. Without it, the tail is invisible until the invoice.
Move the slider to the top 1% of users. Then turn on the usage cap and watch the curve bend.
Questions people ask
- How do you allocate LLM API costs to teams?
Give each team or application its own API key or provider project, so the provider’s usage reports split by owner. For finer detail, route calls through an LLM gateway that records team, feature and customer tags with the token counts of every request, then price those tokens with the provider’s rates and reconcile the total against the invoice.
- What is an LLM gateway?
An LLM gateway, or AI gateway, is a proxy between applications and model providers. It forwards each request, logs the model, token counts, latency and cost with metadata such as team and feature, and can enforce budgets, rate limits, caching and model routing. For FinOps it is the main source of allocation data for AI usage.
- Should AI training costs be allocated differently from inference?
Yes. Training and fine-tuning are periodic investments that produce a model used for months, so they are usually tracked separately and often amortised over the model’s useful life, and may be treated as R&D. Inference runs every day to serve users and behaves like cost of goods sold, so it is allocated to the products and customers it serves.
Related material
elsewhere in the notebooks
further reading
- FinOps for AI Overview FinOps Foundation The working-group paper on AI cost: tagging training, inference and batch separately, and why billed tokens differ from typed ones.
- FinOps Certified: AI Value FinOps Foundation The certification whose first level, on AI allocation and data sources, this notebook follows.
- OpenCost CNCF project Open-source cost allocation for Kubernetes, including GPU workloads.