notebook № 205 plates · 14 min to read, longer to play ·
Same answers, fewer tokens
On context diets, cache hits and the right model for the job
part 4 of 6 in What AI costs · by Slavik Shynkarenko
Most LLM bills carry the same waste: prompts that resend everything, the biggest model for every question, and answers bought at full price that nobody needed for a day. Five levers fix most of it, and each one trades a little of something else: latency, quality or engineering time.
plate I · prompts and context
Where the tokens go
Open a single request from a production assistant and count. The user’s question is a few dozen words. Around it sit a system prompt, tool definitions, ten turns of history and ten retrieved document chunks. The question is under one percent of what you pay to send.
That’s why I start with the context rather than the model. A system prompt that grew one line at a time can usually lose half its length. Old turns can be summarised instead of resent, as notebook 08 showed. Retrieval that returns the top three chunks often answers as well as the top ten.
Output is the other side. It’s priced several times higher than input, and models are happy to write three paragraphs where one would do. An instruction to be brief and a sensible maximum length are the cheapest optimisation there is.
Turn on each trim in turn. Which one saves the most? Then check what’s left: how much of the request is the user’s question now?
plate II · prompt caching
Pay once for the same prefix
Most of a prompt is the same from one request to the next: the system prompt, the tools, the shared documents. Prompt caching lets the provider keep that prefix ready and charge a fraction for reading it again. One provider prices cache reads at about a tenth of normal input. Another caches automatically and discounts cached tokens heavily.
With explicit caching there’s a catch: writing to the cache costs about a quarter more than normal input, and entries expire after a few minutes without use. A cache that’s rarely hit costs more than no cache at all. Order the prompt so the stable part comes first and the changing part last, or nothing matches.
A second kind of cache works one level up. A semantic cache stores whole answers and returns them for questions that mean the same thing. Every hit there costs no model tokens at all, though a wrong match returns a wrong answer, so it suits FAQs better than conversations.
Drag the hit rate below the break-even rate and watch the cache cost money. Then turn on the answer cache.
plate III · model routing
The right model for each request
Sending every request to the largest model is like shipping every parcel by courier. Most questions are easy: classify this, extract that, rephrase a sentence. A small model answers them as well as a large one, at a twentieth of the price.
A router decides per request. It can be a few rules on the request type, or a small model trained to guess difficulty. Research on trained routers reports cost cut by more than half in some benchmarks without losing quality. The router’s mistakes are the price: a hard question sent to a small model comes back wrong.
So routers carry a margin, a bias toward the bigger model when in doubt. A wider margin buys quality back with money, and only an evaluation set tells you where the right setting is.
Compare “all large” with “all small”. Then switch on the router and widen the margin until quality matches the large model. What did it cost?
plate IV · batch and rate optimisation
Not everything needs an answer now
The major providers sell the same models at about half price through batch APIs: submit a file of requests, collect the answers within 24 hours, often much sooner. It’s the AI cousin of spot capacity: cheaper because the provider schedules it when it suits them.
A surprising share of AI spend can wait. Nightly summaries, evaluation runs, catalogue enrichment, re-embedding a document store: nobody is watching a spinner for any of them. They often run through the real-time API only because that’s the code someone wrote first.
Rate levers work the same way. Steady, predictable volume can move to provisioned capacity or a committed-spend agreement, like a cloud commitment. Real-time pricing is for work with a user on the other end.
Move every workload that can wait to batch. Which ones can’t move, and what share of the bill do they leave?
plate V · the trade-off
Stack the levers, watch quality
The levers compound. Trim the context and there’s less to cache. Route to a small model and the batch discount applies to a cheaper price. Stacked, they often take a bill to a third of where it started, or less, which is why AI optimisation pays back faster than most cloud work.
Each lever also touches something users notice. Shorter context can drop the detail that answered the question. Routing can send a hard question to a model that gets it wrong. Cost is the easy number. Quality is the one that needs a test. It’s also what turns a cost into a cost per good answer.
So keep an evaluation set of real requests with known good answers, and run it before and after every change. With evaluations, you give up a little saving and keep the quality. Without them, you find out from users.
Turn on every lever without evaluations and read the quality. Then turn evaluations on. What did they cost, and what did they save?
Questions people ask
- What is the fastest way to reduce LLM API costs?
Look at what each request sends. Input tokens usually dominate: long system prompts, full conversation history and too many retrieved documents. Trimming them and caching the stable prefix often cuts the bill by half without touching the model. Routing easy requests to a smaller model and batching offline work come next.
- How much does prompt caching save?
Cached input tokens cost a fraction of the normal price, about a tenth with some providers. Savings depend on how much of each prompt is a stable prefix and how often it is reused before the cache expires. With explicit caching, writes cost extra, so a low hit rate can cost more than no cache at all.
- What is LLM model routing?
Model routing sends each request to the cheapest model likely to answer it well. A router, often a small classifier or rules on the request type, sends simple questions to a small model and hard ones to a large model. Done well it cuts cost substantially. Done carelessly it lowers quality, so it needs evaluation.
Related material
elsewhere in the notebooks
further reading
- Prompt caching Anthropic documentation Explicit caching: writes at 1.25× the input price, reads at 0.1×, a five-minute default lifetime.
- Prompt caching OpenAI documentation Automatic caching for prompts over about a thousand tokens, with cached input heavily discounted.
- Batch API OpenAI documentation Half-price requests that complete within 24 hours. Anthropic’s batch processing works the same way.
- RouteLLM: Learning to Route LLMs with Preference Data Ong et al., 2024 Trained routers between a strong and a weak model, cutting cost by over 2× in some benchmarks without losing quality.
- FinOps Certified: AI Value FinOps Foundation The certification whose third level, on workload and rate optimisation for AI, this notebook follows.