Retrieval-augmented generation: before the model answers, a search step finds the passages most relevant to the question and adds them to the prompt. The model answers from those passages instead of from memory alone, which keeps answers current and lets them cite a source.

What is a vector database?

A database that stores embeddings, lists of numbers that capture the meaning of a piece of text, and finds the ones closest to a query. Approximate nearest-neighbour indexes such as HNSW let it search millions of vectors without comparing against every one.

What is chunking in RAG?

Splitting documents into smaller passages before embedding them, so a search returns the paragraph that answers the question rather than the whole document. Size and overlap matter: too small and a fact gets cut in half, too large and the match gets diluted.

notebook № 086 plates · 15 min to read, longer to play

How agents remember

On what to keep in mind

A language model remembers nothing from one call to the next. Everything an agent seems to remember is text that someone chose to put back in front of it. This notebook follows that choice: the context window, cutting documents into chunks, turning meaning into numbers, searching by meaning, and deciding what is worth keeping at all.

plate I · the goldfish

A brilliant goldfish

Here is the strangest fact about a language model: it remembers nothing. Every call starts from zero. When a chatbot seems to recall what you said ten minutes ago, it’s because the whole conversation so far is sent to it again, every single time.

That bundle of text is the context window: everything the model can see at once, measured in tokens (pieces of words; a token is roughly three-quarters of an English word). Windows have grown from a few thousand tokens to a million or more, but they always end somewhere.

When the conversation outgrows the window, something has to go. Drop the oldest messages and the model forgets them cleanly. Summarise them and it keeps the gist, but a summary is a choice about what mattered, and it can choose wrong.

If you’ve read the CPU notebook, this will feel familiar. The window is the cache: small, fast, and the only memory the worker actually touches. Everything else has to be fetched into it.

At 150 tokens, ask for the dog’s name with drop the oldest, then with summarise. Now ask what she’s scared of.

fig. 8.1~/context-windowslide the window, ask a question

A simulation. Token counts are a rough estimate (four tokens for every three words), and the answers are written by hand from what is still inside the window.

plate II · chunking

Cutting the book into cards

An agent can’t carry a whole manual, a codebase or a year of chat in its window. So long-term memory lives outside, in a store, and the agent fetches only the pieces it needs. Step one is cutting documents into those pieces, called chunks.

Where you cut matters more than it sounds. Cut too small and a fact loses its context. Cut too big and every chunk is about five things at once, so it matches everything a little and nothing well. Cut in the wrong place and the answer is sliced in half.

The usual fix is a little overlap: each chunk repeats the end of the one before, so a sentence on the border lands whole in at least one of them. You pay for it in storage, and in near-duplicates turning up side by side in search results.

Find a chunk size that keeps the key whole with no overlap at all. Is there only one?

fig. 8.2~/chunkssize and overlap

Colours mark chunks, small numbers mark where each one starts, dotted words are stored twice. Real systems often cut on sentences or headings instead of word counts; the trade-off is the same.

III

plate III · embeddings

Meaning as distance

To search by meaning, a computer needs meaning as numbers. An embedding turns a piece of text into a list of numbers, a point in space, arranged so that texts about similar things land close together.

Real embedding models learn their numbers from enormous amounts of text, and use hundreds or thousands of them per point. The one in this plate is a toy with eight directions I wrote by hand: pet, food, home, money, travel, health, outdoors and time. The arithmetic is real; the dictionary is tiny.

Type a question. Your words become eight numbers, the star lands on the map, and lines run to its three nearest notes. “Pup” and “hungry” appear nowhere in the feeding note, yet it comes out on top: closeness of meaning, not of spelling.

Ask “where is the house key?”. The boiler note comes a close second. Why would a note about clicking noises sit so near a key?

fig. 8.3~/embedding-maptype, or click a dot

A hand-made, eight-number embedding of twelve notes. The map squashes eight dimensions into two, so neighbours are judged in all eight: trust the lines and their numbers, not the distances on screen.

plate IV · search

Finding the right card

With every note stored as a point, finding the right one is geometry: measure how closely the question and each note point the same way, and rank them. The usual measure is cosine similarity. Near 1 means the same direction, 0 means unrelated.

Then take the top few (the top k) and paste them into the prompt. Switch to exact-word search and ask about a hungry pup: nothing matches, because no note says “pup”. Exact words still win for names, codes and IDs, which is why many systems run both kinds of search and merge the results.

Twelve notes are easy. A real store holds millions, and comparing against every one would be too slow, so vector databases use indexes such as HNSW graphs that find very close neighbours while checking only a small fraction. Approximately right, and very fast.

fig. 8.4~/searchmeaning or exact words

Real cosine similarity on the toy embedding from fig. 8.3, against all twelve notes. Exact-word scores are the share of the question’s words (minus little ones like “the”) that appear in the note.

plate V · retrieval

The open-book exam

Put the pieces together and you get retrieval-augmented generation, RAG for short: take the question, search the store, paste the best notes into the prompt, and let the model answer from them. It turns a closed-book exam into an open-book one.

When it works, it works beautifully: the model answers with facts it was never trained on, and can point at where it read them. But the model can only use what retrieval hands it. Ask where the passport is with top k at 1: the search ranks the flight first, the passport note never reaches the prompt, and the answer can be no better than that miss.

fig. 8.5~/ragquestion · retrieval · top k · rule

Step 1 is the real search from fig. 8.4. Steps 2 and 3 are a simulation: the prompt is built for real, the answers are written by hand.

The last switch matters most. Without a rule like “if the notes don’t say, say you don’t know”, a model whose search missed tends to guess, fluently and with confidence. With the rule, a miss usually becomes an honest gap. A visible “I don’t know” is a bug you can find and fix; a confident wrong answer is one you never hear about.

Raise top k to 3 for the passport question. Then turn retrieval off entirely, drop the rule, and read what comes back.

plate VI · keeping

What’s worth remembering

Once an agent can write things down, the hard question is what to write. People who build agents borrow words from psychology. Working memory is the context window. Episodic memory is a log of what happened. Semantic memory holds lasting facts. Procedural memory is how to do things.

And forgetting is a feature. Everything stored competes in every future search, so noise in the store becomes noise in the prompt. Some things should never be written down at all: a secret in memory can come back in any prompt later.

These are my picks, not rules; there are good arguments for filing a few of them elsewhere. An agent, in the end, is a careful librarian working for a forgetful genius. The agent notebook shows the loop that decides when to go to the shelves.

fig. 8.6~/memory-kindsfile each one

Eight scraps from an imaginary coding agent’s day. Hover a button for what each kind of memory is for.

Questions people ask

What is RAG?: Retrieval-augmented generation: before the model answers, a search step finds the passages most relevant to the question and adds them to the prompt. The model answers from those passages instead of from memory alone, which keeps answers current and lets them cite a source.
What is a vector database?: A database that stores embeddings, lists of numbers that capture the meaning of a piece of text, and finds the ones closest to a query. Approximate nearest-neighbour indexes such as HNSW let it search millions of vectors without comparing against every one.
What is chunking in RAG?: Splitting documents into smaller passages before embedding them, so a search returns the paragraph that answers the question rather than the whole document. Size and overlap matter: too small and a fact gets cut in half, too large and the match gets diluted.

Related material

elsewhere in the notebooks

№ 07 Anatomy of an AI agent On a loop that uses tools № 06 How a large language model works On guessing the next word № 01 What a CPU is, and how it works On the clock that never stops