notebook № 066 plates · 15 min to read, longer to play
How a large language model works
On guessing the next word
A large language model does one thing: it guesses the next piece of text, again and again. This notebook takes that one trick apart: how text becomes numbers, how a guess is made and picked, how a model learns from text, and why it sometimes forgets what you said.
plate I · tokens
Words into numbers
A model can’t read letters. It only does arithmetic, so the text has to become numbers first. A tokenizer does that: it chops text into pieces called tokens and gives each piece a number.
The pieces aren’t chosen by hand. The tokenizer learns them by counting: find the pair of pieces that appears together most often, glue it into a new piece, and repeat thousands of times. Common words end up whole. Rare words are spelled out of smaller bits.
The one on the right learned its pieces a moment ago, in your browser, from the text of this notebook. Real tokenizers learn tens of thousands of pieces from a mountain of text, and in English a token is about three quarters of a word.
Drag “pieces learned” to 0, then slowly up. Watch common words snap together first, and the long word hold out longest.
plate II · the one job
One job: the next word
Give a model some tokens and it gives back one thing: a probability for every possible next token. Not an answer. A list of odds.
Something then has to pick one. Always taking the top word sounds safe, but it makes dull, loopy text. So we roll a die weighted by the odds. Temperature bends the odds first: low makes the favourite even more likely, high flattens the list so long shots get a chance. Top-k crosses out everything but the k best.
The picked token is added to the text, and the whole thing runs again for the next one. A whole essay is that loop, a few thousand times.
Set temperature to 0 and pick ten times. Then set it to 2. Which one would you rather read a story from?
plate III · learning
A model you can train
Where do the odds come from? From reading. Below is the simplest language model there is: it reads this notebook and counts, for every run of a few letters, which letter came next. To write, it looks at its last few letters and rolls the die from plate II.
With one letter of memory it babbles. Give it three and words appear; give it five and it starts quoting the page back at you. The rose letters are where it found itself somewhere it had never been, and had to fall back on a shorter memory.
That last part is the whole difference. A counting model only knows what it has seen exactly. A real LLM swaps the counting table for a neural network with billions of adjustable numbers, trained on trillions of words to make its guesses a little less wrong each time. It doesn’t memorise the text; it learns patterns that work on sentences it has never seen.
Training happens in two steps. First it reads a large slice of the internet and learns to continue anything. Then it is trained on examples of good answers, so that it continues a question with a helpful reply instead of three more questions.
Switch to your text, paste a page of something with a strong voice, and set memory to 4. Then press write a few times.
plate IV · embeddings
Meaning as direction
Inside the model, a token isn’t a number for long. It becomes a list of numbers, an embedding: a point in a space with thousands of directions. The model places these points itself while it learns, and words used in similar sentences end up close together.
The strange part is that directions mean things too. The step from man to king points the same way as the step from woman to queen. So king − man + woman lands near queen. Nobody designed that; it fell out of the training.
This map is drawn by hand in two dimensions so you can see it. A real one has thousands, and the arithmetic only works roughly, but the idea holds.
plate V · attention
Paying attention
A word on its own is ambiguous. In “the animal didn’t cross the street because it was too tired”, what is it? You knew instantly. The model needs a way to find out.
That way is attention, the idea at the heart of the transformer, the design behind every modern LLM. Each word looks back at the words before it and decides how much each one matters, then mixes their meaning into its own. After that, it carries a bit of animal inside.
A model does this many times over: dozens of heads per layer, each looking for something different, and dozens of layers stacked up. A word can only look back, never ahead; the future hasn’t been written yet.
plate VI · memory
The context window
A model doesn’t remember your chat. Every time it replies, the app sends it the whole conversation again, from the first message, and it reads it all from scratch. Everything it knows about you sits in that text.
The text has a size limit, the context window: from a few thousand tokens in early models to a million or more today. When a chat outgrows it, something has to go, usually the oldest messages. That is when a model “forgets” what you told it an hour ago.
It also explains the bill. You pay for the tokens the model reads, and every turn rereads everything, so long chats get more expensive with every message. Attention adds to it: every token looks at every earlier one, so twice the text is roughly four times the comparisons.
Agents live and die by this desk. The memory notebook is about what to put on it.
Questions people ask
- What is a token in an LLM?
The unit a model reads and writes: a common word, a piece of a rarer word, or a punctuation mark. Tokenizers such as byte-pair encoding learn these pieces from text. In English a token is roughly three quarters of a word.
- What does temperature do in an LLM?
It reshapes the probabilities before the next token is picked. Low temperature sharpens them, so the model almost always takes its top guess; high temperature flattens them, so less likely words get chosen more often. Plate II lets you slide it and watch.
- What is a context window?
The most text, counted in tokens, that a model can take into account at once: your prompt, the conversation so far and its own answer. Anything outside the window is invisible to the model, which is why long chats eventually lose their beginning.
Related material
elsewhere in the notebooks
further reading
- Neural networks 3Blue1Brown The best visual introduction there is, all the way to transformers and attention.
- The Illustrated Transformer Jay Alammar Plate V, drawn in full: queries, keys, values and many heads.
- Let’s build GPT: from scratch, in code, spelled out Andrej Karpathy Plate III grown up: a real transformer, trained live, in two hours of code.
- nanoGPT Andrej Karpathy The smallest code that trains a real GPT. Readable in an evening.
- Attention Is All You Need Vaswani et al., 2017 The paper that introduced the transformer. Shorter than you’d think.