Chapter 5 · About 10 minutes

Context and prompts

A model only reads the text in front of it, so what you put in that text, the prompt, steers every guess.

You will learn

  • Say what a context window is and what happens to text that falls out of it
  • Explain why adding a sentence to a prompt can change the answer
  • Tell a model that continues text from one trained to follow instructions

The first chapters followed one sentence through the model. This one asks a practical question. If the model only ever guesses the next token, how do people get it to do what they want? The answer starts with what the model can read.

The model only reads a window

Sara had a cat named Fluffy. Every morning, Sara gave milk to

14 tokens in all. The model can read 128, so it reads all 14, and the note is inside its window.

  • Fluffy83%
  • her7%
  • the2%
  • Spot1%
  • Sara1%
Add sentences between the note and the question, and watch the note fall out of the window.

Story

A model reads a fixed number of tokens at a time, called its Context window How many tokens a model can read at once. Text before that is cut off. Ours holds 128 tokens. Open the glossary. Anything earlier is cut off, as if it was never written.

Our model reads that Sara’s cat is called Fluffy in the first sentence, and guesses “Fluffy” for who gets the milk. Add sentences in between, and the note slips further back. Once it falls out of the window, the model has never heard of Fluffy.

Numbers

Our model’s window is 128 tokens, because it learned one position embedding for each of 128 places. With the note right before the question, the text is 14 tokens and “Fluffy” gets 83%. With 5 sentences in between, it is 119 tokens and “Fluffy” drops to 43%. With 6, the text is 140 tokens and the note is cut off. “Fluffy” is gone, and “her” with 44% and “the” with 43% are nearly tied.

GPT-2 has a window of 1,024 tokens (Radford et al., 2019). Many large models today fit far more, but the rule is the same. What falls outside is not read.

Formula

With a window of ww tokens, the guess for the next token only depends on the last ww tokens.

Pmodel(tn+1∣t1,…,tn)=Pmodel(tn+1∣tn−w+1,…,tn),w=128P_{\text{model}}(t_{n+1} \mid t_1, \dots, t_n) = P_{\text{model}}(t_{n+1} \mid t_{n-w+1}, \dots, t_n), \qquad w = 128

Code

const window = model.config.contextLength; // 128
const trace = forward(model, ids.slice(-window)); // only the last 128 tokens go in

A text is 200 tokens long. Our model reads it. How many of its first tokens does the model never see?

A prompt is the start of the text

Sara had a cat named Fluffy. Every morning, Sara gave milk to
  • Fluffy83%
  • her7%
  • the2%
  • Spot1%
  • Sara1%
The same question, with and without one sentence before it.

Story

The text you give a model is called a Prompt The text given to a model. To the model it is the start of the text, and it guesses what comes next. Open the glossary. To the model, it is simply the start of the text, and its job is to guess what comes next.

So everything in the prompt steers the guesses. Without the first sentence, the model has no reason to think of Fluffy, and guesses “her” or “the”. One extra sentence changes the answer.

Numbers

With the note, “Fluffy” gets 83%. Without it, “her” gets 39% and “the” gets 26%, and “Fluffy” is not in the top five.

Examples in the prompt

Ben said, "Zog." Mia said, "Zog." Tim said, "
  • Z20%
  • I9%
  • Don6%
  • T5%
  • No5%
Zog is a made-up word. Two examples are enough for the model to try it.

Story

A prompt can also show the model what to do, by giving a few examples first. Here Ben and Mia both say a made-up word, “Zog”. When Tim speaks, the model’s top guess starts with “Z”. “Zog” is not a real word, so the model is unlikely to have seen it in training. It copied the pattern from the prompt.

Large models do this with whole tasks. Show a few questions with their answers, and they answer the next question the same way (Brown et al., 2020). Nothing in the model changes. The examples are just more text to read.

Numbers

After the two examples, “Z” is the top guess with 20%. Without them, the top guesses are “I” with 20% and “No” with 12%, and “Z” is not in the top five.

Researchers found attention heads that do this kind of copying, called induction heads. Each one spots a token that appeared before and looks at what came right after it last time (Olsson et al., 2022).

Trained to follow instructions

Write a poem about a cat.
  • It13%
  • The11%
  • I9%
  • You7%
  • They6%
Our model reads an instruction as the start of a story.

Story

Ask our model to write a poem, and it does not. It guesses “It”, “The” or “I”, the way a story might go on after that sentence. It only ever learned to continue children’s stories.

Chat models start the same way, as models that continue text. Then they are trained further on examples of instructions with good answers, and on people’s rankings of their answers (Ouyang et al., 2022). That is what turns a model that continues text into an assistant that answers. Underneath, it still guesses one token at a time.

Numbers

After “Write a poem about a cat.”, the top guesses are “It” with 13%, “The” with 11% and “I” with 9%.

Common myth

A chat model does not remember your conversation by itself. Each time, the app sends the conversation so far as part of the prompt. Apps that seem to remember older chats put notes from them into the prompt too. When a conversation outgrows the window, the oldest part has to be cut or summed up.

Recap

  • A model reads only the last tokens that fit in its context window. Ours fits 128, GPT-2 fits 1,024.
  • Anything outside the window is gone. The model cannot use what it cannot read.
  • A prompt is the start of the text. Facts and examples in it change every guess after it.
  • Chat models are trained further to follow instructions. Our model only continues stories.

Go deeper