Chapter 6 · About 10 minutes
Retrieval and tools
A model only knows what it learned and what is in its window, so programs around it look things up and do sums, and put the results in the text.
You will learn
- Say why a model gets facts and sums wrong, and what retrieval adds
- Follow the steps of a tool call, from the model's text to the program's answer
- Explain what makes a model an agent
The last chapter showed that a sentence in the prompt can change the answer. This chapter turns that into a plan. If the model is missing something, put it in the text.
What the model cannot know
A choice of texts. For the chosen one, its tokens and bars for the model's five likeliest next tokens with their chances.
Story
A model has only two places to find an answer. One is what it learned in training, stored in its numbers. The other is the text in its window. It cannot open a book or a web page, and it has no calculator.
Our model learned from simple stories, so it does not know that Paris is the capital of France, and it cannot add. Large models know far more, but they still forget things, mix them up, and get long sums wrong. They also know nothing that happened after their training.
Numbers
After “The capital of France is”, the top guesses are “a” with 12% and “not” with 11%. After the apples, “a” gets 17%, “two” 14% and “three” only 7%.
Look it up first
A choice of texts. For the chosen one, its tokens and bars for the model's five likeliest next tokens with their chances.
Story
Retrieval Looking up passages that might help, and putting them in the prompt before the model answers. Open the glossary means looking up text that might help before the model answers. A program searches a collection of documents, picks the passages that match the question best, and puts them in the prompt. Then the model answers with that text in its window (Lewis et al., 2020).
Our model shows the idea on a small scale. Given a note that says where the key is, it guesses “rug”. Without the note, it guesses “tree” or “bed”, like most stories would.
Numbers
With the note, “rug” gets 46%. Without it, “tree” gets 30% and “bed” 25%, and “rug” is not in the top five.
Our simplification
Real retrieval systems often find passages by comparing lists of numbers for the question and for each passage, much like the embeddings in an earlier chapter. Here the note is simply written into the prompt by hand.
Asking a program for help
- TextLily has 2 apples. Tom gives her 1 more. Now Lily has
- Model
[Calculator(2 + 1)] - ProgramRuns the calculator and adds its answer, 3, to the text.
- ModelReads it all and goes on with 3 apples.
4 steps in order, each saying who acts and what they write.
Story
A Tool call A request the model writes as text, such as a sum for a calculator. A program runs it and adds the answer to the text. Open the glossary lets a model use a program, such as a calculator or a search engine. The model is trained to write a request in a set format, as ordinary text. The program around the model watches for that format, runs the tool, and adds the result to the text. Then the model goes on guessing, with the answer in its window.
The model never runs anything itself. It only writes text. A model can learn when a call is worth making from a few examples of each tool (Schick et al., 2023).
Numbers
Writing the answer into the text is not enough for our tiny model. Even after “2 plus 1 is 3.” is added before the question, as in the third tab of the first figure, it still guesses “two” with 14% and “three” with only 7%. Using a tool’s answer is a skill a model has to learn, and ours never did.
Agents repeat the loop
- ModelWrites what it thinks it should do next.
- ModelWrites an action, such as a search or a click.
- ProgramCarries out the action and adds what happened to the text.
Then back to the first step, until the task is done.
3 steps in order, each saying who acts and what they write. The last step leads back to the first.
Story
An Agent A model that repeats tool calls on its own, reading each result and deciding what to do next, until a task is done. Open the glossary is a model that repeats the tool loop on its own. It writes what it thinks it should do, writes an action, reads what the action returned, and goes on, until the task is done (Yao et al., 2023).
Every step lands in the same window. A long task fills it up, so agents have to keep notes short or drop old steps, the same limit you saw in the last chapter.
Common myth
When a chat app shows a model searching the web, the model is not browsing. It writes a search request as text. A program runs the search and pastes the results into the prompt, and the model reads them like any other text.
A model is asked about yesterday's weather. Its training ended a year ago, and nothing is looked up. What can it do?
It can only guess, or say it does not know. Yesterday is not in its training and not in its window. Retrieval or a tool call would put the answer in the text first.
Recap
- A model has no way to look anything up. It only has what it learned and what is in its window.
- Retrieval finds text that might help and puts it in the prompt before the model answers.
- For a tool call, the model writes a request as text, a program runs it, and the answer is added to the text.
- An agent repeats this loop, writing what to do next, reading the result, and going on until the task is done.
Go deeper
- Lewis et al., 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- Schick et al., 2023. Toolformer: Language Models Can Teach Themselves to Use Tools
- Yao et al., 2023. ReAct: Synergizing Reasoning and Acting in Language Models