Glossary
- Agent
A model that repeats tool calls on its own, reading each result and deciding what to do next, until a task is done.
- Attention
The step where each token gathers information from itself and the tokens before it, taking more from the ones that match what it is looking for.
- Attention head
One of several attention steps that run side by side in a layer, each with its own queries, keys and values.
- Byte pair encoding
A way to build a vocabulary. Start with single bytes, then keep joining the pair of pieces that appears together most often.
- Causal mask
The rule that a token may only look at itself and the tokens before it, never at later ones.
- Chain of thought
Steps a model writes before its answer. Each step joins the text it reads for the next guess.
- Context window
How many tokens a model can read at once. Text before that is cut off. Ours holds 128 tokens.
- Dot product
A way to compare two lists of numbers. Multiply the numbers in matching places and add up the results.
- Embedding
The list of numbers a model uses for a token. Each token id picks one row of a learned table.
- Greedy decoding
Always picking the single most likely next token. It gives the same text every time.
- Key
The list of numbers that says what a token has, like a name tag. Queries are matched against keys.
- Language model
A program that reads text and gives every possible next piece of text a probability.
- Logit
The raw score a model gives one possible next token, before softmax turns the scores into probabilities. It can be any size, even below zero.
- Parameter
One of the numbers inside a model that is set during training. Our tiny model has about 1.3 million.
- Position embedding
A list of numbers for each place in the text, added to the token's list so the model knows the order.
- Probability
A number from 0 to 1 that says how likely something is. The probabilities of all options add up to 1.
- Prompt
The text given to a model. To the model it is the start of the text, and it guesses what comes next.
- Query
The list of numbers a token uses to ask what it is looking for, like a question card.
- Reasoning model
A model trained to write long reasoning before it answers, often with rewards for right answers.
- Retrieval
Looking up passages that might help, and putting them in the prompt before the model answers.
- Sampling
Picking the next token at random, where each token's chance of being picked is its probability.
- Softmax
A way to turn any list of scores into numbers between 0 and 1 that add up to 1. Bigger scores get bigger shares.
- Temperature
A number the scores are divided by before softmax. Low makes the top guess win more often, and high spreads the chances out.
- Token
One piece of text from a fixed list the model knows, often a whole word or part of one.
- Token id
The number that stands for a token in the vocabulary. The model only ever sees these numbers.
- Tool call
A request the model writes as text, such as a sum for a calculator. A program runs it and adds the answer to the text.
- Top-k sampling
Sampling only from the k most likely tokens, after dropping all the others.
- Top-p sampling
Sampling only from the fewest most likely tokens whose probabilities add up to at least p. Also called nucleus sampling.
- Training
Adjusting a model's parameters, a little at a time, so its predictions on example text get better.
- Transformer
The design behind most modern language models. It passes text through layers of attention and feed-forward steps.
- Value
The list of numbers a token shares when others look at it, like a backpack.
- Vector
A list of numbers. Ours have 128 numbers each.
- Vocabulary
The fixed list of every token a model knows. Ours has 4,096 tokens, and GPT-2 has 50,257.