The whole model
Tap any part to see what it does, then read more about it below the map. Or press Next for a guided tour.
Tap a part, or press Next for a tour.
The transformer drawn as stacked blocks, from the inputs at the bottom to the output probabilities at the top. Each block is a button that explains that part.
The paper is listed on the references page. GPT-2 keeps only the right-hand side, because it continues text instead of translating it.
The Llama style view shows our second model, with the changes most models since have made. LLaMA brought RMSNorm (Zhang and Sennrich, 2019), rotary positions (Su et al., 2024) and the SwiGLU feed forward (Shazeer, 2020), all in one model (Touvron et al., 2023). Llama 2's larger models added grouped-query attention (Ainslie et al., 2023, Touvron et al., 2023).
Prompts, retrieval, tools and reasoning all use this same model. See how in Context and prompts and the chapters after it.
The list below describes every part.
Every part as a list
The same parts as the map, in order, for reading without it.
- Inputs. Your text, split into tokens. Each token is an id number from the vocabulary. In the paper the right side read the translation so far, shifted one place right so each position predicts the next word.
- Embedding. Each id picks one row from a table of learned numbers. That row is the token's starting list of numbers.
- Position. Each position also gets a list of numbers, added to the token's list. That is how the model knows the order. The paper used fixed sine and cosine waves. GPT-2 learns a table of positions instead.
- Encoder attention. In the encoder, each word of the source sentence can look at every other word, before or after it.
- Add and norm. Add, called a residual connection, keeps what came in and puts the new result on top, so earlier numbers are kept. Norm rescales each token's numbers to a steady range. GPT-2 moves the norm to the start of each step, before attention and before the feed-forward layer.
- Feed forward. Each token's numbers go through a small network on their own, without looking at other tokens. The paper used ReLU and grew 512 numbers to 2,048.
- Masked attention. Each token looks at itself and the tokens before it, and pulls in what it needs. Masked means it cannot look ahead.
- Cross-attention. Here the translation side looks at the encoded source sentence. GPT-2 has no source sentence, so it drops this step.
- Repeated blocks. Everything inside the frame is one block. The model repeats it, and each block refines the numbers further.
- Linear. Turns each token's numbers into one score for every token in the vocabulary. It reuses the embedding table.
- Softmax. Turns the scores into chances between 0 and 1 that add up to 1.
- Output. The model's guess, a chance for every possible next token. Pick one, add it to the text, and run again.
- Final norm. One more norm after the last block, before the scores are made. GPT-2 added it. The paper has none.