Questions and answers
Short answers to questions people often ask. Each one links to the step that explains it, and you can search the whole site too.
Do the attention weights explain why the model chose a word?
Not on their own. They show where a head looked, not why the model decided what it did. Studies of other kinds of models found that quite different weights can lead to the same prediction, and others argued the weights can still be useful clues. Treat them as a clue to test, not an explanation.
Read more in Many heads, many layers (Attention).
How is this different from a chat model?
A chat model follows the same plan with far more layers and learned numbers, trained on far more text. It is then trained further to follow instructions, using examples written by people and their ratings of its answers. This site covers the plan they share.
Read more in Inside, a transformer does the guessing (What a language model does).
Does the model know what words mean?
Nobody tells it. The model only sees token ids, and it learns numbers for each one that help it guess the next token. Along the way, words used in similar ways end up with similar numbers, such as girl and boy. Whether that counts as knowing what a word means is a question this site leaves open. What it shows is what the numbers do.
Read more in Similar words get similar numbers (Embeddings and position).
How does a model write a whole answer?
One piece at a time. The model gives every possible next piece a chance, one piece is picked, it is added to the text, and the model guesses again. Chatbots repeat this hundreds of times to write a reply.
Read more in Then it does it again (What a language model does).
Does this site send what I type anywhere?
No. The model runs in your browser, so the site does nothing with your text on any server. The playground and search do put your text in the page's address, so it can be shared. If you open, reload or share that address, it goes to the site's host like any web address.
A transformer is the kind of model behind today's chatbots. It turns text into numbers and passes them through a stack of layers where each word gathers information from the other words. In models like GPT, each word looks only at the words before it. At the end comes a score for every possible next piece of text.
Read more in Inside, a transformer does the guessing (What a language model does).
Why does each layer add to a token's numbers instead of replacing them?
This is called a residual connection. Each layer adds its result to the token's numbers, so the earlier numbers are kept, not overwritten, and each layer only has to learn a change. This also makes deep stacks of layers much easier to train.
Read more in One score for every token (Prediction).
- He et al., 2016, Abstract
We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions.
Why can the same text give a different answer each time?
The model gives chances, not one answer. The next piece is drawn at random using those chances, so two runs can pick different pieces and then go different ways. The temperature controls how adventurous the draw is. Greedy decoding always takes the top guess and gives the same text every time.
Read more in Scores become chances (Prediction), Turning the temperature up or down (Prediction) and Greedy, top-k and top-p (Prediction).
Why does attention divide the scores by the square root of the list length?
A dot product adds up one product per number, so the longer the lists, the bigger the scores tend to get. Very big scores make softmax put almost all the weight on one token, and then training learns very slowly. Dividing by the square root of the length keeps the scores in a range where softmax still spreads its weight. In our model each head's query and key lists have 32 numbers, so the scores are divided by about 5.66.
Read more in Questions, name tags and backpacks (Attention).
- Vaswani et al., 2017, §3.2.1 and footnote 4
We suspect that for large values of d_k, the dot products grow large in magnitude, pushing the softmax function into regions where it has extremely small gradients.
Why can a token not look at the tokens after it?
The model learns by guessing each next token. If a token could see the tokens after it, it could copy the answer instead of learning to guess. The causal mask hides later tokens by setting their scores to minus infinity, so their weight after softmax is exactly 0.
Read more in Words need context (Attention) and Softmax turns scores into weights (Attention).
- Vaswani et al., 2017, §3.2.3
We need to prevent leftward information flow in the decoder to preserve the auto-regressive property.
Why is the model on this site so small?
So it runs in your browser and every number can be shown. It has about 1.3 million learned numbers, 4 layers and a vocabulary of 4,096 tokens. It was trained on simple children's stories, and research has shown that models with under 10 million learned numbers can still write fluent short stories like these.
Read more in Inside, a transformer does the guessing (What a language model does).
- Eldan and Li, 2023, Abstract
We show that TinyStories can be used to train and evaluate LMs that are much smaller than the state-of-the-art models (below 10 million total parameters)
Why does the model read tokens instead of letters or whole words?
Whole words would need an endless list, and new or rare words would be missing. Single letters would make texts very long. Tokens sit in between. Common words are one token, and rare words are built from smaller pieces, so any text can still be written.
Read more in Common words stay whole, rare words split (Tokens) and How the list was made (Tokens).
- Sennrich et al., 2016, Abstract
making the NMT model capable of open-vocabulary translation by encoding rare and unknown words as sequences of subword units
How does the model know the order of the words?
Attention on its own does not see order. So the model also learns a list of numbers for each place in the text and adds it to each token's list. The same word then looks a little different in a different place.
Read more in Each place gets numbers too (Embeddings and position).
- Vaswani et al., 2017, §3.5
Since our model contains no recurrence and no convolution, in order for the model to make use of the order of the sequence, we must inject some information about the relative or absolute position of the tokens in the sequence.