Prediction

Story

After attention, each token’s list also passes through a small network of its own. Each step adds to the list instead of replacing it, and the whole block repeats 4 times. A last norm tidies the list.

Then the last token’s list is compared with every row of the embedding table, using a dot productDot product A way to compare two lists of numbers. Multiply the numbers in matching places and add up the results. Open the glossary. That gives each of the 4,096 tokens a score, called a logitLogit The raw score a model gives one possible next token, before softmax turns the scores into probabilities. It can be any size, even below zero. Open the glossary.

Numbers

The final list for “·her” starts 0.82, 2.46 and -3.15. Its dot product with the row for “·ball” is 2.60, the highest of all 4,096 scores. “·friends” comes next at 2.48.

Using the same table twice is called tying the weights (Press and Wolf, 2017). In our model it saves 524,288 numbers. GPT-2 does it too, like the first GPT (Radford et al., 2018, Radford et al., 2019).

Formula

With $\mathbf{u}$ the last token’s list after the final norm, the scores are

so token $t$ gets the dot product of its own embedding row with $\mathbf{u}$.

Code

const trace = forward(model, ids);
trace.logits; // 6 × 4,096, one row of scores per token
const scores = rowView(trace.logits, ids.length - 1); // the last row is the guess

Story

Scores are not chances yet. SoftmaxSoftmax A way to turn any list of scores into numbers between 0 and 1 that add up to 1. Bigger scores get bigger shares. Open the glossary turns them into chances between 0 and 1 that add up to 1, just as it did inside attention.

Then a token is picked at random, using those chances. This is called samplingSampling Picking the next token at random, where each token's chance of being picked is its probability. Open the glossary. Press Sample a few times. The same chances can give a different word each time.

Numbers

“·ball” gets 11.5%, “·friends” 10.2% and “·toy” 8.5%. The 50 best tokens hold 80% of the chance together. The other 4,046 tokens share the last 20%, so a rarer word gets picked about one time in five.

Formula

The sum runs over all 4,096 tokens.

Code

const probs = probabilities(scores, 1); // softmax at temperature 1
const next = sample(probs, createRng(1)); // one token id, picked at random

Story

Before softmax, every score is divided by a number called the temperatureTemperature A number the scores are divided by before softmax. Low makes the top guess win more often, and high spreads the chances out. Open the glossary.

A low temperature makes the gaps between scores bigger, so the top guess wins more often and the text gets safer. A high temperature shrinks the gaps, so rare words get picked more and the text gets wilder. At 1, the chances are the model’s own.

Numbers

At temperature 0.1, “·ball” gets 72% and “·friends” 22%. At 0.5, “·ball” gets 27%. At 2, “·ball” gets only 1.5%, and the 50 best tokens hold just 22% together.

Formula

With temperature $\tau$,

As $\tau$ goes to 0, all the chance goes to the top score. As $\tau$ grows, the chances move toward equal.

Code

probabilities(scores, 0.1); // sharp, "·ball" gets 0.72
probabilities(scores, 2); // flat, "·ball" gets 0.015

Story

GreedyGreedy decoding Always picking the single most likely next token. It gives the same text every time. Open the glossary decoding skips the draw and always takes the top guess. It gives the same text every time, which often ends up dull and repeats itself (Holtzman et al., 2020 ).

Top-kTop-k sampling Sampling only from the k most likely tokens, after dropping all the others. Open the glossary picks only from the k likeliest tokens. Top-pTop-p sampling Sampling only from the fewest most likely tokens whose probabilities add up to at least p. Also called nucleus sampling. Open the glossary picks from the fewest likeliest tokens whose chances add up to p. Both stop very rare words from sneaking in.

Numbers

With top-k and k = 5, only “·ball”, “·friends”, “·toy”, “·doll” and “·new” stay in, and their chances are scaled up to add to 1.

With top-p and p = 0.9, the number kept depends on how sure the model is. At temperature 1 it keeps about 200 tokens. At temperature 0.5 it keeps only 7.

Formula

Top-p keeps the smallest set $V_p$ of likeliest tokens with $\sum_{t \in V_p} P(t) \ge p$, then rescales.

Code

argmax(scores); // greedy, the id of "·ball"
nextTokenDistribution(scores, { temperature: 1, topK: 5 });
nextTokenDistribution(scores, { temperature: 1, topP: 0.9 });