Embeddings and position

Story

An id like 665 is only a label, and its size means nothing. So each id picks one row from a big table, a list of 128 numbers called an embeddingEmbedding The list of numbers a model uses for a token. Each token id picks one row of a learned table. Open the glossary. A token gets the same list every time it appears.

These numbers start out random. Training nudges them, bit by bit, until they help the model guess well. Tap a cell to see one number.

Numbers

The table has 4,096 rows, one for each token, and 128 numbers in each row. That is 524,288 numbers, about 2 in every 5 numbers the model learned.

The first five numbers for “Lily” are 0.008, 0.006, -0.046, 0.001 and -0.209. A list of numbers like this is called a vectorVector A list of numbers. Ours have 128 numbers each. Open the glossary.

Formula

Looking up a row of the embedding table $W_E$ gives the token’s vector.

The lookup is the same as multiplying a one-hot vector, all zeros except a 1 at $t_i$, by $W_E$.

Code

const lily = rowView(model.wte, 665); // 128 numbers
trace.tokenEmbeddings; // one row per token, 6 × 128

Story

Nobody tells the model what words mean. Still, after training, words used in similar ways end up with similar lists (Mikolov et al., 2013). “Girl” lands near “boy”, “red” near the other colors, and “ran” near “run” and “walked”.

The model found this on its own, because it helps to guess the next word. Pick a word to see its closest tokens.

Numbers

The score is the cosine similarity. 1 means the two lists point the same way, 0 means they have nothing in common and -1 means they point opposite ways.

“·girl” and “·boy” score 0.74. “·three” and “·3” score 0.88, so the model uses the word and the digit almost alike. (“·3” is one of the few numbers with a token of its own.) “·Lily” and “Lily” at the start of a text score 0.76, even though the Tokens chapter showed they are two different tokens.

Formula

Cosine similarity divides the dot productDot product A way to compare two lists of numbers. Multiply the numbers in matching places and add up the results. Open the glossary of two vectors by their lengths, so only their direction counts.

Code

const dot = (a, b) => a.reduce((sum, x, k) => sum + x * b[k], 0);
const cosine = (a, b) => dot(a, b) / Math.sqrt(dot(a, a) * dot(b, b));
cosine(rowView(model.wte, 477), rowView(model.wte, 497)); // ·girl and ·boy, 0.74

Story

To compare two lists, multiply the numbers in matching places and add up the results. This is the dot productDot product A way to compare two lists of numbers. Multiply the numbers in matching places and add up the results. Open the glossary.

It is big when both lists have large numbers in the same places with the same signs. It is near zero when they have nothing in common, and below zero when the signs disagree. The dot product comes back in the attention chapter, where it decides which tokens look at which.

Numbers

Take (1, 2, 3) and (4, 0, -1). Multiply place by place to get 4, 0 and -3. Add them up to get 1.

Formula

For our embeddings $d = 128$.

Code

const dot = (a, b) => a.reduce((sum, x, k) => sum + x * b[k], 0);
dot([1, 2, 3], [4, 0, -1]); // 1

Story

So far, “play” gets the same list wherever it appears. But order matters. “Lily saw Max” and “Max saw Lily” use the same words.

So the model also learns one list of 128 numbers for each place in the text, called a position embeddingPosition embedding A list of numbers for each place in the text, added to the token's list so the model knows the order. Open the glossary. It adds the place’s list to the token’s list, number by number. That sum is what the first layer reads.

Numbers

Our model has 128 position lists, one for each place, so it reads at most 128 tokens at a time. Nearby places get similar lists. Places 2 and 3 score 0.98, while places 2 and 51 score -0.22.

Our model learns these lists, like GPT (Radford et al., 2018) and GPT-2. The original transformer used fixed sine and cosine waves instead (Vaswani et al., 2017).

Formula

Places count from 1 but table rows count from 0, so place $i$ uses row $i-1$.

Code

trace.positionEmbeddings; // one row per place, 6 × 128
trace.embeddings; // tokenEmbeddings + positionEmbeddings