Chapter 1 · About 10 minutes

Tokens

Before a model can read text, it cuts the text into pieces called tokens and swaps each piece for a number.

You will learn

  • Say what a token and a token id are
  • Predict when a word stays whole and when it splits
  • Explain how byte pair encoding builds the list of tokens

Text becomes tokens

See it on the map

28 characters, 6 tokens

A dot marks a space. The small number is the token id.

Type anything to see its tokens.

Story

A model cannot read letters. It reads Token One piece of text from a fixed list the model knows, often a whole word or part of one. Open the glossary, pieces of text from a fixed list called the Vocabulary The fixed list of every token a model knows. Ours has 4,096 tokens, and GPT-2 has 50,257. Open the glossary. Each token has a number, its Token id The number that stands for a token in the vocabulary. The model only ever sees these numbers. Open the glossary.

Our sentence, “Lily wanted to play with her”, is six tokens, one for each word. The model never sees the letters, only the six ids. Try typing your own sentence in the box.

Numbers

Our vocabulary has 4,096 tokens, with ids from 0 to 4,095. The 28 characters of our sentence become 6 ids.

Token Lily ·wanted ·to ·play ·with ·her
Id 665 408 266 324 329 336

GPT-2 uses the same method with a much bigger list of 50,257 tokens (Radford et al., 2019).

Formula

Tokenizing turns a text into a sequence of ids, each one from the vocabulary.

text  ↦  (t1,t2,…,tn),ti∈{0,1,…,4095}\text{text} \;\mapsto\; (t_1, t_2, \dots, t_n), \qquad t_i \in \{0, 1, \dots, 4095\}

For our sentence n=6n = 6.

Code

const ids = tokenizer.encode('Lily wanted to play with her');
// [665, 408, 266, 324, 329, 336]

How many tokens is the word "Tokenization" for our model?

Common words stay whole, rare words split

20 characters, 7 tokens

A dot marks a space. The small number is the token id.

Rare words are built from smaller pieces.

Story

Words the model saw often in training, like “·play” or “·happy” after a space, are one token each. A rare word is built from smaller pieces that are in the list. “Tokenization” becomes four pieces.

Nothing is ever left out. The smallest pieces go all the way down to single letters, so every word can still be written.

Numbers

“Tokenization is fun!” is 20 characters and 7 tokens. Small changes can matter too. “·played” is one token, but “·plays” is “·play” plus “s”.

A bigger vocabulary keeps more words whole, so the same text takes fewer tokens. A smaller one splits more often.

Formula

Turning the ids back into text gives exactly the text you started with.

decode⁡(encode⁡(x))=x\operatorname{decode}(\operatorname{encode}(x)) = x

Code

tokenizer.decode(tokenizer.encode('Tokenization is fun!'));
// 'Tokenization is fun!'

Like building blocks

Think of tokens as building blocks, with a big block for each common word and small blocks for everything else. Where this breaks. Real blocks look the same wherever they go. The same word can become a different block after a space or with a capital letter, as the next step shows.

Spaces and capitals count

27 characters, 8 tokens

A dot marks a space. The small number is the token id.

Look at the ids for Lily and for happy.

Story

The space before a word is part of its token. “Lily” at the very start and “·Lily” after a space are two different tokens with two different ids.

Capitals matter too. “·happy” is one token, but “·Happy” splits into two, because it was much rarer in training. To the model these are different pieces, and it has to learn that they mean nearly the same thing.

Numbers

Token Lily ·Lily ·happy ·H + appy
Id 665 405 376 308 + 3189

So “Lily was happy. Happy Lily!” is 27 characters and 8 tokens.

Formula

Capitals and spaces change the ids. The same word can get different ids.

encode⁡(Lily)=(665),encode⁡(⋅Lily)=(405)\operatorname{encode}(\text{Lily}) = (665), \qquad \operatorname{encode}(\text{·Lily}) = (405)

Code

tokenizer.encode('Lily'); // [665]
tokenizer.encode(' Lily'); // [405]

How the list was made

17 characters, 9 tokens

A dot marks a space. The small number is the token id.

Try an emoji or a long, rare word.

Story

The vocabulary is built before training, with a method called Byte pair encoding A way to build a vocabulary. Start with single bytes, then keep joining the pair of pieces that appears together most often. Open the glossary (Sennrich et al., 2016). Like GPT-2 (Radford et al., 2019), ours starts with the 256 possible bytes, the small codes a computer stores text in, so any text at all can be written. Then it finds the pair of pieces that sits side by side most often inside words and joins them into a new token. It repeats this until the list is full.

Numbers

Our list has 1 special token that marks the end of a story, 256 single bytes and 3,839 joined pieces, 4,096 in all. The very first join was a space followed by “t”, as in “·the” and “·to”.

Our stories had few numbers, so in “·123” the space and each digit are tokens of their own. An emoji the model never saw is written as its raw bytes, four for 😀.

Formula

Each round joins the pair of neighboring pieces (a,b)(a, b) that appears most often in the training text, then counts again.

(a∗,b∗)=arg⁡max⁡(a, b)count⁡(a,b)(a^{*}, b^{*}) = \arg\max_{(a,\, b)} \operatorname{count}(a, b)

Code

tokenizer.encode('😀').map((id) => tokenizer.tokenText(id));
// ['<0xF0>', '<0x9F>', '<0x98>', '<0x80>']

Recap

  • A model reads tokens, pieces of text from a fixed list called the vocabulary.
  • Each token has an id number, and the model only ever sees those numbers.
  • Common words are one token. Rare words are built from smaller pieces.
  • Byte pair encoding builds the list by merging the most common pairs of pieces.

Go deeper