References

Everything this site draws on, grouped by topic. Each chapter also lists its own sources under Go deeper. Today's language models goes past our tiny model, to the ideas behind the models people use now.

The transformer

  1. Ba, J. L., Kiros, J. R., and Hinton, G. E. (2016). Layer Normalization. arXiv preprint. arXiv:1607.06450v1.

    Layer normalization, which rescales each token's numbers to a steady range. Every transformer block uses it.

  2. He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep Residual Learning for Image Recognition. Proceedings of CVPR 2016. arXiv:1512.03385v1.

    Residual connections. Each layer adds its result to what came in, which lets very deep networks train. Every transformer block uses them.

  3. Hendrycks, D. and Gimpel, K. (2016). Gaussian Error Linear Units (GELUs). arXiv preprint. arXiv:1606.08415v5.

    GELU, the smooth activation that GPT-2 and our model use in the feed-forward layer.

  4. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., and 2 others (2017). Attention Is All You Need. Advances in Neural Information Processing Systems 30 (NIPS 2017). arXiv:1706.03762v7.

    Introduced the transformer, built on attention alone, for translation. Its Figure 1 is the diagram our map redraws.

  5. Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. (2018). Improving Language Understanding by Generative Pre-Training. OpenAI.

    GPT, a decoder-only transformer pre-trained to predict the next token on a large collection of books, then fine-tuned for other tasks.

  6. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019). Language Models are Unsupervised Multitask Learners. OpenAI.

    GPT-2. A bigger GPT trained on web pages, with the norm moved to the start of each step. Our tiny model follows its design.

  7. Zhang, B. and Sennrich, R. (2019). Root Mean Square Layer Normalization. Advances in Neural Information Processing Systems 32 (NeurIPS 2019). arXiv:1910.07467v1.

    RMSNorm. Divides by the root mean square and scales, without taking the mean away.

  8. Shazeer, N. (2020). GLU Variants Improve Transformer. arXiv preprint. arXiv:2002.05202v1.

    Gated feed forwards such as SwiGLU, where one projection opens or closes another.

  9. Xiong, R., Yang, Y., He, D., Zheng, K., Zheng, S., Xing, C., and 4 others (2020). On Layer Normalization in the Transformer Architecture. Proceedings of ICML 2020. arXiv:2002.04745v2.

    Shows why putting the norm before each step, as GPT-2 does, trains more smoothly than the original order.

Tokens and embeddings

  1. Bengio, Y., Ducharme, R., Vincent, P., and Jauvin, C. (2003). A Neural Probabilistic Language Model. Journal of Machine Learning Research 3, 1137 to 1155.

    An early neural network that predicts the next word using learned word vectors.

  2. Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013). Efficient Estimation of Word Representations in Vector Space. ICLR 2013 Workshop. arXiv:1301.3781v3.

    Word vectors learned from text, where words used alike get similar vectors.

  3. Sennrich, R., Haddow, B., and Birch, A. (2016). Neural Machine Translation of Rare Words with Subword Units. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL 2016). arXiv:1508.07909v5. doi:10.18653/v1/P16-1162.

    Adapted byte pair encoding from data compression to split rare words into smaller pieces.

Attention

  1. Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., and Sanghai, S. (2023). GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. Proceedings of EMNLP 2023. arXiv:2305.13245v3.

    Grouped-query attention. Groups of query heads share one key and value head, which saves memory while the model generates text.

Prediction and sampling

  1. Shannon, C. E. (1951). Prediction and Entropy of Printed English. Bell System Technical Journal 30(1), 50 to 64. doi:10.1002/j.1538-7305.1951.tb01366.x.

    People guessing the next letter of English text, an early look at next-token prediction.

  2. Press, O. and Wolf, L. (2017). Using the Output Embedding to Improve Language Models. Proceedings of EACL 2017. arXiv:1608.05859v3.

    Showed that a language model can reuse its input embedding table to score the next token, with fewer numbers to learn.

  3. Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. (2020). The Curious Case of Neural Text Degeneration. ICLR 2020. arXiv:1904.09751v2.

    Introduced top-p (nucleus) sampling, and showed that always taking the likeliest token gives dull, repetitive text.

Training and data

  1. Eldan, R. and Li, Y. (2023). TinyStories: How Small Can Language Models Be and Still Speak Coherent English?. arXiv preprint. arXiv:2305.07759v2.

    Short, simple stories written by GPT-3.5 and GPT-4, small enough for a tiny model to learn from. Our model trains on them.

Today's language models

  1. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., and 25 others (2020). Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems 33 (NeurIPS 2020). arXiv:2005.14165v4.

    GPT-3. The same design as GPT-2, grown to 175 billion parameters, and able to do new tasks from a few examples in the prompt.

  2. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., and 4 others (2020). Scaling Laws for Neural Language Models. arXiv preprint. arXiv:2001.08361v1.

    Found that a language model's loss falls smoothly and predictably as the model, its data and its compute grow.

  3. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., and 6 others (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems 33 (NeurIPS 2020). arXiv:2005.11401v4.

    Retrieval-augmented generation. The model looks up passages in a collection of documents and uses them to write its answer.

  4. Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Ré, C. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. Advances in Neural Information Processing Systems 35 (NeurIPS 2022). arXiv:2205.14135v2.

    FlashAttention. Computes exactly the same attention, much faster, by working in small tiles that fit in fast GPU memory.

  5. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., and 16 others (2022). Training Compute-Optimal Large Language Models. Advances in Neural Information Processing Systems 35 (NeurIPS 2022). arXiv:2203.15556v1.

    Chinchilla. For a fixed budget, model size and training text should grow together, so many models had been too big for their data.

  6. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., and 14 others (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35 (NeurIPS 2022). arXiv:2203.02155v1.

    InstructGPT. How a next-token predictor is tuned with human feedback to follow instructions, the recipe behind chat assistants.

  7. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., and 3 others (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Advances in Neural Information Processing Systems 35 (NeurIPS 2022). arXiv:2201.11903v6.

    Chain-of-thought prompting. Asking a model to write out its steps before the answer makes it better at problems with many steps.

  8. Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Hambro, E., and 3 others (2023). Toolformer: Language Models Can Teach Themselves to Use Tools. Advances in Neural Information Processing Systems 36 (NeurIPS 2023). arXiv:2302.04761v1.

    Tool calling. A model learns to write calls to a calculator, a search engine and other tools in its text, and to use their answers.

  9. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., and 8 others (2023). LLaMA: Open and Efficient Foundation Language Models. arXiv preprint. arXiv:2302.13971v1.

    LLaMA. Open models whose recipe many others follow, with RMSNorm before each step, SwiGLU feed-forward layers and rotary positions.

  10. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., and 62 others (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv preprint. arXiv:2307.09288v2.

    Llama 2. Its larger models use grouped-query attention, where several query heads share one key and value head.

  11. Turpin, M., Michael, J., Perez, E., and Bowman, S. R. (2023). Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting. Advances in Neural Information Processing Systems 36 (NeurIPS 2023). arXiv:2305.04388v2.

    Written reasoning can look convincing and still leave out what really drove a model's answer.

  12. Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. (2023). ReAct: Synergizing Reasoning and Acting in Language Models. ICLR 2023. arXiv:2210.03629v3.

    Agents. A model takes turns writing its reasoning and an action, reads what the action returned, and goes on.

  13. Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., and Liu, Y. (2024). RoFormer: Enhanced Transformer with Rotary Position Embedding. Neurocomputing 568, 127063. arXiv:2104.09864v5. doi:10.1016/j.neucom.2023.127063.

    Rotary position embeddings. Each position turns the queries and keys by an angle, so attention sees how far apart two tokens are. Most models today use them.

  14. DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. Nature 645, 633 to 638. arXiv:2501.12948v2.

    Reasoning models. Reinforcement learning on problems with checkable answers teaches a model to write long reasoning before it answers.

Looking inside models

  1. Jain, S. and Wallace, B. C. (2019). Attention is not Explanation. Proceedings of NAACL-HLT 2019. arXiv:1902.10186v3.

    Found that attention weights often disagree with other measures of what mattered to a prediction.

  2. Wiegreffe, S. and Pinter, Y. (2019). Attention is not not Explanation. Proceedings of EMNLP-IJCNLP 2019. arXiv:1908.04626v2.

    A reply arguing that whether attention explains depends on what you mean by explanation.

  3. Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., and 19 others (2021). A Mathematical Framework for Transformer Circuits. Transformer Circuits Thread.

    Reads a transformer as a residual stream that every layer reads from and writes to, and works out what small attention-only models compute.

  4. Geva, M., Schuster, R., Berant, J., and Levy, O. (2021). Transformer Feed-Forward Layers Are Key-Value Memories. Proceedings of EMNLP 2021. arXiv:2012.14913v2.

    Feed-forward layers act like memories. Neurons respond to patterns in the text and push up likely next tokens.

  5. Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., and 20 others (2022). In-context Learning and Induction Heads. Transformer Circuits Thread. arXiv:2209.11895v1.

    Induction heads. Attention heads that find an earlier copy of the current token and predict what came after it, a likely source of learning from the prompt.

Other explainers and tools

  1. Cho, A., Kim, G. C., Karpekov, A., Helbling, A., Wang, Z. J., Lee, S., and 2 others (2025). Transformer Explainer: Interactive Learning of Text-Generative Models. Proceedings of the AAAI Conference on Artificial Intelligence 39(28), 29625 to 29627. doi:10.1609/aaai.v39i28.35347.

    An interactive tool that runs GPT-2 in the browser and shows each step of a forward pass.