Inner workings of a transformer network

This blog post is still incomplete. Come back later for more.

Natural language modeling

Many tasks (translation, question answering, sentiment analysis etc.) can be expressed through natural language. Once expressed that way, doing a task is equivalent to generating the output given an input and a task description. That is the next-token prediction problem11 Is a token the same as a word in a language? Did I mean next-word prediction? . Given a sequence of tokens $x_1, x_2, x_3, \cdots, x_n$ can we predict $x_{n+1}$ that comes after it (and do it repeatedly)?

What is a transformer?

A transformer network is a kind of neural network architecture that is commonly used in natural language modeling scenarios22 Language Models are Unsupervised Multitask Learners, Radford et al., see Section 2 .