A sentence goes through a complete, tiny language model: tokens, embeddings, position encoding, one self-attention head and a prediction of the next word. It's the same machinery as a large language model, with 8 words and 4 dimensions instead of 100,000 tokens and thousands of dimensions. Each stage has the matching NumPy code, with the output it prints for the current values; open it with Show code. Edit any embedding or weight and watch the prediction change.
Every number in a language model has one of four origins. Each stage below is tagged with the same labels.
The model can't read letters. Each word is looked up in a fixed vocabulary and replaced by its number, the token id. Real tokenizers split text into pieces of words (“walk”, “ing”) so that any text fits a vocabulary of around 100,000 pieces. Here every word is one token. Click a word to add it, or type your own text; click a token to inspect it in the attention step.
Crossed-out words aren't in this model's 8-word vocabulary, so it skips them: it has no numbers for them. A real tokenizer never skips anything: it breaks an unfamiliar word into smaller pieces it does know, down to single letters or bytes if needed.
None of these words are in the vocabulary, so there's nothing for the model to work with. Add at least one of the words above.
How does a real tokenizer get its vocabulary? Byte pair encoding starts from single characters (real ones start from the 256 byte values, so nothing is ever unknown). It counts every pair of neighbours in a large body of text, merges the most frequent pair into a new token, and repeats, tens of thousands of times. Frequent words end up as one token; rare words are split into frequent pieces. The vocabulary is learned from data, but by counting, not by training.
Here the merges are learned from the four training sentences plus your text, and then used to split your text. Words this model skips in step 1 still get tokens here. This step is a side view: the model below keeps its 8 whole-word tokens.
Each token id picks one row from the embedding table: a list of numbers that stands for the word's meaning. Words used in similar ways get similar rows. These 4 columns were set by hand so you can read them; in a real model there are hundreds or thousands of columns, learned during training, and no column has a neat name. The rows for the words in the sentence are highlighted. All values are editable.
Cosine similarity compares the direction of two rows: 1 means the same direction, 0 unrelated, −1 opposite. Cat and dog score close to 1; that closeness is what lets a model use what it learned about one for the other.
Attention on its own doesn't see word order: “the cat sat” and “sat the cat” would look the same. So each position adds its own fixed pattern of sines and cosines to the embedding. The two “the”s in “the cat sat on the” start out identical and end up different.
Each word asks a question (its query), offers a label (its key) and carries content (its value), all made by multiplying X with a learned matrix. A word's query is compared with the keys of itself and every earlier word; softmax turns those scores into weights that add up to 1. A word can't look ahead: when the model predicts, the future doesn't exist yet. The three matrices are editable.
Attention weights: row = the word that is looking, column = the word it looks at. Click a row or a token above to see its calculation.
What attention gathered is projected with one more learned matrix and added to the input (the residual connection). The last position now holds the model's guess. Its row is compared with every token's embedding (the same table as step 2, reused): a high score means “the next word looks like this one”. Softmax turns the scores (logits) into probabilities. Temperature divides the logits first: below 1 makes the top choice stronger, above 1 flattens the choices.
“Add the top word” always picks the most likely word. “Sample a word” draws at random using the probabilities, which is how chat models vary their answers. Either way, the new word is added to the sentence and the whole model runs again: that's how a language model writes, one token at a time.