Attention Lab
The mechanism behind every modern language model, running live: a small character-level GPT I trained from scratch on Shakespeare, exported with its attention weights as a first-class output. Type anything and watch each head choose which earlier characters to read before predicting the next one. No server, no API: ONNX and WebAssembly, like the rest of the ML Lab.
85 / 128 characters (the context window). It only knows the 65 characters that occur in Shakespeare, so digits mostly bounce off.
Loading the model (about 2.6 MB, runs entirely in your browser)...
What comes after ...
The model
619,776 parameters: 3 layers, 4 heads, 128-dim, character-level. Trained from scratch by me on tiny shakespeare (1.1 mb, character-level) for 8,000 steps (val loss 1.5047), then exported to ONNX with the attention tensor as an output. Training code in sketch-lab. For scale: GPT-2 small is 200x bigger, and it is tiny by today's standards. The mechanism you are watching is the same one.
What it writes when left alone
That good tears and honour, they say you they say To her passe your tent of the father daughter-babe My brother partise their poison accustion Descrate not whereof what we would have I did smiled, With that your sweet and of his daughter queen? For they have shall say you, throw to dead: Tell her by
Layer 2, head 4: who is the last character reading?
Coral shading = how much attention this head pays from the focused character to each earlier one (shaded relative to its strongest read). Click any character to move the focus; causal masking means a character can only read what came before it.
Waiting for text.
Every head at once
Each square is one head's full attention pattern: row = reading character, column = character being read, brighter = more attention (rows shaded relative to their strongest read). The coral line is the focused character's row. Click a head to drive the panel above.
What this model actually grew (measured, not hoped for)
- Previous-token heads: L3·H1 and L3·H3 put roughly half their attention on the immediately preceding character (0.48 and 0.49 mean weight), the bright diagonal stripe in the grid. It is the bigram statistic made flesh; almost every trained transformer grows one.
- A sentence anchor: L2·H4 largely ignores nearby text and parks on the capital letter that opened the current sentence, with a side glance at the period before it. Look for its lonely vertical bars.
- Word-start readers: several heads (L2·H1, L3·H4) read the space just before the word being written. A character-level model has to discover where words begin; nobody told it.
- The missing head: larger transformers grow induction heads, which find the previous occurrence of the current pattern and read what came after it last time; that is how they copy a name they just saw. I measured all 12 heads here and none does it (max induction score 0.02), which is why the Copy test preset ends with the model guessing my instead of completing mine. In-context copying is an emergent capability, and at 620k parameters it has not emerged yet. That absence is the most instructive thing on this page.