Self-attention is attention applied within a single sequence — every token attends to every other token in the same input, including itself. This is the specific mechanism transformers are built around.
"Self" — What It's Contrasted With
In the original 2017 transformer (designed for translation), the decoder also attended to a separate sequence — the encoder's output (see Encoder-Decoder). Self-attention specifically means a sequence attending to itself — the mechanism that lets a sentence "understand itself" internally, which is the primary mechanism in modern decoder-only LLMs.
Worked Example — Building One Token's Representation
Sequence: "The cat sat on the mat"
To build a richer representation of "sat", self-attention:
1. Compares "sat" against every token in the sequence
(including "sat" itself)
2. Computes a relevance score for each comparison
3. Uses those scores as weights to blend information from
all tokens into a new representation of "sat"
Result: "sat"'s new representation is informed by "cat" (who
sat), "mat" (where), and its own original meaning — richer
than looking at "sat" in isolation.Every token in the sequence goes through this same process simultaneously — not one at a time, which is what makes self-attention parallelizable across an entire sequence.
Causal (Masked) Self-Attention — Specific to Generation
When generating text, a model must not "peek" at future tokens it hasn't generated yet. Causal self-attention enforces this: each token can only attend to itself and earlier tokens, never later ones. This masking is exactly what makes autoregressive, left-to-right text generation possible — see Decoder-Only Transformers.
Practical Use Case
Self-attention is why LLMs handle long-range dependencies well — a pronoun on line 40 of a document can still directly attend to its referent on line 2, something older architectures struggled to do reliably at that distance.
Common Mistakes
- Confusing self-attention (within one sequence) with cross-attention (between two different sequences, e.g. encoder-to-decoder) — they're related but distinct mechanisms
- Forgetting that generation-time self-attention is causal/masked — a common point of confusion when first learning the difference between training-time and generation-time behavior
Interview Relevance
Q: "What does 'causal' mean in causal self-attention, and why does it matter for generation?" — the masking mechanism that prevents a token from attending to future tokens is the expected answer, tied directly to why autoregressive generation works at all.
Practice Question
Explain why, during generation, the 5th token being generated can attend to tokens 1-4 but not to a hypothetical 6th token that hasn't been generated yet.
