Multi-head attention runs several self-attention computations in parallel — each "head" can learn to focus on a different kind of relationship between tokens — and combines their results into one richer representation.
Why One "Head" of Attention Isn't Enough
A single attention computation produces one specific weighting of relevance. But language has multiple simultaneous kinds of relationships worth capturing — grammatical structure (subject/verb), coreference (pronoun/referent), semantic similarity, and more. Multiple heads let the model learn to specialize — one head might end up focusing on syntactic relationships, another on longer-range thematic connections, without being explicitly told to.
Illustrative Example
Sentence: "The developer who wrote the buggy function fixed it quickly."
Head 1 might learn to focus on: "it" → "function" (coreference)
Head 2 might learn to focus on: "fixed" → "developer" (who did the action)
Head 3 might learn to focus on: "quickly" → "fixed" (how the action happened)
Each head computes its own attention weights independently;
their outputs are combined afterward.Note: real heads don't have human-readable labels like this — what each head ends up specializing in emerges from training and isn't assigned by a developer. This example is illustrative of the kind of specialization that can emerge, not a literal description of any specific model's heads.
The Mechanical Process
1. Split the token representations into H separate "heads"
(each head works with a smaller slice of the full representation)
2. Run self-attention independently within each head
3. Concatenate all heads' outputs back together
4. Apply a final linear transformation to combine themThis runs in parallel (not sequentially), so multi-head attention doesn't multiply computation time by the number of heads the way running H separate full-size attention passes would.
Practical Use Case
The number of attention heads is one of the architectural choices that varies across model sizes — larger models typically use more heads (and larger overall representations), part of what's captured in a model's parameter count (see LLM Parameters).
Common Mistakes
- Assuming each attention head has a fixed, known, human-interpretable job — in practice, what heads learn emerges from training and isn't perfectly clean or predictable
- Assuming more heads is unconditionally better — it's one architectural dimension among several, tuned alongside model depth and width during design
Interview Relevance
"Why use multiple attention heads instead of one larger attention computation?" — the expected answer: multiple heads let the model capture different types of relationships between tokens in parallel, rather than being limited to a single weighting scheme.
Practice Question
Explain in your own words why running attention heads in parallel is more efficient than running them one after another.
