Attention is a mechanism that lets a model decide, for each token, how much weight to give every other token when building that token's representation — instead of treating all context equally, or only looking at nearby words.
The Intuition, With a Concrete Example
Sentence: "The animal didn't cross the street because it was too tired."
When processing the word "it", attention computes a relevance
score against every other word in the sentence:
"animal" → high relevance (likely what "it" refers to)
"street" → low relevance
"cross" → low relevance
"tired" → moderate relevance (relates to the reason)
These scores become weights — "it"'s final representation
becomes a weighted blend, drawing heavily from "animal".This is what lets a model correctly resolve pronouns, long-range dependencies, and context that a fixed-window approach (only looking a few words back) would miss entirely.
Why "Attention" Is the Right Word for This
Just like a person reading a long sentence pays more mental "attention" to the words that matter most for understanding a given word's meaning, and less to filler words — the mechanism computes exactly this kind of weighting, learned automatically from training data rather than hand-programmed.
Attention Is Not Unique to Transformers — But They're Built Around It
Attention mechanisms existed in earlier sequence models (as an add-on to RNN-based architectures) before transformers. The transformer's key innovation was making attention the primary mechanism — specifically self-attention — rather than a secondary addition, and doing so in a way that's highly parallelizable.
Practical Use Case
Understanding attention conceptually explains real model behavior — why LLMs are generally good at resolving pronouns and references across long passages, and why irrelevant filler text in a prompt can still subtly compete for the model's "attention" and occasionally dilute focus on what actually matters, a practical reason to keep prompts and RAG context focused rather than padded.
Common Mistakes
- Assuming attention only looks at nearby words, like older fixed-window approaches — it can directly relate any two tokens in a sequence, regardless of distance
- Treating "attention" as a vague marketing term rather than a specific, learnable weighting mechanism with a precise mathematical definition (see Query, Key, Value)
Interview Relevance
Q: "Explain attention in one or two sentences, without the math." — the pronoun-resolution example above is a strong, intuitive way to answer this without needing formulas.
Practice Question
For the sentence "The trophy didn't fit in the suitcase because it was too small," which word should "it" attend to most strongly, and how does that differ from the earlier trophy/suitcase example in this note?
