Contextual Embeddings

Word2Vec and GloVe both share one fundamental limitation: every word gets exactly one fixed embedding vector, regardless of the sentence it appears in. Contextual embeddings fix this — the same word gets a different vector depending on its surrounding context.

The Problem, Concretely — "Bank"

Word2Vec and GloVe assign "bank" the exact same vector in both sentences — the embedding has no way to distinguish these two very different meanings, since it's computed once, independent of any surrounding sentence. This is called a static embedding.

How Contextual Embeddings Solve This

Instead of a fixed lookup table, a contextual embedding model (like BERT, covered in the next note) processes an entire sentence through a deep network — typically Transformer layers using self-attention — and each word's final representation is computed fresh, incorporating information from every other word in that specific sentence. This is exactly what self-attention (see Self-Attention) naturally provides: "bank"'s final representation in "I deposited money at the bank" will have attended strongly to "deposited" and "money," while "bank" in "we sat on the river bank" will have attended strongly to "river" — producing genuinely different output vectors for the same input word.

Static vs Contextual, Side by Side

Code — Seeing the Difference Directly

from transformers import AutoTokenizer, AutoModel
import torch

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModel.from_pretrained("bert-base-uncased")

def get_bank_embedding(sentence):
    inputs = tokenizer(sentence, return_tensors="pt")
    outputs = model(**inputs)
    tokens = tokenizer.convert_ids_to_tokens(inputs["input_ids"][0])
    bank_index = tokens.index("bank")
    return outputs.last_hidden_state[0, bank_index]

vec1 = get_bank_embedding("I deposited money at the bank")
vec2 = get_bank_embedding("we sat on the river bank")

similarity = torch.cosine_similarity(vec1.unsqueeze(0), vec2.unsqueeze(0))
print(similarity)   # noticeably less than 1.0 -- the two "bank" vectors are genuinely different

Common Mistakes

• Using a static embedding model (Word2Vec/GloVe) for a task where word sense disambiguation genuinely matters — sentiment analysis, question answering, and many other modern NLP tasks benefit substantially from contextual embeddings for exactly this reason.
• Assuming contextual embeddings are strictly "an upgrade" with no downsides — they're considerably more computationally expensive to obtain (a full model forward pass, versus a static lookup), which matters for latency-sensitive applications.

Interview Relevance

Q: "Why can't Word2Vec distinguish between different meanings of the same word, and how do models like BERT solve this?" Word2Vec assigns exactly one fixed vector per word, computed once from the training corpus, with no mechanism to adjust based on a specific sentence's context. BERT (and similar models) instead compute each word's representation fresh, for every sentence, using self-attention to incorporate information from every other word in that specific sentence — so the same word can end up with meaningfully different vectors depending on which meaning is actually being used.

Practice Question

Would you expect contextual embeddings for the word "bat" in "the bat flew out of the cave" and "he swung the bat" to be similar or different? Why?

Related DL Notes

• Self-Attention
• BERT
• Word2Vec