Before any attention computation happens, every token is converted into an embedding — a dense vector of numbers that represents its meaning. This is the entry point of the transformer pipeline, and the concept reappears (differently applied) in semantic embeddings used for search and RAG.
From Token ID to Embedding
Token: "cat"
↓ tokenization
Token ID: 5234
↓ embedding lookup (a learned table, one row per vocabulary entry)
Embedding vector: [0.12, -0.45, 0.88, ..., 0.03] (hundreds to
thousands of numbers)This embedding table is learned during training — the model discovers vector representations where tokens with related meanings end up positioned closer together in this high-dimensional space, purely as a side effect of optimizing next-token prediction.
Why Not Just Use the Token ID Directly?
A token ID (like 5234) is an arbitrary index — it carries no information about meaning, and "5234" being numerically close to "5235" says nothing about whether those two tokens are related in meaning. Embeddings solve this: the vector encodes semantic relationships, learned from data, in a form the network's mathematical operations (like attention's dot products) can actually make use of.
This Is Distinct From (But Related To) Retrieval Embeddings
The embeddings inside a transformer are an internal architectural component, updated during training and specific to that model's vocabulary. The embeddings used for semantic search / RAG are typically produced by a (possibly different) trained embedding model, applied to full sentences or documents, and stored externally for retrieval. Same underlying idea — dense vectors capturing meaning — different scope and purpose.
Practical Use Case
Understanding that transformer input starts as embeddings clarifies why LLMs are fundamentally working with continuous numeric representations, not text — every step after this initial embedding lookup (attention, feed-forward layers) operates purely on vectors of numbers.
Common Mistakes
- Confusing a transformer's internal token embeddings with the separate embedding models used for RAG/semantic search — related concept, different systems, often different models entirely
- Assuming embedding dimensions are arbitrary — the dimensionality is a real architectural choice affecting both capability and compute cost
Interview Relevance
"What's the first thing that happens to a token before it reaches the transformer's attention layers?" — the token ID is looked up in an embedding table to produce a dense vector representation; this is a common warm-up architecture question.
Practice Question
Explain why two tokens with similar meanings ("happy" and "joyful") would be expected to have embedding vectors that are close together in vector space, based on how embeddings are learned.
