A token is the basic unit an LLM actually reads and writes — usually a word, part of a word, or punctuation mark. Models don't process raw text or whole words; everything is broken into tokens first.
A Concrete Example
Text: "Tokenization isn't always intuitive."
Tokens: ["Token", "ization", " isn", "'t", " always",
" intuitive", "."]
Count: 7 tokens for 5 words — because uncommon words get
split into smaller pieces ("Token" + "ization"), while
common short words often stay whole.Roughly, 1 token ≈ 4 characters or ~0.75 words in English — but this is a rough average, not a rule; it varies significantly by language and text type. See Token Count for how to measure this precisely.
Why Tokens (Not Words or Characters)?
| Unit | Problem If Used Directly |
|---|---|
| Whole words | Vocabulary would need to cover every possible word in every language — millions of entries, and it still couldn't handle typos, made-up words, or rare technical terms |
| Individual characters | Sequences become extremely long, making the model slower and less effective at capturing meaning across long spans of text |
| Subword tokens (the actual approach) | A manageable vocabulary (tens of thousands of entries) that can represent any text, including unfamiliar words, by combining smaller known pieces |
Why This Matters Practically
Tokens are the unit that determines: how much text fits in a model's context window, how much an API call costs (see Token Cost), and how long generation takes (more output tokens = more sequential generation steps, see LLM Inference). Every practical constraint you'll hit building with LLMs traces back to tokens, not words or characters.
Common Mistakes
- Estimating cost or context usage in words instead of tokens — the two numbers can differ substantially, especially with code, non-English text, or unusual formatting
- Assuming all languages tokenize with similar efficiency — many tokenizers are trained predominantly on English text and represent other languages using more tokens per word, an important cost/context factor
Interview Relevance
Q: "Why don't LLMs just process whole words?" — the expected answer covers vocabulary size limits and the need to handle unfamiliar/rare words gracefully, which subword tokenization solves.
Practice Question
Without a tokenizer tool, estimate roughly how many tokens this sentence contains: "The quick brown fox jumps over the lazy dog." Then explain what makes your estimate approximate rather than exact.
