Tokens and Tokenization (LLMs)

Building on Tokenization, this note covers what tokens mean specifically in the context of using and reasoning about LLMs — where token counts directly affect cost, speed, and how much text a model can actually process at once.

Tokens ≠ Words ≠ Characters

A common rough rule of thumb for English text: roughly 1.3 tokens per word on average — though this varies significantly by language, domain (code, technical jargon), and the specific tokenizer used.

Why Token Counts Matter Practically

• Cost: most commercial LLM APIs charge per token, for both input and generated output.
• Context window budget: every model has a maximum number of tokens it can process at once (covered fully in Context Window) — token count determines whether a given piece of text fits.
• Latency: generation happens one token at a time (autoregressively), so more output tokens directly means more generation steps and more time.

Code — Counting Tokens Before Sending a Request

import tiktoken

encoding = tiktoken.encoding_for_model("gpt-4")
text = "The quick brown fox jumps over the lazy dog."
tokens = encoding.encode(text)
print(len(tokens))        # count of tokens, not words or characters
print(tokens[:5])          # the actual token IDs
print(encoding.decode(tokens[:5]))   # decode back to see what those specific tokens represent

Common Mistakes

• Estimating cost or context usage by word count or character count instead of actual token count — the two can diverge significantly, especially for non-English text, code, or unusual vocabulary.
• Assuming every model uses the same tokenizer — different model families (and even different versions of the same family) frequently use different tokenizers, so a piece of text can tokenize into a meaningfully different number of tokens depending on which model you're targeting.

Interview Relevance

Q: "Why might the same English sentence use noticeably more tokens when translated into another language, even with the same word count?" Subword tokenizers (BPE) are typically trained predominantly on English-heavy corpora, so common English words often map to a single token, while words in less-represented languages more frequently get split into multiple smaller sub-word pieces, since the tokenizer's vocabulary contains fewer whole-word entries for that language.

Practice Question

A piece of Python code contains many multi-character variable names and symbols. Would you expect this to tokenize more or less efficiently (fewer tokens per character) than typical English prose, and why?

Related DL Notes

• Tokenization
• Context Window
• Vocabulary