Output tokens are what the model generates — typically billed at a higher per-token rate than input, and the direct driver of generation latency, since each output token is produced sequentially (see LLM Inference).
Why Output Tokens Cost More Per Token
Generating each output token requires a full forward pass through the model at that step; processing input tokens (during the "prefill" phase) can be parallelized far more efficiently. This computational difference is typically reflected in provider pricing — output tokens usually cost noticeably more per token than input tokens.
Output Length Directly Drives Latency
Rough mental model (illustrative, not a precise formula):
time_to_first_token ≈ mostly driven by input processing (prefill)
total_generation_time ≈ time_to_first_token + (output_tokens × per-token decode time)
A request asking for a 1,000-token response will take
meaningfully longer than one asking for a 50-token response,
even from an identical prompt.Practical Techniques to Manage Output Tokens
| Technique | Effect |
|---|---|
| Explicit length instructions ("answer in 2 sentences") | Reduces unnecessarily long responses — effectiveness varies by model and should be tested, not assumed |
max_tokens cap | Hard ceiling on cost/latency for that request — see Max Output Tokens |
| Structured output (JSON) | Naturally bounds verbosity compared to free-form prose — see Structured Output |
| Streaming | Doesn't reduce total generation time, but improves perceived latency — see Streaming |
Practical Use Case
A summarization feature that consistently generates 800-word summaries when 150 words would suffice is both slower and more expensive than necessary — tightening the prompt's length instruction (and testing that it's actually followed) is often the single highest-leverage cost optimization in a text-generation feature.
Common Mistakes
- Not measuring actual output token usage in production — assuming a feature is cheap without checking real generation lengths across real user requests
- Relying purely on prompt instructions to control length for cost-sensitive features, with no hard
max_tokenssafety net
Interview Relevance
"Why would you cap max_tokens even if cost isn't a concern?" — latency: a runaway long generation degrades user experience regardless of budget, which is a distinct reason from pure cost control.
Practice Question
A code-explanation feature sometimes generates extremely long responses for simple questions. Propose two changes — one prompt-level, one API-parameter-level — to address this.
