max_tokens (or an equivalently-named parameter) caps how many tokens the model is allowed to generate in a single response — a deliberate limit you set, distinct from the model's overall context window.
Basic Usage
# Conceptual — parameter name varies by provider
response = llm_client.generate(
prompt="Summarize this article in a few sentences.",
max_tokens=150 # caps the response at ~150 tokens
)What Happens When the Cap Is Hit
If the model hasn't naturally finished its response by the time it reaches max_tokens, generation stops mid-response — output can be cut off mid-sentence or mid-thought. Most APIs return a "finish reason" field indicating whether generation stopped naturally or was cut off by the token limit — always check this field, don't assume a returned response is complete just because it was returned successfully.
if response.finish_reason == "length":
# response was truncated — handle accordingly:
# retry with a higher limit, or treat as incomplete
...Setting a Sensible Value
| Task | Guidance |
|---|---|
| Short, bounded answers (classification labels, short extraction) | Set a tight cap — no reason to allow more |
| Open-ended generation (articles, detailed explanations) | Estimate a reasonable upper bound based on expected output length, with some headroom |
| Structured output (JSON with known fields) | Estimate based on the schema's expected size — too tight a cap can truncate valid JSON mid-structure, breaking parsing |
Why This Parameter Matters Beyond Just "Not Running Out of Room"
It's also a direct cost and latency control (see Output Tokens) — setting it deliberately, rather than leaving a generous default unchanged, is a real, low-effort optimization for any production feature.
Common Mistakes
- Not checking the finish reason, silently shipping truncated responses to users or downstream systems
- Setting the cap too low for structured output tasks, causing valid JSON to be cut off mid-object and fail to parse
- Leaving a very generous default cap unchanged across every feature regardless of actual expected output length
Interview Relevance
"A JSON response from an LLM API sometimes fails to parse. What would you check?" — whether max_tokens is large enough to accommodate the full expected structure, and whether the response was truncated (finish_reason), is a strong first diagnostic step.
Practice Question
A structured-output request returns malformed JSON about 5% of the time. Propose a specific check and fix related to max_tokens.
