Token optimization means deliberately reducing how many tokens a feature consumes — per request and in aggregate — without meaningfully hurting output quality. It's a real, ongoing engineering practice for any LLM feature running at scale.
Prompt-Level Optimization
# Before: verbose, repetitive system prompt
"You are a very helpful and knowledgeable assistant. Please make
sure to always be polite and thorough in your responses. When
answering questions, please try your best to be accurate and
provide detailed explanations wherever possible..."
# ~50 tokens, mostly filler
# After: concise, equally effective
"You are a support assistant. Be accurate, concise, and polite."
# ~12 tokens, same practical instructionLong, flowery system prompts are a surprisingly common source of avoidable token waste — especially since the system prompt is resent on every single request.
Context/Retrieval-Level Optimization
- Retrieve fewer, better chunks instead of a large number of loosely relevant ones — see Reranking
- Summarize instead of resending long conversation history verbatim on every turn
- Trim tool/function results before adding them to context — send only the fields actually needed (see Agent Observations for the same principle in agent systems)
Output-Level Optimization
- Explicit length constraints in the prompt ("respond in 2-3 sentences")
- Structured output (JSON) instead of verbose free-form prose, when the use case allows it
- A sensible
max_tokensceiling matched to the task, not a generous default left unchanged
What NOT to Sacrifice for Token Savings
Token optimization has a real ceiling: stripping context a task genuinely needs (e.g. removing retrieved chunks that actually support the answer) trades cost savings for accuracy and hallucination risk — the wrong tradeoff for most production use cases. Optimize redundant/wasteful tokens first, not tokens that are doing real work.
Common Mistakes
- Over-trimming retrieved context to save cost, degrading answer quality and increasing hallucination risk — usually a worse tradeoff than it appears
- Optimizing prompt wording once and never revisiting it as the feature and its usage patterns evolve
- Focusing only on output-token savings while ignoring input-token growth from conversation history or retrieval, which is often the larger cost driver
Interview Relevance
"A production LLM feature's cost has grown 3x since launch with the same request volume. What would you investigate?" — a good answer checks conversation history growth, retrieval context size, and whether output lengths have drifted upward, not just "switch to a cheaper model."
Practice Question
Rewrite this system prompt to be more token-efficient while preserving its intent: "You are an extremely capable and detail-oriented assistant whose job is to help users by answering their questions as completely and thoroughly as you possibly can."
