Input tokens are everything sent to the model in a request — system prompt, conversation history, retrieved context, and the current message. They're usually billed at a lower per-token rate than output tokens, but they can dominate cost in context-heavy applications like RAG.
What Counts as Input
Input tokens = system_prompt
+ conversation_history (if resent each turn)
+ retrieved context (e.g. RAG chunks)
+ the current user message
+ tool/function definitions (if using tool calling)Every one of these competes for the same context window budget — see Context Window.
Why Input Tokens Can Dominate Cost in RAG
A typical RAG request:
system prompt: 150 tokens
retrieved chunks: 2,500 tokens ← often the largest share
conversation history: 400 tokens
user question: 30 tokens
────────────────────────────────
total input: 3,080 tokens
model's response: 200 tokens (output)Here, input tokens (3,080) vastly outweigh output tokens (200) — a common pattern for RAG and other context-heavy applications, which is why input-token efficiency (retrieving only what's actually needed, not over-fetching) meaningfully affects cost at scale.
Practical Optimization Angle
- Retrieve fewer, more relevant chunks instead of many loosely relevant ones (see Reranking)
- Summarize long conversation history instead of resending it verbatim on every turn
- Avoid re-sending large static content (like full documentation) on every request if it can be cached or referenced more efficiently — see LLM Caching
Common Mistakes
- Optimizing only for output length while ignoring input token growth — in RAG and long-conversation applications, input often costs more overall
- Resending entire conversation history verbatim indefinitely as a chat grows longer, instead of trimming or summarizing older turns
Interview Relevance
"In a RAG application, are input or output tokens usually the bigger cost driver?" — input tokens, typically, due to retrieved context — a good practical-cost-awareness question.
Practice Question
A chatbot resends the full 20-turn conversation history on every new message. Propose one concrete change to reduce input token usage without losing important context.
