The context window is the maximum number of tokens a model can consider at once — input and output combined, for most API-based models. Once you exceed it, older content has to be dropped or the request fails outright.
What Counts Toward the Context Window
Context window budget includes:
- System prompt
- Conversation history (all previous turns, if resent)
- The current user message
- Any retrieved context (e.g. RAG chunks)
- The model's own output tokens
Total must fit within the model's context window limit.See Context Window vs Token Limit for how this differs slightly from a simple "output limit."
Why Conversations "Forget" Earlier Messages
Chat applications typically resend the recent conversation history with every new request (the model itself has no persistent memory between API calls — see How LLMs Work). Once a long conversation's total token count approaches the context window limit, older messages usually get truncated or summarized to make room — which is why very long conversations can appear to "forget" earlier details.
Practical Example
# Simplified illustration of a context budget check
system_prompt_tokens = 150
conversation_history_tokens = 3200
new_user_message_tokens = 40
context_window_limit = 8000
available_for_response = context_window_limit - (
system_prompt_tokens + conversation_history_tokens + new_user_message_tokens
)
# available_for_response = 4610 tokens left for the model's outputLarger Context Windows — Real Tradeoffs, Not Just a Bigger Number
Larger context windows let you include more documents, longer conversation history, or more few-shot examples — but processing more input tokens increases cost and can increase prefill latency (see LLM Inference). A bigger context window is also not a free substitute for good retrieval — see RAG vs Long Context for that tradeoff specifically.
Common Mistakes
- Assuming a large context window means the model uses all of that context equally well — in practice, models can pay less attention to information buried in the middle of a very long context ("lost in the middle" effect), so relevance-based retrieval often still outperforms just stuffing everything in
- Not accounting for the model's own output tokens when budgeting context — a request that maxes out input tokens can leave too little room for a complete response
Interview Relevance
"A user's long conversation with our chatbot suddenly starts giving worse answers. Why?" — a strong answer investigates whether the conversation has exceeded (or is nearing) the context window, causing truncation of earlier relevant context.
Practice Question
A model has an 8,000-token context window. Your system prompt is 500 tokens, and you want to leave room for a 1,500-token response. How many tokens of conversation history can you include?
