At every step of the loop, the LLM has to decide: call a tool (and which one, with what arguments), ask a clarifying question, or produce a final answer. This decision is driven entirely by what's in its current context — there's no separate "reasoning module."
What Actually Drives the Decision
Decision = f(system instructions, goal, available actions, history so far)The LLM isn't running a separate planning algorithm — it's generating the same way it always does (next-token prediction, see How Generative AI Works), just with a prompt specifically structured to produce a structured decision instead of free-form prose. Framework instructions (e.g. "think about what information you still need, then choose ONE tool") shape this via prompting, not a fundamentally different mechanism.
A Structured Decision Format
{
"reasoning": "The user asked for order status, but I don't have
the order ID yet. I need to ask for it before I
can look anything up.",
"action": "ask_clarifying_question",
"content": "Could you share your order number so I can look that up?"
}Structuring the decision output (see Structured Output) makes it far easier for the orchestration layer to parse and act on reliably, compared to trying to extract intent from free-form text.
Handling Uncertainty
A well-designed agent should be able to choose "ask for clarification" or "escalate" as valid actions when the goal is ambiguous or the agent lacks confidence — rather than being forced to guess. Encouraging the model (via system instructions) to prefer clarification over guessing on ambiguous requests measurably reduces incorrect actions in practice.
Practical Use Case
A well-designed decision step is what separates "the agent tried something reasonable and it happened to be wrong" from "the agent had no way to express uncertainty and guessed" — the latter is a design gap, not an unavoidable model limitation.
Common Mistakes
- Not giving the model a way to express "I'm not sure" or "I need more information" as a first-class action, forcing it into a guess
- Free-form, unstructured decision output that's fragile to parse reliably in code — small wording variations can break simple string-matching logic
- Assuming decision quality is purely a function of model capability, ignoring how much prompt/context design affects it in practice
Interview Relevance
"How does an agent 'decide' what to do next — is there separate reasoning logic?" — the accurate answer: no separate reasoning engine; it's the same LLM generation process, shaped by prompt structure and available actions.
Practice Question
Design a structured decision format (like the JSON example above) for an agent that can search, calculate, or answer directly.
