Beyond the broader risks covered in Generative AI Limitations, LLMs specifically have a set of technical limitations that come directly from how they're built and trained — worth understanding at the architecture level, not just as a general disclaimer.
Knowledge Cutoff
A model only knows what was in its training data, up to the point pretraining data was collected (see Pretraining). It has no awareness of anything after that, unless the information is supplied at inference time via RAG or similar.
Bounded Context
A model can only consider a limited number of tokens at once (see Context Window) — it cannot reason over an entire codebase or a 10,000-page document set in a single request without some form of chunking or retrieval strategy.
Inconsistent Multi-Step Reasoning
Example failure pattern:
"If a train travels 60 km in 1.5 hours, then speeds up by
20% for the next 45 minutes, how far does it travel in total?"
An LLM can make arithmetic or logical slips on problems like
this, especially multi-step ones, without any visible signal
of reduced confidence — the response reads just as fluently
whether it's correct or not.This is distinct from hallucination (see LLM Hallucinations) — it's an error in the reasoning process itself, not a fabricated fact.
Non-Determinism
At non-zero temperature, the same prompt can produce different outputs on different calls — complicating reproducible testing and requiring evaluation strategies that account for output variability, not just single-example spot-checks.
No Built-In Fact-Checking
An LLM has no internal mechanism to verify its own output against ground truth — it generates the statistically most plausible continuation, which is usually correct for well-represented facts, but has no guaranteed correctness check.
Practical Use Case: Designing Around These Limits
| Limitation | Common Mitigation |
|---|---|
| Knowledge cutoff | RAG, or web/tool access for current information |
| Bounded context | Chunking, retrieval, summarization pipelines |
| Reasoning errors | Structured prompting, verification steps, human review for high-stakes outputs |
| Non-determinism | Lower temperature for consistency-sensitive tasks, evaluation across multiple runs |
Common Mistakes
- Trusting an LLM's arithmetic or multi-step logical output for high-stakes decisions without verification
- Assuming a bigger/newer model has "solved" these limitations entirely rather than reduced their frequency — they remain architecturally inherent to how these models work
Interview Relevance
"Name three technical limitations of LLMs and how you'd design around each" — a strong answer picks specific, distinct limitations (not just "hallucination" repeated three ways) with a concrete mitigation for each.
Practice Question
You're building a system that uses an LLM to calculate totals from a table of numbers. What limitation from this page is most relevant, and how would you mitigate it?
