You can't reliably optimize a prompt you haven't measured. Prompt evaluation means testing a prompt against a representative set of inputs with defined success criteria — the prerequisite to meaningful optimization.
Building a Minimal Evaluation Set
test_cases = [
{"input": "Where's my order #4521?",
"expected_behavior": "Looks up order, gives status, no
unrelated info"},
{"input": "I want a refund for something I never ordered",
"expected_behavior": "Escalates to human, does not attempt
to process directly"},
{"input": "asdkjfh random gibberish",
"expected_behavior": "Asks for clarification, doesn't
hallucinate an answer"},
]A good test set deliberately includes edge cases and adversarial-ish inputs, not just the "happy path" — the happy path is usually what already works.
Evaluation Approaches
| Approach | How It Works | Best For |
|---|---|---|
| Rule-based checks | Code checks for required elements — valid JSON, required fields present, length limits | Structural/format correctness — fast, cheap, deterministic |
| Human review | A person reads outputs and rates quality against criteria | Nuanced quality, tone, correctness judgment |
| LLM-as-judge | A separate LLM call scores the output against defined criteria | Scaling up evaluation beyond what manual review can cover — imperfect, needs its own validation against human judgment |
See LLM Evaluation for the fuller production evaluation discipline this connects to.
Example — A Simple Rule-Based Check
def evaluate_summary(summary_text, original_text):
checks = {
"under_3_sentences": count_sentences(summary_text) <= 3,
"no_greeting_text": not contains_greeting(summary_text),
"mentions_key_entity": extract_entity(original_text) in summary_text,
}
return checksPractical Use Case
Before shipping any prompt change to production — including a prompt that "worked in a quick manual test" — running it against a maintained evaluation set catches regressions that ad hoc testing misses, especially on inputs you didn't happen to think of trying.
Common Mistakes
- Testing only a handful of "obvious" inputs, missing edge cases that real users will actually send
- Trusting LLM-as-judge scores without ever checking them against real human judgment on a sample — the judge model can have its own systematic biases
- Not re-running the evaluation set after every prompt change, treating evaluation as a one-time exercise rather than an ongoing practice
Interview Relevance
"How would you know if a new version of a prompt is actually better than the old one?" — a defined test set with consistent criteria, run against both versions, is the expected answer — not "it felt better in a few tries."
Practice Question
Design 5 test cases (including at least one edge case) for evaluating a prompt that classifies incoming emails as "urgent" or "not urgent."
