You can build an impressive AI demo in an afternoon. Turning it into something you trust in production is where most teams stall, and the reason is almost always the same: they cannot tell, reliably, whether a change made the system better or worse. Evaluation - measuring the quality of model outputs - is the hardest unsolved problem in shipping AI, because unlike traditional software, there is often no single correct answer to check against.
Why It Is Hard
A function that adds two numbers has one right answer, and a test asserts it. A prompt that "summarizes a support ticket helpfully" has thousands of acceptable answers and no exact string to compare. Quality is fuzzy, subjective, and context-dependent. And LLMs are non-deterministic - the same input can produce different outputs - so you cannot even rely on a fixed result. This is why "it looked good when I tried it" is the trap that sinks AI projects.
Start With an Eval Set, Not a Metric
Before choosing how to measure, assemble what to measure against: a curated set of representative inputs, ideally including the hard cases and past failures. This eval set is the single highest-leverage asset in an AI project. Every prompt tweak, model swap, or retrieval change is run against it so you can compare versions on the same ground instead of arguing from anecdotes. Twenty carefully chosen examples beat two thousand random ones.
The Ladder of Evaluation Methods
1. Deterministic checks. Whenever the task has any objective component, assert it in code - cheap, fast, and reliable. Is the output valid JSON? Does it match the required schema? Does the extracted date parse? Does the answer contain the one fact it must? Automate everything that can be automated before reaching for anything fancier.
def check(output):
assert is_valid_json(output)
data = json.loads(output)
assert "refund_amount" in data
assert isinstance(data["refund_amount"], (int, float))2. Reference-based metrics. When you do have a gold answer, you can score similarity - exact match for classification, or embedding similarity for open-ended text (does the answer mean the same thing as the reference, even if worded differently). Useful, but limited: many good answers legitimately differ from your one reference.
3. LLM-as-judge. For genuinely subjective quality, use a strong model to grade outputs against a rubric you write. "Rate this summary 1-5 on faithfulness to the source, and explain." This scales human-like judgment cheaply and correlates surprisingly well with human ratings - when the rubric is specific.
Judge prompt:
"You are grading a support reply. Score 1-5 on each:
- Accuracy: does it correctly address the ticket?
- Tone: is it professional and empathetic?
- Completeness: does it resolve the issue or escalate correctly?
Return JSON with a score and one-line reason per category."4. Human review. Still the gold standard for the cases that matter most and for calibrating your automated judges. You cannot review everything, so sample - especially the disagreements where your judge is unsure.
The Traps in LLM-as-Judge
It is powerful but biased in known ways. Judges favor longer answers, favor the first option when comparing two, and can favor outputs from the same model family. Mitigate by writing tight rubrics, randomizing option order, using pairwise comparison ("is A or B better?") rather than absolute scores when you can, and periodically checking the judge against human labels. A judge you never audit is a metric you should not trust.
Evaluate the Pieces, Not Just the Whole
In a RAG system, a wrong final answer could come from bad retrieval or bad generation. Measure each: retrieval quality (did the right chunks come back?) separately from answer quality (given good chunks, was the answer faithful?). Component-level evals tell you where to fix; end-to-end evals tell you whether the fix helped.
Close the Loop With Production
Your eval set will never cover everything real users do. Log real inputs and outputs, capture user signals (thumbs up/down, edits, escalations, follow-up questions that indicate the first answer failed), and feed the failures back into the eval set. The system that improves is the one where production failures become tomorrow's test cases.
The Bottom Line
Treat evaluation as the foundation, not an afterthought. Build the eval set first, automate every objective check, use LLM-as-judge for the subjective majority with its biases in mind, keep humans in the loop for calibration, and turn production failures into new tests. Teams that measure rigorously ship AI that works; teams that go on vibes ship demos that quietly break.