Evaluating Agent Behavior
Build graders that measure agent output before you ship — rubrics, golden sets, and the code-vs-model tradeoff.
Why vibe-checking fails
“I read some outputs and they looked good” does not survive contact with a model update, a prompt tweak, or a traffic spike. Without a grader you cannot tell an improvement from a regression, and every change becomes a gamble.
A usable evaluation has three parts: cases, a grader, and a baseline score.
Start with 20 real cases, not 200 synthetic ones
The most common mistake is generating a large synthetic dataset before having any real usage. Synthetic cases encode what you assumed users would do, which is usually wrong.
Collect from reality instead:
- Real failed interactions you remember
- The awkward inputs users actually typed
- Edge cases that broke something once
- Cases where “correct” is genuinely arguable
Twenty real cases beat two hundred imagined ones. Expand later, once real traffic tells you what else matters.
Choose the cheapest grader that can judge
Match the grader to what is being measured — do not reach for a model when code will do:
| What you are checking | Grader | Why |
|---|---|---|
| Exact format / valid JSON | Code | Deterministic, free, instant |
| Contains required citations | Code (regex) | No judgment needed |
| Numeric answer within tolerance | Code | Cheaper and stricter than an LLM |
| Tone, clarity, completeness | Model (rubric) | Requires judgment |
| Whether the plan is sound | Model + human spot-check | Too consequential to automate fully |
A code grader that catches 80% of failures today is worth more than a model grader you are still tuning next month.
Writing a rubric that agrees with itself
Model graders drift. Constrain them:
- Score each dimension separately (correctness, completeness, tone) rather than one overall score
- Use 1–5 with every level defined in words, not just endpoints
- Require the grader to quote the span that justified the score
- Grade the same case 3 times; if scores swing more than 1 point, your rubric is ambiguous
That last check is the one teams skip, and the reason their eval results do not replicate.
Reading the numbers
Track these across versions:
- Pass rate on the golden set — the headline number
- Regression list — cases that used to pass and now fail (most actionable artifact)
- Grader disagreement rate — spikes here mean the rubric broke, not the agent
When pass rate drops, look at the regression list before changing anything. Two or three specific failures usually explain the entire delta.
Practical caveat
An eval measures your golden set, not your product. Revisit the set whenever user behavior changes — a stale suite confidently green-lights regressions it no longer covers.