Evaluation

Agents are confidently wrong often enough that you cannot trust their output unverified, and every time you change a prompt, a tool, or a model you need to know whether the change helped or hurt. Eyeballing a couple of runs answers neither question — it hides regressions, because a tweak that fixes the case in front of you can quietly break five you did not look at. Evaluation is the pattern for checking agent output systematically rather than by impression.

The forms, cheapest first

• Deterministic checks. Does the code compile, do the tests pass, does the output match a known answer or a required schema? Where one applies, this is the best kind of evaluation — fast, repeatable, and free of opinions. Reach for it before anything fancier.
• A judge agent (LLM-as-judge). A separate agent scores the output against a rubric. Useful where there is no single right answer — for example, “is this summary faithful to the source?” — which a deterministic check cannot settle.
• Adversarial verification. Spawn an independent agent (or several) whose job is to refute the result, and keep the result only if it survives. This is the evaluator shape from multi-agent orchestration , turned skeptical on purpose.
• Human review. Still the ground truth for high-stakes work. The cheaper forms above are how you make human time count — they triage, so a person looks at the few outputs that need a person.

Tying them together is an evaluation set: a fixed collection of representative cases you run every change against. Without it, “better or worse” is a feeling; with it, it is a number you can compare.

Doing it well

1. Prefer a deterministic check over a judge whenever one exists. Tests do not have moods, do not drift, and do not need a second opinion.
2. Treat a judge as the agent it is. It can be wrong too, so give it a clear rubric, and for decisions that matter use more than one independent judge and take the majority — and do not let a model grade its own work uncritically, since it tends to like what it wrote.
3. Keep adding cases. Every real failure you find becomes a new case in the set, so the same bug cannot return unnoticed.

What evaluation does not tell you

Passing your checks does not mean the output is right — it means it did not fail the checks you thought to write. Evaluation can show you that something is wrong; it cannot prove something is correct. So an agent result that clears its checks is a stronger draft, not a guarantee, and the set is only ever as good as the cases in it.

Where to go next

• The skeptical-reviewer shape in context: Multi-agent orchestration .
• Where output to evaluate comes from: Tool use and Retrieval (RAG) .