A Free Guide to AI Agent Evaluation

Friday, 5:47 PM. A developer named James presses deploy and goes home. He has spent three weeks building a support agent that reads a customer message, checks their order, and decides: approve the refund, or pass it to a human. He wrote 340 tests. Every one passed.

Monday, 8:58 AM. Forty-three tickets.

I have written a free short book about AI agent evaluation, and James is the one who lives through it. You can read the whole thing or download the PDF here, no email required.

Why 340 passing tests told him nothing

When James wrote those tests, he also wrote the customer messages. Clear English, one request per message, every edge case he could think of. His test suite was a list of problems he had already thought of.

Real customers write the way they text a friend. Short messages, missing words, two questions at once, sometimes in another language, sometimes at midnight when they are already annoyed.

A test tells you how an agent behaves on the inputs you gave it. It tells you nothing about the inputs you did not think of. In production, most inputs are ones you did not think of. That gap is not a testing bug you can fix by writing more tests, because more tests means more inputs you thought of. It is a different kind of problem and it needs different machinery.

What the guide covers

Five parts, following one agent from that Monday to something its owner can actually sign off on.

Part I is the disaster. Part II is why traditional testing breaks on systems that do not answer the same question the same way twice. Part III is how to judge behavior instead of output, which turns out to be the whole game: an agent can reach the right answer down a route you would fire a human for taking. Part IV builds a first evaluation set and rubric from nothing, including the mistakes that make a set useless (a single overall score, a dataset made entirely of traps, a rubric so vague two people score it differently). Part V is production, where the evaluation set stops being a thing you built and becomes a thing production grows for you, one surprise at a time.

It is written for someone who knows what a test is and does not want a research paper. The non-determinism problem is real and studied, but you do not need the literature to ship. You need a harness.

The idea the title is hiding

Take inventory of what James owns three months later: an evaluation set production keeps growing, rubrics precise enough that any judge lands on the same score, code judges and a calibrated AI judge, a step budget, a cost meter, checkpoints inside long tasks, alarms on the numbers that matter.

Almost none of it is the agent. The prompt is still about 40 lines. Everything else is machinery around the agent.

Engineers have a name for machinery like that. When you need to trust something powerful and unpredictable, you never take the engine's word for it. You build a test harness around it: instruments on every output, load on every input, alarms on every limit. The engine provides the power. The harness provides the knowing.

Which means "how do you know your AI agent actually works?" was never a question about the agent. It is a question about you. Did you build the harness?

Free, and free of the usual

No email gate. No signup wall. Download it, read it in the browser, print it, send it to your team. There is an email field if you want the next one, and ignoring it costs you nothing.

If you have shipped an agent and quietly wondered whether you would notice it going wrong, this is for you. If you have not shipped one yet, you get to skip James's Monday.

Next in the series: Building Your First Evaluation Dataset, because the honest weakness of everything in this book is that it all rests on 150 example messages labeled in one afternoon. That is enough to start. It is not enough to lean on for a year.