A practical, no-fluff curriculum on building and running evals for AI agents. It starts with fundamentals, LLM-as-judge, trajectory scoring, and rubric design, then deep-dives into RAG evals, production monitoring, red-teaming, hard agent shapes, and the benchmark landscape. Written for engineers, PMs, and anyone shipping agents without a grading system yet.
An agent that looks right in a demo and an agent that works in production are two different products. The difference is measurement. Evals are the tests, graders, and review habits that tell you, before your users do, where your agent succeeds, where it fails, and whether your latest change made things better or worse.
This curriculum teaches you to build that system yourself. The first four modules build the core discipline: fundamentals and your first golden cases, scaling grading with LLM-as-judge, scoring the full path your agent takes rather than just its final answer, and rubrics that turn "good" into something checkable. The rest deep-dive where agents actually live: evaluating retrieval, running evals in production, testing your agent against adversaries, handling the hard shapes (multi-agent pipelines, memory, real-world writes, voice), and learning from the public benchmarks without worshipping them. Questions at any point? .
9 modules, 36 lessons. Open any module to read its lessons, or jump straight into one below. Short on time? Jump to the short version.
What an eval actually is, why "it looks right" doesn't scale, and how to build your first small, high-signal set of test cases.
Using a model to grade a model: when it works, how to build judge prompts, and how to keep the judge honest by calibrating against human labels.
Agents don't just answer; they act. Learn to grade the full path: tool calls, retrieved context, intermediate decisions, and where a run first went wrong.
"Good" is not a spec. Rubrics turn vague quality bars into criteria a human (or a judge model) can check the same way twice.
Most agents read before they act, and retrieval is where they fail silently: the reply sounds confident while the wrong documents sit underneath it. This module makes every leg of the triangle of question, context, and answer measurable.
Your eval suite is only half the system; the other half runs where your users are. Tracing, live judges, review queues, and the CI gates that decide what ships.
Every module so far assumed a user trying to succeed. Some inputs are designed to make your agent fail. This module is about testing your own agent against them before someone else does.
The course so far graded one text agent talking to one user. Real deployments are messier (routers dispatching sub-agents, memory that outlives the session, hands that touch production, voices on a phone line), and every one of these shapes decomposes into evals you already know how to build.
Public benchmarks won't grade your product, but their grading designs are the best free education in eval construction. Steal the designs, read the scores critically, and finish by turning your own suite into a benchmark.
Start small and real. Read your agent's real failures before you write a single test. Turn each one into a golden case with a written-down expectation, and keep that case passing forever. A small set you actually read and update will teach you more than a thousand generated cases nobody looks at. Every new production failure joins the suite, so the suite grows exactly where your agent is weak.
Use the cheapest grader that works. Start with plain code checks, because exact matches and schema validations are free, instant, and never argue. Reach for an LLM judge only for qualities code cannot express, like tone or faithfulness. Before you trust that judge, label a sample yourself and check that it agrees with you. Humans stay the ground truth that everything else is calibrated against.
Grade the trajectory, not just the answer. An agent does not just reply, it acts. It calls tools, retrieves documents, and makes decisions along the way, and that path is where the bugs live. A correct answer reached by guessing will fail you later, and a friendly reply that fired the wrong tool already has. Grade the steps and the side effects, and run each case several times, because an agent that passes three runs out of five is not the same agent as one that passes five.
Make "good" checkable. Two reviewers who both want quality will still disagree about "good" until you decompose it. A rubric turns quality into concrete yes-or-no questions that two people answer the same way. Derive the criteria from failures you have actually seen, not from brainstorming. Prefer pass or fail over a 1 to 10 scale, because nobody can defend the difference between a 6 and a 7.
Evaluate retrieval on its own. Most agents read before they act, and retrieval is where they fail silently. Think of RAG as a triangle of question, context, and answer, where each leg gets its own eval. When something breaks, the metrics tell you which leg it was. A missing document needs new content, a buried chunk needs better retrieval, and an invented claim needs a prompt fix. Without the decomposition you are guessing at all three.
Instrument first, evaluate forever. Record a full trace of every run from day one, because traces are the raw material for everything else. Offline suites replay them, online judges score samples of live traffic, and review queues route the suspicious ones to a human. Evals then run on every code change, so a regression blocks the merge instead of reaching a customer. Production is not where evals stop, it is where they get their data.
Attack your own agent. Everything else assumes a user trying to succeed, but some inputs are designed to make your agent fail. Its attack surface is everything it reads, including retrieved documents and tool results that an attacker can write. Build adversarial cases the same way you build golden ones, just with hostile intent. And always measure both directions, because an agent that refuses attackers while refusing real customers has only moved the failure.
Hard shapes decompose into machinery you already own. Routers are classifiers, so grade the routing decision like any tool choice. Memory reads and writes are tool calls, so grade them as trajectory steps. Agents that write to production get a shadow mode first, where intended actions are logged and graded but never executed. Voice agents get layered metrics, because no downstream eval can fix a word the agent misheard.
Steal benchmark designs, not scores. Public leaderboards will never grade your product, but the people who built them solved grading at scale, and their designs are published. The serious ones engineer tasks whose success is verifiable by code, like a passing test suite or a correct final state. Use leaderboards to shortlist models, then let your own suite make the decision. The end state is a benchmark for your own product that anyone on the team can run and trust.
Stuck on a concept, or wondering how to build an eval for your own agent? Ask away. Answers are grounded in this curriculum and cite the lessons they come from.
buildevals is a free, open curriculum that teaches you how to build and run evals for AI agents. It spans 9 modules and 36 lessons, from fundamentals and golden cases through LLM-as-judge, trajectory scoring, rubrics, RAG evaluation, production monitoring, red-teaming, hard agent shapes, and public benchmarks.
Basically anyone interested in evals. Whether you are shipping AI agents and want a grading system instead of vibe checks, or just curious how evals work, the lessons explain each idea from the ground up, so a PM can follow them as easily as an engineer.
Yes. Every lesson is free to read. There is no account to create, no signup, and no paywall.
The tutor answers only from this curriculum. Each question is matched against the lessons, and if nothing relevant is found it declines rather than guessing. Answers cite the specific lesson they come from, so you can read the full version, and it will not answer questions unrelated to building evals.
Yes. The curriculum is free and open, so you are welcome to share links, quote it, and use it with your team. A link back to buildevals.com is appreciated.
The curriculum was last updated in July 2026. It is maintained as the field changes, and new failure modes and techniques are folded into the relevant lessons rather than bolted on as a changelog.