# buildevals.com

> A free, practical curriculum on building and running evals for AI agents: fundamentals, LLM-as-judge, trajectory scoring, rubrics, RAG evaluation, production monitoring, red-teaming, hard agent shapes, and benchmarks. 9 modules, 36 lessons, written for engineers and PMs. By Tamas Szuromi (https://x.com/tamas_szuromi).

Full curriculum text: https://buildevals.com/llms-full.txt

## [Module 1: Fundamentals](https://buildevals.com/modules/1-fundamentals)

What an eval actually is, why "it looks right" doesn't scale, and how to build your first small, high-signal set of test cases.

- [Lesson 1.1: What is an eval, really?](https://buildevals.com/lessons/1-1-what-is-an-eval-really): Inputs, graders, and scores. Key idea: An eval is a repeatable question with a checkable answer.
- [Lesson 1.2: Vibes vs. measurement](https://buildevals.com/lessons/1-2-vibes-vs-measurement): Why demos deceive. Key idea: Intuition finds hypotheses; only measurement can confirm them.
- [Lesson 1.3: Build your first golden cases](https://buildevals.com/lessons/1-3-build-your-first-golden-cases): Cases you refuse to break. Key idea: Ten cases you maintain beat a thousand you generated and never read.
- [Lesson 1.4: Choosing what to measure](https://buildevals.com/lessons/1-4-choosing-what-to-measure): Critical paths over coverage. Key idea: Measure the paths users actually take, in metrics that name the broken component.

## [Module 2: LLM-as-judge](https://buildevals.com/modules/2-llm-as-judge)

Using a model to grade a model: when it works, how to build judge prompts, and how to keep the judge honest by calibrating against human labels.

- [Lesson 2.1: When a model should grade a model](https://buildevals.com/lessons/2-1-when-a-model-should-grade-a-model): And when code should instead. Key idea: Use the cheapest grader that works: code first, judge for the fuzzy remainder, humans to calibrate.
- [Lesson 2.2: Writing judge prompts](https://buildevals.com/lessons/2-2-writing-judge-prompts): Criteria, examples, output shape. Key idea: One criterion, a concrete definition, real examples, reasoning before a binary verdict.
- [Lesson 2.3: Calibrating against human labels](https://buildevals.com/lessons/2-3-calibrating-against-human-labels): Trust, but verify the judge. Key idea: A judge's scores mean nothing until they agree with a human you trust, on data you labeled.
- [Lesson 2.4: Common judge failure modes](https://buildevals.com/lessons/2-4-common-judge-failure-modes): Position bias, leniency, drift. Key idea: Judges fail systematically: design against the biases and spot-check forever.

## [Module 3: Trajectory evals](https://buildevals.com/modules/3-trajectory-evals)

Agents don't just answer; they act. Learn to grade the full path: tool calls, retrieved context, intermediate decisions, and where a run first went wrong.

- [Lesson 3.1: Why the path matters](https://buildevals.com/lessons/3-1-why-the-path-matters): Right answer, wrong reasons. Key idea: Agents produce trajectories, not answers, and the bugs live in the trajectory.
- [Lesson 3.2: Scoring tool calls](https://buildevals.com/lessons/3-2-scoring-tool-calls): Order, arguments, side effects. Key idea: Grade tool calls as structured data: constraints on calls and effects, not one blessed sequence.
- [Lesson 3.3: Loops, detours & dead ends](https://buildevals.com/lessons/3-3-loops-detours-and-dead-ends): Finding the first real failure. Key idea: Find the first divergence; everything after it is contamination, not signal.
- [Lesson 3.4: End-to-end vs. step-level checks](https://buildevals.com/lessons/3-4-end-to-end-vs-step-level-checks): Choosing the right granularity. Key idea: Step-level tells you what broke, end-to-end tells you whether it matters; run both.

## [Module 4: Rubrics](https://buildevals.com/modules/4-rubrics)

"Good" is not a spec. Rubrics turn vague quality bars into criteria a human (or a judge model) can check the same way twice.

- [Lesson 4.1: From "good" to checkable criteria](https://buildevals.com/lessons/4-1-from-good-to-checkable-criteria): Decompose quality into checks. Key idea: A rubric is quality decomposed into yes/no questions two graders answer the same way.
- [Lesson 4.2: Designing score scales](https://buildevals.com/lessons/4-2-designing-score-scales): Binary beats 1 to 10, usually. Key idea: Precision comes from more cases, not more points on the scale.
- [Lesson 4.3: Pass/fail vs. graded rubrics](https://buildevals.com/lessons/4-3-passfail-vs-graded-rubrics): Matching stakes to scoring. Key idea: Binary per criterion, aggregated to fit the decision, and critical failures veto everything.
- [Lesson 4.4: Keeping rubrics honest over time](https://buildevals.com/lessons/4-4-keeping-rubrics-honest-over-time): Prune, recalibrate, repeat. Key idea: A rubric is a living document: prune it, refresh it from real failures, and version it like code.

## [Module 5: RAG & retrieval evals](https://buildevals.com/modules/5-rag-and-retrieval-evals)

Most agents read before they act, and retrieval is where they fail silently: the reply sounds confident while the wrong documents sit underneath it. This module makes every leg of the triangle of question, context, and answer measurable.

- [Lesson 5.1: The six relationships](https://buildevals.com/lessons/5-1-the-six-relationships): The complete RAG eval space. Key idea: Every RAG eval is one of six directional relationships among question, context, and answer.
- [Lesson 5.2: Measuring retrieval](https://buildevals.com/lessons/5-2-measuring-retrieval): Recall, rank, synthetic questions. Key idea: Label retrieval by generating questions from your own chunks, then read recall per query class, not in aggregate.
- [Lesson 5.3: Faithfulness, step by step](https://buildevals.com/lessons/5-3-faithfulness-step-by-step): Grade claims, not answers. Key idea: Faithfulness is supported claims over total claims: decomposed, verified against the context, no golden answer required.
- [Lesson 5.4: The retrieval flywheel](https://buildevals.com/lessons/5-4-the-retrieval-flywheel): Localize, segment, fix, repeat. Key idea: Localize each failure to a triangle leg, segment metrics by query class, and fix where volume meets failure rate.

## [Module 6: Evals in production](https://buildevals.com/modules/6-evals-in-production)

Your eval suite is only half the system; the other half runs where your users are. Tracing, live judges, review queues, and the CI gates that decide what ships.

- [Lesson 6.1: Tracing: instrument everything](https://buildevals.com/lessons/6-1-tracing-instrument-everything): Spans, metadata, one standard format. Key idea: Traces are the substrate of every production eval: capture complete spans in a standard format, with PII removed at the source.
- [Lesson 6.2: Online evaluation](https://buildevals.com/lessons/6-2-online-evaluation): Judges on live traffic. Key idea: Score sampled live traffic with the judges you calibrated offline, and eval the guardrails as strictly as the agent.
- [Lesson 6.3: Human review workflows](https://buildevals.com/lessons/6-3-human-review-workflows): Queues, labels, closed loops. Key idea: Route the right traces to a human in a frictionless tool, and turn every review into a golden case or a calibration example.
- [Lesson 6.4: CI gates and experiments](https://buildevals.com/lessons/6-4-ci-gates-and-experiments): When each suite runs. Key idea: Version the suite with the code, gate on bars set before the run, and read the case-level diff, not just the delta.

## [Module 7: Adversarial & safety evals](https://buildevals.com/modules/7-adversarial-and-safety-evals)

Every module so far assumed a user trying to succeed. Some inputs are designed to make your agent fail. This module is about testing your own agent against them before someone else does.

- [Lesson 7.1: The agent attack surface](https://buildevals.com/lessons/7-1-the-agent-attack-surface): Everything the agent reads. Key idea: An agent's attack surface is everything it reads plus everything its tools can do.
- [Lesson 7.2: Building adversarial suites](https://buildevals.com/lessons/7-2-building-adversarial-suites): Golden cases with hostile intent. Key idea: Adversarial cases are golden cases with hostile intent, graded by attack success rate per class.
- [Lesson 7.3: Guardrails and how to eval them](https://buildevals.com/lessons/7-3-guardrails-and-how-to-eval-them): The guardrail is a classifier. Key idea: A guardrail is a classifier: score it on an attack corpus and a benign corpus, or you're guessing.
- [Lesson 7.4: Refusals, both directions](https://buildevals.com/lessons/7-4-refusals-both-directions): Blocking attacks, not customers. Key idea: Report under-refusal and over-refusal together: improving one silently degrades the other.

## [Module 8: Hard agent shapes](https://buildevals.com/modules/8-hard-agent-shapes)

The course so far graded one text agent talking to one user. Real deployments are messier (routers dispatching sub-agents, memory that outlives the session, hands that touch production, voices on a phone line), and every one of these shapes decomposes into evals you already know how to build.

- [Lesson 8.1: Routers, sub-agents, handoffs](https://buildevals.com/lessons/8-1-routers-sub-agents-handoffs): Grade the seams too. Key idea: A multi-agent system is a classifier, some handoff artifacts, and seams: grade each in isolation, then grade the whole.
- [Lesson 8.2: Memory and long sessions](https://buildevals.com/lessons/8-2-memory-and-long-sessions): Replay, correct, remember. Key idea: Grade the session before the turn, replay real prefixes to grade the next decision, and test memory with a fresh session.
- [Lesson 8.3: Agents that touch the world](https://buildevals.com/lessons/8-3-agents-that-touch-the-world): Shadow first, act later. Key idea: Shadow mode grades real traffic with zero side effects: clear a pre-committed bar there before the agent acts alone.
- [Lesson 8.4: Voice and other modalities](https://buildevals.com/lessons/8-4-voice-and-other-modalities): One layer at a time. Key idea: Score each layer in order (transcription, then text, then voice) because no downstream eval fixes an upstream miss.

## [Module 9: Benchmarks & the landscape](https://buildevals.com/modules/9-benchmarks-and-the-landscape)

Public benchmarks won't grade your product, but their grading designs are the best free education in eval construction. Steal the designs, read the scores critically, and finish by turning your own suite into a benchmark.

- [Lesson 9.1: How the big benchmarks grade](https://buildevals.com/lessons/9-1-how-the-big-benchmarks-grade): Grading designs worth stealing. Key idea: Serious benchmarks don't ask a judge: they engineer tasks whose grading is verifiable.
- [Lesson 9.2: Preference evals and Elo](https://buildevals.com/lessons/9-2-preference-evals-and-elo): Blind votes, Elo ratings. Key idea: Arena ratings say which model people prefer in chat, not which one runs your agent best.
- [Lesson 9.3: What leaderboards can't tell you](https://buildevals.com/lessons/9-3-what-leaderboards-cant-tell-you): Reading scores with suspicion. Key idea: Use leaderboards to shortlist models; use your own suite, on your own scaffold, to decide.
- [Lesson 9.4: Build your own benchmark](https://buildevals.com/lessons/9-4-build-your-own-benchmark): Your suite, benchmark-grade. Key idea: A benchmark is an eval suite with a pinned environment and a results format your team can trust.