noted.do
Ailoot Engineering

Build a calibration set for an LLM judge

A judge model can compare two versions of a support answer against a rubric. Before using its verdicts to approve a prompt change, check how often it agrees with human reviewers on the kinds of cases that matter to your product.

Choose cases that test different failures

An initial set of 30 cases might include 12 ordinary support answers, 12 borderline cases and six violations of a strict requirement. Include two answers that use different wording but give the same correct next step. Include a polished answer that promises an action the assistant cannot perform. Check strict rules such as valid JSON and required fields in code before asking a model to assess the remaining qualities.

Write the human labels before running the judge

Give two reviewers the same question, reference material and rubric. Ask them to label each pair independently as baseline wins, candidate wins, or tie, and record the fact that decided their choice. Resolve disagreements before you tune the judge. Save a separate set of cases for the final check so the rubric cannot improve its reported score by learning the same examples it is judged on.

Count responses the judge cannot score

Suppose the judge returns usable verdicts for 26 of 30 cases, and 20 match the human labels. That is 20 of 26 usable verdicts in agreement, or about 77 percent. Across the whole set, 20 of 30 were both usable and correct, or about 67 percent. Record the four unusable verdicts beside the six usable disagreements. Review those categories separately and look at how errors group by case type. A judge can agree on ordinary answers yet still miss the failures that would make you halt a release.

The Ailoot guide explains when to use pointwise or pairwise evaluation and how to test common judge biases. How to Use LLM-as-a-Judge