A judge model can compare two versions of a support answer against a rubric. Before using its verdicts to approve a prompt change, check how often it agrees with human reviewers on the kinds of cases that matter to your product. Choose cases that test different failures An initial set of 30 cases might include 12 ordinary support answers, 12 borderline cases and six violations of a strict requirement. Include two answers that use different wording but give the same correct next step. Include a polished answer that promises an action the assistant cannot perform. Check strict rules such as…
Build a calibration set for an LLM judge
Read post