LLM-as-judge is an evaluation technique in which one language model grades the output of another — scoring a response against a rubric, comparing two candidate answers, or checking a result for policy compliance. It substitutes scalable, repeatable model judgment for human review on tasks where correctness has no single string to match.
Why LLM-as-judge matters
Agent outputs are open-ended: summaries, plans, drafted emails, generated code, multi-step tool traces. There is rarely one correct string to compare against, so classic exact-match metrics measure nothing useful, and human review — the gold standard — cannot keep up with thousands of runs per day. Teams end up shipping prompt and model changes without knowing whether quality moved.
Model-graded evaluation makes judgment cheap enough to apply everywhere it is needed: on every change that touches a prompt, across the regression suites of agent evaluation, on samples of live traffic, and in head-to-head comparisons between candidate models or configurations. What was a review bottleneck becomes a routine, repeatable measurement.
How it works
- Define the rubric. Decide what good means for the task — faithfulness to sources, completeness, tone, policy compliance — and write it as explicit criteria with a scoring scale.
- Choose a protocol. Direct scoring grades a single output against the rubric; pairwise comparison asks which of two outputs is better; reference-guided grading compares an output to a known-good answer.
- Prompt the judge. The judge model receives the original input, the output under evaluation, and the rubric, and returns a verdict — ideally as structured output carrying a score plus a written rationale, so verdicts can be aggregated and audited.
- Calibrate. Judges have measurable biases: position bias favors the first answer in a pairwise setup, verbosity bias favors longer answers, and self-preference favors outputs from the judge’s own model family. Standard mitigations include swapping answer order and averaging, keeping rubrics narrow, judging with a stronger model than the one being judged, and periodically checking verdicts against human labels.
LLM-as-judge vs. guardrails
A judge measures; a guardrail enforces. Judging is an evaluation activity — scoring quality during or after a run so a team can track it, compare it, and catch regressions. A guardrail is a runtime control that returns pass or fail and can block a response, trigger a retry, or escalate to a human before a bad output does harm. The two compose cleanly: wire a judge’s verdict to an enforcement action and it becomes the decision logic inside a guardrail, at the cost of one extra model call of latency.
In practice
Offline, judges run inside test suites, grading fixed datasets on every change the way unit tests gate code. Online, the same rubric can score sampled production traffic for monitoring or run inline as a quality gate. Verdicts earn the most trust when they are recorded next to the run itself: storing scores alongside the execution history lets a team trace any bad grade back to the exact steps that produced it, and feeds the dashboards used for observability. See testing agents for wiring judges into an evaluation loop.