Costiva
Home New to AI
Tools

All 47 tools → Features Use cases Live demo Impact analytics Pricing Blog FAQ About
Sign in Get started free
Free tool

LLM-Judge Calibrator

Compare judge verdicts against human labels and find out whether you can trust automated grading — agreement rate, Cohen's kappa, bias checks, and a concrete recommendation.

Get started
  • Cohen's kappa, not a bare percentage — corrects for the agreement two raters would show by chance alone, so a high number actually means something.
  • Checks the three most common quiet judge failure modes — favoring longer answers, favoring hedged language, and favoring answers from its own model family — each with a sample-size caveat, never a causal claim.
  • A trust threshold you can see, and a warning wired straight into the Eval Runner: once a judge model calibrates below the line, every scorecard using it says so.
Get started

Frequently asked questions

The calibration itself (agreement rate, Cohen's kappa, confusion matrix, bias checks, recommendation) is deterministic PHP — no model call. The optional "Write a calibration report" button uses Claude (Anthropic) to turn those already-computed numbers into a short narrative.

You provide them — either by labeling the cases from a completed LLM-as-judge Eval Runner run directly in this tool, or by uploading a CSV/JSON file that already pairs a judge verdict with a human verdict per case.

Yes. Calibrating against your own Eval Runner history reads your organization's data and requires a Team plan.

This tool uses Cohen's kappa ≥ 0.60 (substantial agreement, on the standard Landis & Koch scale) as the line — shown on every result, never applied silently. Below it, the recommendation says so plainly and suggests what to do next.

The next time you view a scorecard from a run that used the same judge model, the Eval Runner shows a warning banner naming the calibration result — so a low-trust judge is never silently relied on elsewhere in the product.