The judge that said 94%

November 3, 2026

Here is a strange question. If you want to know whether something works, should you ask the smartest judge you can find, or the fastest one?

Most of us answer "smartest" without thinking. Let's slow down and see why that might be wrong.

Imagine two teams. Both ship a support bot. Both want to know if it works.

Team Blue picks the smartest model available as the judge. It reads every answer carefully and thinks before it grades. Quality is great. One full eval takes an afternoon and costs real money.

So Team Blue runs it weekly. This is the sensible thing to do if you only look at the invoice.

Monday: 94%

The dashboard says 94% pass. Everyone relaxes. On Tuesday, a prompt change ships.

By Thursday, support tickets are climbing. The bot is confidently telling customers to reboot things that do not need rebooting. Nobody knows, because the next eval is on Monday.

Three days of damage. The judge was not wrong. The judge was not there.

A dashboard says 94% pass while production is on fire

Feedback loop length is the whole game. A perfect measurement that arrives late is a postmortem.

Team Red

Team Red has the same bot and the same problem. Different judge.

Theirs is a small, fast model. It does not reason out loud. It reads an answer and returns pass or fail. It is dumber than Blue's judge and about 100x cheaper. (Rough guess. Check your own bill.)

Cheap means they can run it on everything: every change, every pull request, a slice of live traffic.

Tuesday, same prompt change. Tuesday afternoon, the dashboard dips. Tuesday evening, rolled back. Customers notice nothing.

Genius judge runs once a week. Tiny judge runs on every pull request.

Why the dumb judge won

Simple math. A judge's value is roughly:

how right it is x how often it runs

Blue maximized the first number. Red maximized the second. For catching regressions, the second number is bigger. A 90% accurate judge that runs on all traffic finds more bugs than a 99% accurate judge that runs on a sample.

Speed also changes behavior. When an eval takes seconds, engineers run it before they push. When it takes hours, they push and hope. People respond to the cost of feedback. This is not a morale problem. It is physics, with humans in it.

Where the dumb judge breaks

Of course it breaks. Here is how.

It passes confident, fluent, wrong answers. Wrong with good grammar is still wrong.
It misses bad math. Anything that needs three steps to check, it checks in zero.
On close calls between two good answers, it is a coin flip with opinions.
It tends to like text that looks like its own.

Fine. Engineer around it.

1. Ask small questions. "Is this good?" is not a question. "Does the answer use only facts from the help article?" is. Small models are fine at narrow yes/no checks. Split a big rubric into five small ones. Bonus: when something fails, you know which check failed.

2. Calibrate against humans. Hand-label a few hundred real outputs. Measure agreement. If the judge disagrees with you, the judge is wrong until proven otherwise. Fix the prompt, not the dashboard.

3. Layer them. Fast judges run on everything. When they disagree, or a score is near the line, escalate to a slow reasoning model. Send a small sample of those to a human. Feed the human labels back to tune the fast judge. Cheap on the common case, careful on the hard one.

4. Swap the order. For A vs B tests, also run B vs A. Position bias is real. If the answer flips, the answer is "tie".

When not to do this

The task needs real reasoning to verify: math, code logic, legal text.
A wrong verdict is expensive and no human looks.
You have not calibrated yet. Uncalibrated judges are vibes with an API.

One more way to see it

Step back from bots for a moment.

A judge is a way of turning a messy world into a number. We build one because we cannot hold the whole world in our heads. The number is a promise: "this is what is happening."

Now notice what Blue's number and Red's number have in common. Neither one is the truth. Each is a small window onto it. The only real difference is how often the window opens, and how much we have checked that the glass is clear.

So the question was never "which judge is smartest?" It was "how quickly will I find out I am wrong?"

The most dangerous number in any system is the one we stopped questioning. A fast judge, checked against people, keeps asking.

Rule: never trust a judge you have not checked against a human.

Team Blue and Team Red are made up. The failure mode is not.