I found two kinds of cheap. One cut my judge calls by 30% and returned the same verdicts. The other cut them further and promoted weak answers to passes. What set them apart was the direction of the errors rather than the error rate.
You can run an LLM-as-a-judge from a CLI for very little. The catch is that you need to tell a free saving from a flattering one, and you can’t do that until you’ve measured how much your judge already disagrees with itself.
TL;DR: Run the judge over the same examples more than once before trusting any number it produces. Then use sequential majority-of-three, strip the judge’s context, and reject any shortcut whose verdict changes are larger or more one-directional than ordinary run-to-run variation.
A cheaper judge that is wrong in one direction is a lie with a smaller invoice.
How I measured this
Every number below comes from one harness, so you can compare them with each other:
- The task is 100 questions. For each, the judge reads a question and a payload and returns one of
full,partial, orno, plus a separate boolean for a specific failure mode I track (a figure attributed to the wrong entity). - The headline is the count of
fullout of 100. When I say a score “moved,” I mean that count changed. When I say a verdict moved, I mean an individual question’s label changed, in either direction, by any distance. - The judge is
claude -p, modelclaude-opus-4-8, one fresh process per ballot, verdict by majority of three. - There are two comparison sets: the 100-question run above, plus a longer history of 204 recorded ballot triples used for the ballot-rule arithmetic.
Keep “verdicts moved” and “the score moved” apart as you read. A shortcut can move a dozen verdicts and barely touch the headline if the changes cancel, so the thing to watch is which way they point.
1. Measure the noise floor directly, and do not infer it
Before you trust any result, run the same examples twice under identical configuration: same model, same prompt, nothing changed between the two passes.
When I did this, 13 of 100 verdicts changed, and the headline went from 71 to 67. Of those 13 changes, 3 moved up and 10 moved down. Across four identical passes the headline landed on 74, 71, 67, and 65.
Two things follow, and I want to be careful about how far they reach:
- The observed spread across four passes is 9 points, and the largest swing between any two passes was 4 points. So I treat single-digit differences in this harness as unstable unless they repeat across runs. That’s an empirical stability range, not a confidence interval, and I haven’t computed one.
- I don’t call a difference a finding unless it clears that spread. The mode comparisons I do report clear it by 9 and 15 points. A 4-point gap in a subgroup does not, no matter how much I want it to.
Don’t shortcut this by estimating instability from ballots you already have. I tried, and depending on the estimator the same 204 recorded triples implied a floor anywhere from 7% to 35%. Three ballots can’t pin a distribution. Worse, an earlier draft of my results quoted the 7% figure as if it were the floor, which made an 11-point comparison look like a real effect. Re-running measured 13%. So pay for the second pass, because every claim you make afterward gets calibrated against it. It’s the same argument as running an agent trial more than once before quoting its reliability, pointed at the instrument instead of the agent.
2. Spend ballots one at a time
You can’t trust a single ballot on the questions that divide the judge. Across 204 triples, 16% split at least one ballot, and 5.7% of all recorded ballots differed from the majority their own triple settled on. Most questions are stable; the ambiguous ones can land on either side. So majority-of-three is the floor for anything you plan to quote.
That doesn’t mean you pay for three ballots per question. Spend them one at a time and stop once the remaining ballots cannot change the outcome:
The stopping test I use is “the leader beats the runner-up by more than the ballots still unspent” rather than “two ballots agreed,” because it generalises to any number of votes:
def decided(cast: list[str], total: int) -> bool:
"""True when unspent ballots cannot change the plurality."""
remaining = total - len(cast)
counts = sorted(Counter(cast).values(), reverse=True)
best = counts[0] if counts else 0
runner_up = counts[1] if len(counts) > 1 else 0
return remaining <= 0 or best > runner_up + remaining
cast: list[str] = []
for _ in range(total):
cast.append(judge_one(question, payload)["answerable"])
if decided(cast, total):
break
If your ballot returns more than one field, every field has to settle before you stop. Mine also returns a boolean flag, and a flag needs a strict majority of all votes rather than a plurality of those cast, so a 1-of-2 flag is still undecided even when the label is settled.
Replayed over those triples this costs 2.11 calls per question instead of 3, and my 100-question run spent 2.08. That’s about a 30% saving.
It’s also lossless relative to majority-of-three: you always get the same majority that three scheduled ballots would have produced, because two agreeing votes already are that majority. That’s a guarantee about reproducing the procedure. It doesn’t say you’re any closer to ground truth.
3. Check the direction of a shortcut’s errors, not just its price
I haven’t seen this tip anywhere else, and it cost me the most to learn.
There’s an obvious further saving, and you may have thought of it already: take one ballot, and only escalate to three when it comes back as anything other than a pass. Most answers pass, so most questions cost one call. I measured it at 1.76 calls per question, cheaper than 2.11.
I rejected it. It graded 14 partial and 4 non-answers as full.
Look at where those errors land. The rule only escalates when the first ballot is not a pass, so it can promote a weak answer to a pass but can never demote a pass. All 18 errors pushed the headline the same way: up.
When a rule is wrong in random directions, you get noise you can quantify. When it’s wrong in one direction, you get a bias, and you will publish it. Before adopting any sampling shortcut, work out which way its mistakes point. If the answer is “always the flattering way,” the money you saved bought a worse number that looks better.
4. Strip the judge’s context, then check churn and direction
A judge needs the payload and the rubric. It has no use for your tools, project settings, or MCP servers, and stripping them is the largest lever on input tokens:
claude -p "$rubric_and_payload" \
--setting-sources '' \
--strict-mcp-config \
--mcp-config '{"mcpServers":{}}' \
--output-format json
That took me from 20,704 to 7,170 fresh input tokens per call, roughly a 65% cut.
Then validate it. A changed-verdict count alone isn’t enough here, for the reason tip 3 gives: counting churn hides direction. So compare both against the noise floor:
| identical config | context stripped | |
|---|---|---|
| verdicts changed | 13 | 11 |
| moved up / down | 3 / 10 | 4 / 7 |
| headline | 71 → 67 (net -4) | 74 → 71 (net -3) |
The stripped run is smaller on churn and smaller on net movement than two identical runs of the same configuration. The wrong-attribution flag also fell from 13 to 9, which is the direction of fewer false alarms.
That’s enough for me to adopt it. It isn’t proof that accuracy is unchanged, and I wouldn’t claim that from one paired comparison. So the rule I apply is deliberately weaker than “it changed nothing”:
A cost saving is a candidate for adoption when its churn and its net score shift are both no larger than ordinary run-to-run variation.
Without a measured noise floor you can’t evaluate that sentence at all, which is why tip 1 comes first.
5. Cache ballots so replays stay independent
Caching is what makes re-measuring affordable, and it’s easy to build it in a way that silently destroys your majority vote.
The trap: if you key a cache on the prompt alone and store one value, then ballots 2 and 3 of a triple hit the same key and get served ballot 1. Every majority becomes unanimous, and the replication the whole design depends on is gone.
So key on the prompt, but store a queue, and consume one entry per request:
def cache_key(model: str, prompt: str) -> str:
raw = f"{model}\0{prompt.strip()}".encode()
return hashlib.sha256(raw).hexdigest()
def take(cache: dict[str, list[dict]], key: str):
"""Consume one stored ballot for this key, if any remain."""
pending = cache.get(key)
return pending.pop(0) if pending else None
def record(key: str, ballot: dict) -> None:
"""Append the instant it completes, not at end of run."""
with CACHE.open("a") as fh:
fh.write(json.dumps({"key": key, "ballot": ballot}) + "\n")
The cache loads as key -> list of ballots, and pop(0) is what keeps the replicas independent: request one takes b1, request two takes b2, request three takes b3. When the queue is empty, you pay for a real call and append the result.
Three identical prompts still make three real calls the first time. On replay they’re served back in order, which reproduces the original independent samples instead of collapsing them into one cached response. Hashing the full prompt also means changing the rubric or payload changes the key, so a stale ballot can never be served against new input.
Append each ballot the moment it completes, not at the end of the run. An earlier version of mine was killed 26 questions in and threw away 53 completed calls.
6. Keep the verdict space small and categorical
My judge returns one of three labels plus one boolean flag. I didn’t ask it for a 0-to-10 score, and I’d suggest you don’t either.
Ask a judge for a number and it will hand you 7-versus-8 distinctions it can’t make, and you can’t take a majority vote of a continuous score without inventing precision that was never there. Three buckets vote cleanly, disagreements are legible (partial against full is a different problem from full against no), and the flag captures the one failure mode I care about without polluting the main scale.
It also makes the noise floor interpretable. “13 verdicts changed” means something you can go and read. “The mean score moved 0.4” does not.
The workload, end to end
For 100 questions you’re looking at about 2.1 calls each, roughly 7,000 input tokens per call, plus one extra full pass to measure the noise floor. That second pass isn’t one call per question, because you need a real majority verdict to compare against, so it costs the same 2.1 calls each. My two passes spent 208 and 215 ballots. For this harness, a result you can defend cost about 420 judge calls.
I’ve deliberately not converted that to dollars. claude -p bills against a CLI subscription rather than per-token API pricing, so the token count above is a measure of workload and rate-limit pressure, not an invoice. If you run the same design against a metered API, multiply your own rate by those numbers and state the pricing date, because it will move.
The absolute cost matters less than whether the harness is cheap enough to re-run on every meaningful change. If you can only afford to run an eval once, all you have is an anecdote.
If you’re wiring judged checks into CI, I built CI for my AI agent covers the pipeline side, and how to evaluate LLM agents covers what to grade before reaching for a judge at all. A judge is the last resort, for what assertions can’t check. Deterministic checks are free, they never drift, and evaluating a server with an LLM found real bugs with a fraction of this machinery.
Measure the instrument before you optimize it
Run-to-run variation tells you which apparent improvements are real, which subgroup differences to delete before publishing, and which cost cuts need a second look. Without it you’re guessing about your own measuring instrument while using it to measure something else.
Then ask which way the errors point, because that is what separated the two shortcuts in this post. The cheap, unbiased one was an optimization. The cheap, flattering one gave me a nicer-looking wrong answer.
