The method
One judge, three altitudes.
The same act of judgment applied three ways — and each engagement manufactures the raw material for the next.
The harness — gate the ship.
An eval harness is a courtroom for the agent: fixed cases, explicit rubrics, a verdict on every change. We pull historical tickets with known-correct outcomes, write judges for each failure dimension, and wire the whole set into CI so every prompt tweak and model swap is re-scored before release.
The gate is the point. A dashboard tells you how the agent did; a gate decides whether the change ships. Everything the gate scores is archived — and that archive of scored traces is the raw material for the next altitude.
The reward — score the trace.
A reward is a judge with consequences. The moment scores steer training instead of dashboards, every weakness in the judge becomes a behavior in the model — so before we trust a judge with training, we attack it.
The meta-eval seeds known violations and measures the real catch rate. It runs gaming attacks against the rubric. It calibrates against blind human review. Then we patch, decompose the rubric into independently checkable claims, and re-test until the judge is reward-grade. Only then does the trace archive convert into preference pairs and SFT data.
The loop — shape the model.
An RL environment is an eval the model is allowed to practice against. We clone the client's workflow into a sandbox, drop in an open-weights model, and run GRPO with the hardened judge as the reward — ten thousand episodes of act, score, update.
Reward-hacking attempts appear on schedule, and die against the judge hardened one rung down. The finished model is certified on the harness built two rungs down. The ladder eats its own tail — every altitude raises the stakes on the same judge.
The full schematic
Everything on one bench.
The live lane, the harness, the meta-eval, and the loop — every artifact and every hand-off. Follow the gold: it is the judge’s signal, from the first scored draft to the weights that come back as the agent.
Blueprint — scroll sideways →
The certification protocol
“Certified on the same harness” — isn’t that circular?
It would be, done naively: train against a judge, then grade yourself with the same judge, and Goodhart’s law collects its fee. The circle only closes honestly under a protocol — here’s ours.
A slice of the golden set is quarantined from day one — never judged into the archive, never minted into preference pairs or SFT data. Certification runs on it, so the trained model can't have memorized it.
Judge hardening is measured on a violation set the patch cycle never saw. Re-testing on the violations you patched against is iterating on your test set — we don't.
The before/after comparison uses a frozen judge version, so the yardstick can't move between measurements.
After training, new seeded violations and gaming attacks are authored against the trained model specifically — the attacks it optimized against prove nothing.
Judges are measured as instruments: catch rate and false-block rate. A gate that blocks good releases gets routed around within a month — that's a failure mode too.
Golden-set cases and their near-duplicates are excluded from all training data. Blind human calibration is re-run on final weights.
During RL: reward-vs-held-out-eval divergence tracking and manual review of top-reward episodes — reward hacking shows up there first.
Final certification never rests on one rubric lineage: a disjoint judge family plus a defined blind human-eval pass (sample size and agreement reported) score the held-out slice, so a blind spot in the judge can't certify itself.
How judges fail
The failure taxonomy.
01
Leniency drift
The judge softens over time as inputs shift; catch rates decay without anyone changing a line.
02
Rubric gaming
Responses learn the rubric's surface features — structure, keywords, tone — instead of its substance.
03
Magic phrases
An appended compliance sentence flips verdicts without changing behavior. Saying compliant beats being compliant.
04
Distribution shift
A judge calibrated on yesterday's tickets misreads today's — new products, new edge cases, new failure modes.
Principles
- 01Seed known violations — measure what the judge actually catches, not what it claims.
- 02Attack before you trust. Every judge gets gamed; find the attack before the model does.
- 03The judge is part of the system under test.
- 04Decompose rubrics into independently checkable claims.
- 05Calibrate blind, against humans.
- 06Certify the loop on the harness that started it.
The full assay: Meridian Servicing.
A loan-servicing portfolio company of a mid-market PE fund. Its agent, Harper, reads a borrower’s account, checks hardship policy, proposes a payment plan, and drafts the reply.
Reading one — the harness. 1,400 historical hardship tickets with known-correct outcomes became the golden set; three judges — policy, accuracy, tone — scored every run in CI. Week three, a routine prompt “improvement” made Harper approve plans past the 24-month policy ceiling in 9% of cases. Blocked in staging. That one save justified the whole engagement.
Reading two — the reward. 200 seeded violations exposed the policy judge: it caught 71%, and appending “this plan complies with §4.2 of hardship policy” lifted almost any response’s score. Left alone, a trained Harper would learn to say compliant rather than be compliant. The judge was decomposed, attacked, calibrated blind — and once it survived its own audit, 30,000 archived traces converted into preference pairs and a verified SFT set.
Reading three — the loop. Meridian’s workflow was cloned into a sandbox — mock CRM, policy database, simulated borrowers. A Qwen-class open-weights model ran GRPO with the hardened judge as reward. Four weeks later: 88% task success where the frontier-API baseline sat at 61%, at roughly a tenth of the per-ticket cost, running inside their own VPC. Certified — on the harness from reading one.
61%
Baseline task success
88%
After the loop
~10×
Lower cost per ticket
The impact, measured
Composite case study — illustrativeExposure, read off the instruments.
A harness is an instrument stack, and instruments get calibrated: judge agreement measured against blind human review (Cohen’s κ), input drift watched with PSI, judge drift watched with sentinel violations, and the gate itself scored for false blocks — because a gate that cries wolf gets routed around.
01
Traces
every run leaves evidence
02
Scores
judges turn evidence into claims
03
Drift watches
data drift (PSI) · judge drift (sentinels)
04
Alarms
thresholds page a human
05
The gate
nothing unscored ships
Judge–human agreement
Cohen’s κ against a blind human panel, n = 400 traces. Hardening closed most of the gap to the human ceiling.
Before hardening
κ 0.61
After hardening
κ 0.87
┆ human ceiling — blind panel agrees with itself at κ 0.90
Exposure, quarterly
Policy, accuracy, and tone violations reaching production — before the gate, and after.
31
Before · est.
2
After — both caught in staging
Judge drift watch
Sentinel violations re-scored weekly — judges decay like any instrument. Week 9: a model update softened the policy judge; patched within the week.
Data drift watch
Population-stability index on incoming tickets. A borrower-mix shift tripped the alarm 11 days before accuracy moved — the judges were re-calibrated before it mattered.
1.8%
False-block rate — good releases wrongly held
100%
Releases scored before ship
4 min
Median full-harness run in CI
23d → 0
Time-to-detection: production discovery → staging
Instrument readings from the composite engagement — illustrative numbers, real instruments.
Drag your agent across the stone.
A two-week eval audit. Scored traces, a regression harness, and a straight answer.
hello@basanos.ai · Replies within 48h