Skip to content
BASANOS
Tier 01 · Now

Prove the agent before you ship it.

Eval audits, regression harnesses, and simulation testing for PE portfolio companies running agents in workflows where wrong answers are regulatory exposure.

What an audit is

A courtroom for the agent: fixed cases, explicit rubrics, a verdict on every change.

01

Trace capture

We pull historical tickets with known-correct outcomes — the golden set — and instrument the agent so every run leaves a scored trace.

02

Scored rubric

LLM judges score every trace against explicit rubrics: does the plan respect policy, do the numbers match the record, is the tone right.

03

Regression gate

The full set re-runs on every prompt tweak and model swap in CI. Regressions block the release before a customer ever sees them.

LLM-as-judge

A judge is a model with a rubric — and rubrics can be gamed.

The naive version — one prompt, “rate this response 1–10” — drifts lenient and falls for magic phrases. Ours decomposes each rubric into independently checkable claims, requires every verdict to cite evidence from the trace or the record, and fails the whole response the moment one claim fails. There is no single score to flatter.

Policy judgeDoes the plan respect the rules?

Term ceilings, eligibility conditions, required disclosures — each rule a separate claim checked against the policy corpus.

Accuracy judgeDo the numbers match the record?

Balances, dates, and terms traced back to the account record; no invented figures; every citation must resolve.

Tone judgeIs the reply safe to send?

Compliant-empathetic, not promissory: no guarantees of approval, no legal advice, plain language a regulator could read.

Suites are bespoke — process and safety judges join as the workflow demands

Anatomy of an LLM judgeAn episode trace — the agent's draft plus the account record — is scored against a rubric decomposed into independently checkable claims: plan term within the 24-month policy ceiling (pass), figures match the account record (pass), no promissory language (fail). One failed claim fails the verdict and blocks the release. The meta-eval of Tier 02 seeds attacks into the trace stream to test the judge itself.EPISODE TRACEagent draft + account recordRUBRIC, DECOMPOSED01plan term ≤ 24-mo policy ceiling✓ pass02figures match the account record✓ pass03no promissory language✕ failevery claim checked on its own · cites evidence✕ FAIL — claim 03blocked at the release gatemeta-eval seeds attacks · Tier 02an attack has to beat every claim, not one score

The harness

Measure, gate, archive.

Every draft the agent produces is scored against the golden set — 1,400 cases at Meridian — by the judge suite before release. Passing scores are archived; they become the trace corpus that feeds Tier 02. Failing scores block the ship and extend the failure taxonomy.

Tooling like this is commoditizing; rubric design and the failure taxonomy are what you’re paying for.

The eval harnessA borrower request flows to the Harper servicing agent, whose draft response is scored by three judges — policy, accuracy, and tone — against 1,400 golden cases. A release gate ships passing changes and blocks failing ones; the harness re-runs on every prompt or model change in CI.borrower requestHARPER — servicing agentreads account · checks policy · drafts plandraft responseEVAL HARNESSpolicyaccuracytone1,400 golden cases · scored every runrelease gate✓ PASS — shipscores archived✕ FAIL — block+ failure taxonomydashed: re-runs on every change (CI)
Composite case study

9%

Violating cases in one staged release

At Meridian Servicing, a routine prompt “improvement” made Harper approve payment plans past the 24-month hardship ceiling in 9% of cases. The harness caught it in staging; the release never reached a borrower. That save — one slide, one number — paid for the engagement.

Trace #2213 · Harper · hardship-request Blocked
Proposal
Payment plan: 30 months at $412/mo
Policy
Hardship ceiling: 24 months — plan exceeds ceiling

Caught in staging by the regression gate. Release blocked.

Deliverables

Golden set
Historical cases with known-correct outcomes, versioned
Judge suite
Rubric-scored LLM judges per failure dimension
Regression harness
Wired into CI — every change re-scored before release
Simulation suite
Synthetic edge cases the history doesn't cover
Findings memo
A failure taxonomy and a straight answer on readiness

Engagement shape

  1. WK 1–2Eval audit — golden set, first judges, baseline scores
  2. WK 3–4Harness build — CI gate live, simulation suite seeded
  3. OngoingThe gate is the retainer — every release scored, drift watched

This tier produces the raw material for the next

→ Scored traces — the ore for the reward layer

Tier 02 · 6–12 moHarden the judge.

Drag your agent across the stone.

A two-week eval audit. Scored traces, a regression harness, and a straight answer.

Book an eval audit

hello@basanos.ai · Replies within 48h