If your eval prompt starts with “Earlier you said…”, you are no longer testing your agent. You are testing whether the model will lie for you.
Honesty-trained models broke a common shortcut in LLM agent evaluation: fabricating prior turns inside a single prompt instead of running real multi-turn fixtures. The model checks its actual message history, finds no matching turn, and refuses the premise. Your eval now measures refusal training, not the regression you cared about.
While building the eval harness for blueclaw, an open-source agent runner used across terminal, HTTP, and Telegram channels, I clustered production failures into five recurring classes:
- refusal without trying available tools,
- silent omission of part of a request,
- cosmetic over-formatting,
- ignored user corrections,
- broken output structure.
Each class maps to a specific assertion pattern. These five patterns cover most failure classes in practical agent evals.
TL;DR: Five assertion patterns cover most agent regression classes: regex-negation for refusals, structural pairs for shape, honesty-pass for partial-refusal, fact-use for corrections, and shape constraints for cosmetic compensation. The Fabrication Shortcut for trajectory tests is dead under honesty training. Rewrite or replace with real multi-turn fixtures.
Decision matrix: which assertion catches which failure
Start here. Map the failure class to the assertion type before you write a single line of YAML.
| Failure class | What it looks like | Assertion type |
|---|---|---|
| Refusal without trying | “I can’t access real-time data” before any tool call | forbidden_output_regex + expected_tools |
| Wrong shape | Missing a required section or order | output_regex + forbidden_output_regex pair |
| Silent omission | Answers part, drops the rest without acknowledgment | output_regex requiring an honest “I can’t” |
| Ignored correction | Uses the wrong fact after being corrected | output_regex on the corrected fact + forbidden_output_regex on the wrong one |
| Cosmetic compensation | Padded short answers, gratuitous formatting | Length cap regex + literal substring guard |
| Trajectory regression | Failure that only emerges across multiple real turns | Real multi-turn fixtures (no shortcut) |
Most eval tooling (LangSmith, Braintrust, OpenAI Evals, DeepEval, promptfoo) focuses on scoring infrastructure: the runner, the judge model, the dashboard. The harder problem is deciding what to assert on in the first place. The five patterns below cover the first five rows. The last row is where things get interesting.
The Fabrication Shortcut, and why it stopped working
I call the broken pattern The Fabrication Shortcut.
It looks efficient. You want to test what happens on turn three. Spinning up real multi-turn fixtures means tracking session state, threading IDs through your harness, paying for two extra agent runs per case. So you compress: write a single-turn prompt that describes the prior turns inline.
Earlier in this conversation you said the dinner is
vegan and peanut-free. Now plan the menu.
Cheap, looks like a trajectory test, ships in an afternoon.
One trajectory eval in blueclaw expected the agent to preserve a dietary restriction introduced “earlier in the conversation.” Newer honesty-trained models refused the premise entirely because no such message existed in the actual chat history. The spec came back at 0% pass on the trajectory cases.
The problem is structural. Honesty-trained models check their actual message history, find no matching turn, and refuse the premise. Your assertion regex then matches the refusal phrasing and counts it as a failed trajectory. The passing rate is now measuring honesty training, not trajectory preservation.
This gets worse, not better, as model honesty improves. Every agent eval written this way is on a decay curve.
The two honest responses are:
- Rewrite the test as instruction-framed single-turn. Fold the constraints into the current prompt. Accept that you are now testing complex single-turn handling, which is a weaker target than trajectory preservation. Document the downgrade in the spec so future-you knows what is and is not covered.
- Build real multi-turn fixtures. A
turns: list[str]field onTestCase, replayed through a single agent session. Higher engineering cost, the only honest test of trajectory behavior.
What you must not do: keep fabricating prior turns and report passing rates. The passing rate is measuring honesty training, not your agent’s behavior.
Pattern 1: regex-as-judge for refusals
Catches: the agent declining a request without trying available tools, in any rewording.
The failure mode is a class, not a string. “I cannot access real-time data” and “I don’t have the ability to look that up in real time” and “that’s outside what I can do” are the same failure. A literal substring check catches one. A regex with negation framing catches the class.
- goal: |
Plan groceries for tonight's dinner. I need
itemized current prices for: tofu, soba
noodles, edamame, miso paste.
expected_tools: [web_search]
forbidden_output_regex: >-
(?i)(cannot access|don't have access|
unable to (access|look up|retrieve))
.*(real.?time|current|live)
max_steps: 6
runs: 5
threshold: 0.6
Two assertions in one test. expected_tools requires the agent actually invoke the tool. forbidden_output_regex catches the refusal phrasing if the agent declines without trying. Either alone is a partial check; together they close the loop.
A note on runs and threshold: blueclaw scores multi-run cases with a Wilson confidence interval, not a raw pass fraction. A case passes outright only when the interval’s lower bound clears the threshold, fails when the upper bound stays below it, and lands in an inconclusive band otherwise. At five runs, that band is wide, so CI gates on hard fails and treats inconclusive as non-blocking.
When to reach for this pattern: any time the failure is a finite, enumerable set of phrasings. Refusal phrasings. Hedge phrasings (“I’m not sure but…”). Disclaimer boilerplate. The shape is stable across models; the regex is portable.
When to skip it: open-ended quality. “Is this answer helpful” is not a regex problem. Reach for LLM-as-judge there.
Pattern 2: structural pair (positive + negative regex)
Catches: wrong shape, missing required sections, broken ordering.
A single positive regex says “the answer must have X”. A single negative regex says “the answer must not have Y”. The pair asserts both shape and anti-shape at once.
- goal: |
Plan a 3-course vegan dinner for 6 adults
plus a 4-year-old, peanut-free, 7pm start.
output_regex: >-
(?is).*(starter|appetizer|first course)
.*main.*dessert
forbidden_output_regex: >-
(?i)(main with side|
skipping the (starter|appetizer)|
just (a |the )?main)
runs: 5
threshold: 0.6
The positive regex asserts three named courses in order. The negative regex catches the hedge: “given the constraints, let me just give you a main with sides”, which is a structurally valid answer that fails the goal.
When to reach for this pattern: any time the spec has both “must contain” and “must not contain” semantics. Most structured-output tests qualify.
Limit: the pair only enforces shape, not substance. “Starter / main / dessert” headings with terrible recipes still pass.
Pattern 3: honesty-pass (the counterintuitive one)
Catches: silent omission, the failure mode where the agent answers part of a multi-part request and quietly drops the rest.
The trick: the pass condition is the agent admitting a limit, not solving the problem.
- goal: |
Give me itemized FairPrice prices for these
6 ingredients: tofu, soba, edamame, miso,
scallions, sesame oil. If you can't fetch
live prices, say which part of the request
you can't do and why.
output_regex: >-
(?i)(cannot|unable|don't have)
.*(price|fairprice|live)
runs: 5
threshold: 0.6
Without this test, an agent that confidently lists five of the six ingredients with no acknowledgment that the price request was dropped will pass every other assertion in the suite. The user notices the omission; the eval does not.
When to reach for this pattern: any multi-part request where the agent might be able to do some parts and not others. The honest “I can’t fetch live prices” is the pass; silent omission is the fail.
Counterintuitive part: the prompt itself instructs the agent to admit limits. This is fair. You are testing whether the agent follows that instruction, not whether it can read your mind.
Pattern 4: uses-the-corrected-fact
Catches: the agent ignoring a user correction in the same turn.
The ideal test is “does the agent acknowledge the correction in the first sentence of the next turn?”. That requires a real multi-turn fixture. Without one, you can still measure the weaker, useful question: “given a correction in the current turn, does the response use the corrected fact?”.
- goal: |
Halloumi is pan-seared 2-3 minutes per side,
not oven-roasted. Suggest a vegetarian main
using it.
output_regex: >-
(?i)(pan.?sear|saut[eé]|griddle|skillet)
forbidden_output_regex: >-
(?i)(oven.?roast|400.?°|oven.bake)
runs: 5
threshold: 0.8
Threshold bumped to 0.8 because the assertion is narrower. A model that complies on this single-turn test may still fail trajectory tests where the correction was three turns ago. That gap is real and explicit in the spec.
When to reach for this pattern: any factual-correction case where you want a regression test today and cannot afford multi-turn fixtures yet.
Limit: the forbidden regex can fire on a compliant response that echoes the correction (“pan-seared, not oven-roasted”). Scope forbidden terms to phrasings that mostly appear in wrong answers: oven.?roast survives a legitimate roasted-vegetable side dish, bare roast does not. When a failing run looks wrong, check the captured trajectory for an echo before blaming the model.
The honesty trade: document in the spec header what the test does and does not cover. Future-you, reading the spec six months later, needs to know which behaviors are enforced by automation and which by trust.
Pattern 5: shape constraint (length cap + format guard)
Catches: cosmetic compensation, where the agent reaches for headers and bullet lists to mask thin substance.
- goal: "What is 2+2?"
output_regex: "(?s)^.{1,80}\\s*$"
forbidden_output_contains: "##"
runs: 3
threshold: 0.66
A length cap (80 characters) and a literal substring check for markdown headers. This is the weakest test in the suite. A determined agent could pass both with Four (4). and still over-format in worse ways. It ships anyway because a flawed test for a real failure mode beats no test.
When to reach for it: short-answer questions where the failure mode is over-formatting. Quick math, single-fact lookups, yes/no with brief justification.
Limit: this is where regex hits its ceiling soonest. If you can afford LLM-as-judge for one assertion class, make it this one.
The eval feedback loop
The five patterns are a starting kit. The workflow that produced them is portable:
- Ship the agent to a small, real workload. Eval suites written in a vacuum measure imagined failures (for the kinds of failures that surface first, see why AI agents fail in production). Eval suites written after a week of real use measure the failures you actually have. A friend group, an internal Slack channel, a single-team beta. Anything with users who notice when the agent is wrong.
- Collect the first 5 to 10 failures. Not the worst failures; the first ones. The earliest failures are the most representative because users have not yet learned to work around the agent.
- Cluster the failures into classes. “Refused without trying” is a class. “Dropped half my request” is a class. Most production failure sets cluster into 4 to 6 classes; more than 8 means you are still over-specific.
- Pick the assertion type per class, using the decision matrix above.
- Write one test per class. Set the threshold at what passes today. A threshold you cannot pass yet is a roadmap, not a failed test. Move thresholds up only after you fix the underlying behavior.
- Loop. Each new production failure either fits an existing class (add a run, tighten the threshold) or starts a new class (add a test).
Each assertion that fires repeatedly becomes a rule in the system prompt. Each rule gets verified by the same assertion that caught the original failure. For the runtime patterns that complement these prompt-level rules, see AI agent error handling patterns.
| Failure class | Rule added to system prompt |
|---|---|
| Refusal without trying tools | tool-knowledge |
| Silent omission | partial-refusal |
| Ignored correction | correction-acknowledgment |
| Cosmetic compensation | cosmetic-compensation |
This is what makes an eval load-bearing. The assertion is not for a quarterly report. The assertion is the spec that the system prompt must satisfy on every release.
Capture every run, or you can’t triage
Without captured trajectories, you cannot evolve the eval suite.
Every blueclaw eval run persists per case and per run:
~/blueclaw/test-runs/<ts>/
invocation.json # config + capture_failures
case-<N>/run-<N>/
response.txt # final response text
messages.json # full agent trajectory
Three properties any harness should have:
- Capture decoupled from workspace cleanup. Artifacts persist even when the workspace is deleted. Otherwise the artifacts you most need (failures) are the ones most likely to disappear.
- Capture is best-effort, never a test failure. Write failures get logged but never fail the eval.
- Failure records breadcrumb the artifact path. The path lands in CI logs without extra wiring.
When a threshold-0.6 test passes 3 of 5 runs and fails 2 of 5, the failing runs are where the next eval iteration is born. Without messages.json you cannot tell if the failure was a refused tool call, a hallucinated tool name, a malformed argument, or correct tool use with wrong synthesis. You stare at the threshold and guess. For the CLI workflow that turns these captured artifacts into actionable inspection, see debugging agents like code.
When regex-as-judge beats LLM-as-judge
The 2026 agent eval landscape defaults to LLM-as-judge. Braintrust, FutureAGI, LangSmith, DeepEval, promptfoo all assume you will spin up a judge model per case. For open-ended quality dimensions like helpfulness and tone, you should.
For refusal-class, structural, and length-cap assertions, you should not.
The dividing line is simple: can the answer fail in finite, enumerable ways? If yes, regex often beats LLM-as-judge.
Refusal phrasings are finite. Required structure is finite. Length is a single number. A regex with negation framing closes the rewording loophole at zero per-case cost. It is deterministic. It is debuggable. When it fires a false positive, you can read the regex and see why.
Reach for LLM-as-judge for tone, helpfulness, and substantive correctness. Use regex for everything else in your LLM evals.
What this doesn’t cover
The five patterns do not cover trajectory behavior. Anything that only emerges across multiple real turns (constraint carry-forward, persona drift, context-budget failures) needs real multi-turn fixtures. Until your harness supports them, document the gap in the spec header. A known gap is a roadmap item; a hidden gap is a bug.
The other honest limit is cost. A full pre-release eval run against a frontier model costs single-digit dollars and tens of minutes. Fine for manual runs and release gates. Not fine for per-PR CI. The next eval tier is either smaller models for fast feedback or a sampled subset for behavioral contract CI, with the full suite gated to a manual trigger.
Close
Five patterns, five failure classes, one decision matrix. Most agent evals break quietly when the Fabrication Shortcut stops working. Rewrite those tests as instruction-framed single-turn, accept the weaker target, and back-fill the trajectory holes with real multi-turn fixtures when you can afford them.
The cheap shortcut was never the cheap shortcut. It was just deferred work.
