Measurements¶
Every number here is reproducible from this repository with no credentials.
The models are stubs, not providers
All of these run against deterministic stubs. The stubs do real work — the extraction stub parses contracts with regexes, the selection stub scores queries against tool descriptions, the harness stub answers only from what is in its context — but they know the shape of the synthetic data, so their accuracy is optimistic.
That is deliberate. The purpose of these tables is to show that the
measurement harness works and that the comparisons are reproducible. For
numbers about a model, run the same commands with
--model anthropic:claude-opus-5.
What the data is. The golden sets are synthetic, generated by
scripts/generate_golden_set.py (36 contracts, 10 deliberate edge cases) and
evals/tool_cases/cases.py (22 cases, 3 of them expecting no tool call).
Replace them with your own data for a real evaluation.
1. Schema and prompt versions¶
uv run prompt-lab eval --prompt extract_contract@v2 --schema v2 \
--pricing config/pricing.toml --pricing-model claude-opus-5
36 documents, max_attempts=3. The cost column is a projection: the token
profile is the stub's, the prices are claude-opus-5's.
| prompt_ref | schema | First-try pass | Mean attempts | Grounding | Input tok p50 | Cost/doc |
|---|---|---|---|---|---|---|
extract_contract@v1+099030d1 |
v1 | 83.3% | 1.17 | 0.0% | 729 | $0.0080 |
extract_contract@v2+19009d12 |
v2 | 72.2% | 1.28 | 100.0% | 1,982 | $0.0089 |
v2 passes on the first try less often, and that is not a regression. v2 enforces cross-field rules (end after start, high risk requires a rationale, an amount requires a currency) that live outside anything the provider validates. v1 applies none of them, so it "passes" a weaker exam.
The price is 0.11 more attempts per document and 11% more cost. The return is 0% → 100% grounding.
2. Field accuracy¶
v1* is the same run with --no-grounding-drop, which separates "the schema
could not express this" from "grounding could not verify it".
| Field | v1 | v1* (no grounding drop) | v2 | Why |
|---|---|---|---|---|
jurisdiction |
0.0% | 0.0% | 100.0% | absent from the v1 schema |
amount_minor |
2.8% | 2.8% | 97.2% | v1 has amount as a float; minor units are never produced |
start_date |
0.0% | 88.9% | 100.0% | v1's 0% is entirely grounding |
end_date |
2.8% | 88.9% | 100.0% | same |
termination_notice_days |
0.0% | 97.2% | 97.2% | same; the remaining 2.8% is the "two months" conversion |
auto_renewal |
97.2% | 97.2% | 100.0% | absent from the v1 schema |
risk_rationale |
97.2% | 97.2% | 100.0% | v2 requires it at high risk |
parties |
100.0% | 100.0% | 100.0% | unchanged |
currency |
100.0% | 100.0% | 100.0% | unchanged |
risk_level |
100.0% | 100.0% | 100.0% | free string in v1, Literal in v2 |
Three findings:
- A schema without a citation field cannot be grounded. v1's
start_datedrops from 88.9% to 0% purely because there is nowhere to put a quote. Read without the split, this looks like "v1 cannot extract dates", which is false and would send you to fix the wrong thing. - A field the schema does not have cannot be extracted.
jurisdictionandauto_renewalare in the documents and the model has nowhere to put them. Schema design is an output-quality decision. - The weakest field really is the weakest. v2's two remaining errors are the two edge cases they should be:
fields that failed on edge cases:
notice_in_months -> termination_notice_days
tax_inclusive -> amount_minor
The CI gate hangs off exactly that number:
3. Injection resistance¶
12 escape patterns, 5 runs each, structural defense only.
| Pattern | Runs | Held |
|---|---|---|
plain_closing, spaced_closing, uppercase, mixed_case |
20 | 100% |
line_break, nested_opening, trailing_space, double_closing |
20 | 100% |
embedded_instruction, role_switch, prompt_leak, format_escape |
20 | 100% |
| total | 60 | 100% |
This measures whether the wrapper can be escaped and whether document content can reach the system block. It is deterministic, so 100% is the expectation and anything less is a regression.
The behavioral half — does a real model follow an injected instruction — is a separate live-marked test and is probabilistic. Prompt-level defense is the first line, not the only one; real authorization is a tool-layer concern.
4. Tool descriptions¶
Three variants of one registry. Identical schemas; only description text differs.
| Variant | Selection accuracy | Forbidden-call rate | No-tool accuracy |
|---|---|---|---|
| full | 90.9% | 0.0% | 100.0% |
| boundary sentences removed | 90.9% | 0.0% | 100.0% |
| first sentence only | 90.9% | 4.5% | 100.0% |
The expected result did not appear, and the one that did is more useful. Description quality moved no selection accuracy at all with this selector. It moved the safety number: cutting descriptions to a single sentence left the agent just as likely to pick the right tool and measurably more likely to reach for the destructive one.
Which says something about dashboards. Selection accuracy is a quality signal; forbidden-call rate is a security signal, and it degraded first.
5. Loop detection¶
20 scripted tasks, 5 of them stuck runs. The configurations differ only in whether repetition stops a run.
| Configuration | Completion | p50 steps | p95 steps | Loop rate |
|---|---|---|---|---|
| detection on | 75.0% | 2.0 | 3.0 | 25.0% |
| detection off | 100.0% | 2.0 | 7.0 | 0.0% |
In these scripts the stuck runs recover on their own after six repeats — an optimistic assumption, and the one that makes the trade-off visible. Detection saves 4 steps at p95 and costs 25 points of completion.
stop_at is the dial. Where to set it depends on whether your steps are
expensive or your completions are.
6. Harness configurations¶
Read 30 documents, one containing a fact the answer needs. The model answers only from what is in its context.
| Configuration | Steps | Total tokens | p95 fill | Compactions | Answer survived |
|---|---|---|---|---|---|
| naive trim | 30 | 101,114 | 0.979 | 0 | no |
| compaction | 30 | 69,539 | 0.692 | 2 | no |
| compaction + sub-agent | 30 | 12,018 | 0.134 | 0 | yes |
Compaction cut tokens 31% and the p95 fill ratio from 0.98 to 0.69. A run one long result away from bursting became a run with headroom. It did not save the answer.
Honest caveat on the middle row: the needle is sitting in a spill file and is
reachable with read_spill; the simulated agent never follows the reference.
So the row says "a spill summary plus compaction is not enough if the agent
does not follow its references" — a real failure mode, but a different claim
from "compaction does not work".
The third row carries the lesson. Delegating the reading cut tokens 88% and kept the answer, because the main context never held the documents. The cheapest context management is not managing context you never took on.
7. Compaction needle tests¶
Three planted facts, buried under noise, both prompt versions.
| Prompt | Order id ORD-88421 |
Amount 125,000 |
Date 2025-03-14 |
|---|---|---|---|
compaction@v1 ("keep it short") |
lost | lost | lost |
compaction@v2 (names what must survive) |
kept | kept | kept |
Run with uv run pytest tests/test_compaction.py -v.
Model comparison¶
This table is filled from your own runs; the harness produces the JSON.
uv run model-probe --prompt evals/probe/sample-prompt.txt \
--models anthropic:claude-opus-5 anthropic:claude-haiku-4-5 \
-n 5 --json evals/probe/result.json
| model | in~ | in | drift% | out | USD/1k | TTFT p50 | TTFT p95 | total p50 | total p95 | distinct | modal |
|---|---|---|---|---|---|---|---|---|---|---|---|
| (your run) |
in~ is the local tiktoken estimate, in is what the provider reported, and
drift% is the gap. tiktoken is the OpenAI vocabulary; on Anthropic and
Gemini that drift reaches 10-20%, which is why the context budget carries a
safety margin instead of filling to the limit.
Without credentials, the budget gate is still observable: