Proving Ground — demo loop
A working build of the Phase 2 "demo loop" from Proving Ground: Design Thesis (Oct 4, 2026): agents explore a sealed replica of a customer's finance-ops system, prove which exception workflows they can run, and the verifier that admits each task becomes the meter that bills for it.
1. Twin ─────▶ 2. Explore ─────▶ 3. Propose
│ exceptions become new tasks
4. Admit ─────▶ 5. Prove & price ──▶ 6. Operate & bill ◀──────────┘
Output: out/proof_packet.md — one proof packet per workflow (monthly volume, current unit cost,
pass rate on held-out past cases with 95% CI, known failure modes, per-outcome price, live metered
results), plus the admission report showing which verifiers survived the breaker.
The three design rules, as code
| Rule (thesis) | Where |
|---|---|
| Tasks come from the customer's records. A candidate is admitted only with evidence of real volume. | agents.propose must cite volume + unit cost; loop.explore checks every number against /stats/exceptions. |
| No agent grades its own work. Proposer, solver, verifier, breaker are separate, from different model families. | Proposer google/gemini-2.5-flash, solver openai/gpt-4.1-mini (tier openai/gpt-4.1 compared), breaker deepseek/deepseek-chat-v3-0324, verifier = code (policy.py). |
| Proof is replay, not a judge's opinion. | loop.prove reopens 20 held-out historical cases per workflow in sandbox copies; policy.verify recomputes the required end state from the system of record and checks the audit log for out-of-scope writes. No LLM judges anything. |
The breaker is record-blind: scripted shortcut strategies plus an LLM with write-only tools. A task is
admitted only if (a) the verifier agrees with ≥90% of how humans actually resolved past cases and (b) no
blind strategy reaches 70% on it. vendor_bank_change is deliberately a fixed-rule control ("hold + flag"):
the breaker passes it blind, so the gate screens it out of outcome pricing — that is the gate working.
What the run produced (results/, Oct 5 2026, $3.57 model spend)
| Workflow | Admitted | Priced tier | Replay pass (n=20, 95% CI) | Price/outcome | Live metered |
|---|---|---|---|---|---|
| quantity_mismatch | yes | gpt-4.1-mini | 100% [84–100] | $4.58 | 7/8 |
| price_mismatch | yes | gpt-4.1-mini | 95% [76–99] | $3.17 | 7/8 |
| missing_po | yes | gpt-4.1-mini | 90% [70–97] | $5.22 | 8/8 |
| possible_duplicate | yes | gpt-4.1 (mini: 60%) | 100% [84–100] | $2.02 | 8/8 |
| unmatched_payment | yes | gpt-4.1 | 60% [39–78] — below contract bar | $5.72 | 6/8 |
| vendor_bank_change | screened | — | blind "hold + flag" passes 100% | — | — |
Gate passed (5 verifiers survived the breaker). The unmatched-payment result is the useful kind of
failure: both tiers struggle with pair-sum reconciliation, and the packet says so instead of pricing it.
Full packet: results/proof_packet.md.
Hosted demo twin
https://proving-ground-production.up.railway.app — synthetic customer, sealed API, one private sandbox
per token. GET /policy and GET /schema are open; everything else needs Authorization: Bearer <token>
and X-Sandbox: <your actor>. Work an exception, POST /exceptions/{id}/verify, then GET /proof.
See docs/deployment.md and docs/billing.md.
Walk in as an agent
No account, no human. GET / is the menu (JSON, or HTML in a browser; /llms.txt and
/.well-known/agent-card.json say the same thing for crawlers and A2A clients).
# 1. take a seat: a bearer token (shown once), a private sandbox, free credits
curl -s -X POST https://proving-ground-production.up.railway.app/seat \
-H 'Content-Type: application/json' -d '{"name":"my-agent"}'
# 2a. REST: send both headers on every call; reads are free, each resolve/escalate costs 1 credit
curl -s -H 'Authorization: Bearer <token>' -H 'X-Sandbox: my-agent' \
'https://proving-ground-production.up.railway.app/exceptions?status=open&kind=price_mismatch'
# 2b. MCP: point any MCP client at /mcp with the same token (streamable HTTP, no X-Sandbox needed)
# — or with no auth at all: `initialize` seats you and returns Mcp-Session-Id; that session IS a seat
# (same free allowance, same x402 top-up), which is what directory listings that forbid static tokens need
Tools carry MCP title/annotations (readOnlyHint, destructiveHint, idempotentHint), ordered so the obvious first
doors come first. Terms at /terms, privacy at /privacy (drafts, not legal advice); taking a seat is acceptance.
/openapi.json carries x-payment-info on the metered operations (x402scan / Bazaar shape) and info.termsOfService.
Each claim returns the verifier's verdict and your bill; GET /proof (or the proof tool) is the
proof packet. Out of credits → 402 Payment Required with the top-up methods the operator has
enabled (PG_TOPUP); a payment rail tops a seat up through POST /seats/{actor}/credit with the
admin key. The meter is rail-agnostic by design.
Run it
pip install -r requirements.txt # fastapi, uvicorn (everything else is stdlib)
python -m src.cli seed # deterministic synthetic AP twin: out/twin.db (12 months, ~8.7k invoices, 3,960 exceptions)
python -m src.cli selfcheck # asserts: verifier agrees with history, truth stable, reopen/replay round-trips
python -m src.cli run-all # explore → admit → prove → price → operate → packet (starts the API if needed)
python -m src.cli packet # re-render out/proof_packet.md
Model access: OpenRouter key in $OPENROUTER_API_KEY or ~/.config/agent-integrity/openrouter.key.
Models are env-configurable (PG_PROPOSER_MODEL, PG_SOLVER_MODEL, PG_BREAKER_MODEL). A full run
costs about $2–3 of model spend; every call's real cost is recorded and flows into "cost to serve".
Individual steps: serve, explore, admit, prove [--alt], price, operate, packet, cost.
The twin
twin_api.py is the sealed boundary: agents only see HTTP. X-Sandbox: <name> selects an isolated
copy (out/sandboxes/<name>.db); X-Actor is recorded on every write. Harness-only columns
(pre_state, post_state, truth, held_out) are never served.
Six exception kinds, each with mixed outcomes so no constant answer passes: price_mismatch,
quantity_mismatch, possible_duplicate, missing_po, unmatched_payment (bank reconciliation break),
vendor_bank_change. History is produced through the same actions.py the agents use, with a 3% rate of
clerk "manager overrides" so verifier-vs-history agreement is realistic rather than 100%.
ponytail: the twin is a synthetic stand-in for an open-source ERP; the API surface is the adapter
boundary, so an Odoo/ERPNext backend would implement the same endpoints.
Files
src/schema.sql tables src/policy.md the customer's AP policy (served at /policy)
src/seed.py synthetic twin src/policy.py expected outcomes + verifiers (code keyed to the system of record)
src/actions.py all state changes src/twin_api.py sealed API
src/llm.py OpenRouter + tool loop + $ tally src/agents.py proposer / solver / breaker
src/loop.py admit / prove / price / operate / packet src/cli.py entry point
src/selfcheck.py the one runnable check