procurement-code-agent

An MCP server, an HTTP API and a small agent that classify free text purchase descriptions into United States federal procurement codes.

The classification problem is the excuse. What I wanted to show is the part that usually gets skipped in agent demos: typed tool contracts, authorization enforced where a model cannot reach it, idempotency on the one call that changes anything, async that actually yields, and an eval suite honest enough to have changed my mind about the design three times while I was building it.

It runs on a laptop with no API key. pip install -e ".[dev,graph]" && pytest is the whole setup.

Quickstart

python -m venv .venv && . .venv/Scripts/activate   # or .venv/bin/activate
pip install -e ".[dev,graph]"
pytest

96 tests, under 10 seconds, no network and no model downloads.

To run the service:

uvicorn pcagent.api:create_app --factory --port 8000

First start downloads about 210MB of ONNX embedding weights and warms the indexes, which takes roughly 20 seconds. After that it is CPU only and offline.

curl -s localhost:8000/v1/classify \
  -H "Authorization: Bearer dev-reader-token" \
  -H "Content-Type: application/json" \
  -d '{"description":"JANITORIAL AND CUSTODIAL SERVICES FOR FEDERAL BUILDING"}'

{
  "classification_id": "cls_df74127ea8d3fbac",
  "taxonomy": "psc",
  "code": "S201",
  "title": "HOUSEKEEPING- CUSTODIAL JANITORIAL",
  "confidence": 0.8149,
  "alternatives": [
    { "code": "M111", "title": "OPERATION OF OFFICE BUILDINGS" },
    { "code": "M1AA", "title": "OPERATION OF OFFICE BUILDINGS" }
  ],
  "decided_by": "baseline-top1",
  "latency_ms": 19
}

classification_id is derived from the taxonomy and the normalised description, so the same text asked twice is the same classification and the record is updated rather than duplicated. decided_by says what produced the answer, so a degraded or fallback result is never mistaken for a model one.

As an MCP server, for Claude Desktop or any other client:

{
  "mcpServers": {
    "procurement-codes": {
      "command": "pcagent-mcp",
      "env": { "PCAGENT_TOKEN": "dev-reviewer-token" }
    }
  }
}

The five decisions worth arguing about

1. The tool layer is the security boundary, not the prompt

Every tool has the same shape:

async def submit_review(ctx: ToolContext, params: SubmitReviewParams) -> Review:
    ctx.require(Scope.REVIEWS_WRITE)
    ...

params is what the caller, model included, gets to specify. ctx is what only the transport can supply, because it is built after authentication.

The consequence is the bit I care about. submit_review has no reviewer field. The reviewer is ctx.principal.subject, read from the token. A model cannot record a decision in someone else's name because there is nowhere to put the name. Prompt injection against this agent can at worst produce a badly chosen tool call. It cannot produce an unauthorised one.

The scope check is the first line of the function body rather than a decorator. A decorator is invisible at the call site and silently stops applying the day someone calls the underlying function directly. Instead the decorator only records the scope, and a test walks the registry, calls every tool with a principal holding every scope except the declared one, and fails if the call succeeds. A tool that forgets its check passes every other test in the suite and fails that one.

2. Two transports, one core

api.py and mcp_server.py are both thin. Neither contains a business rule. That is what lets the same capability be offered over HTTP and MCP without the two drifting apart.

They differ in exactly one place, on purpose. Over HTTP the idempotency key arrives in the Idempotency-Key header, where an HTTP client expects to put it. Over MCP it is a tool argument, because MCP has no headers. Same guarantee, each transport spelling it the way its callers already speak.

Identity follows the same pattern. HTTP resolves a bearer token per request. An MCP server over stdio is a process started on behalf of one user, so the principal is resolved once at startup from the environment. Both get it from the thing that authenticated the session, and neither gets it from the model.

3. Idempotency belongs in the transaction, not above it

submit_review is the only tool that changes anything, and it is exactly the kind of call an agent retries. A timeout on the caller's side says nothing about whether the write landed.

The protocol is the one payment APIs settled on, and it lives in store/db.py:

• begin_idempotent reserves the key inside a write transaction. First caller gets Proceed. Same key and same body gets Replay with the original response. Same key and a different body gets a 409, because one of the two is a bug and guessing which would be worse than failing.
• complete_idempotent writes the review and stores the replayable response in one transaction. A crash cannot leave a review recorded with no way to replay it, nor a replayable response for a review that was never recorded.
• release_idempotent hands the key back when the work failed, so an honest retry is not locked out.

Three details that only show up once it is running:

Keys are scoped per caller. Two callers both using retry-1 are making two different requests. A global key space would hand the second one the first one's answer.

Validation happens before the reservation. Otherwise a caller who sent a bad body once could never retry with a corrected one under the same key, which is precisely what a client library with automatic retries does.

A shared connection needs a lock as well as BEGIN IMMEDIATE. I had this wrong. A transaction is a property of the connection, not of the coroutine, so eight coroutines racing on the same key all issued BEGIN IMMEDIATE on one connection and got cannot start a transaction within a transaction. The asyncio lock serialises coroutines sharing the connection. BEGIN IMMEDIATE serialises this process against any other process on the file. Both are needed, and test_only_one_of_many_racing_callers_proceeds is the test that found it.

4. Async that actually yields

The ONNX embedding and reranking calls are synchronous and CPU bound. Wrapping them in async def would not make them yield, so they go through asyncio.to_thread, including during startup, because startup is when a readiness probe is waiting.

BM25 runs inline. It is tens of microseconds over a few thousand short documents, and a thread hop would cost more than the work. The rule is do not block the loop, not never compute on the loop.

This is tested rather than asserted. test_async_discipline.py puts a time.sleep inside a stub index and counts heartbeats from a coroutine ticking alongside it. A loop that stayed responsive gets most of its ticks, a blocked one gets almost none. The assertion is on the heartbeat, so the test still means something after a refactor.

The batch agent in agent/batch.py is hand written for the same reason: the interesting decisions are all about restraint. A semaphore caps in flight work, asyncio.wait_for puts a deadline on every item, each failure is caught per item so one bad row cannot cancel nine thousand others, and results come back in input order regardless of completion order. There is a LangGraph version of the same flow in agent/graph.py, which exists to show the tools do not care what is orchestrating them.

5. Retrieval is hybrid, and the reranker did not survive the measurement

BM25 and a dense index both run over the whole corpus and their rankings are fused with reciprocal rank fusion.

Fusion is by rank, not by score, because there is no honest way to put an unbounded sum of IDF terms on the same scale as a cosine similarity that lives between 0.6 and 0.7. Normalising each to 0 and 1 per query just invents a scale that changes with every query. RRF throws the scores away and keeps the ordering, which is the part both rankers agree is meaningful.

There is also a cross encoder reranking stage, fully built, and it is off. That was not the plan, and the sequence is the point:

1. Built retrieve, fuse, rerank, the standard shape.
2. Measured three rerankers. ms-marco-MiniLM-L-6 did nothing at all, spreading four candidates across a single logit on a query with one obviously correct answer. jina-reranker-v1-turbo was worth 28% relative recall@5. Switched.
3. Measured two embedders and upgraded bge-small to bge-base.
4. Re-measured the reranker against the better embedder, and it now lowered MRR on all three corpora. PSC 0.123 to 0.068. NAICS 0.171 to 0.157. arXiv 0.398 to 0.251.
5. Turned it off, for 170ms a query back and better results.

A cross encoder earns its place by fixing a bi-encoder's mistakes. Once the bi-encoder stopped making those mistakes, the reranker was left reordering candidates it understood worse than the embedder did. The stage stays in the codebase and in the ablation rather than being deleted, because this is a property of these corpora and these models, and swapping either could flip the answer back. The ablation is what would tell me.

Both ONNX stages are optional at runtime. If the weights cannot be loaded, retrieval falls back to BM25 alone and says so on the result rather than failing. Someone who clones this on a plane gets answers.

What the evals say

evals/run_eval.py measures four pipelines over the same queries, then end to end agreement, calibration, and what a review threshold actually buys.

The headline

Test split, served pipeline (BM25 and dense, fused, no reranking):

Random guessing would be 0.0065, 0.001 and 0.0003 respectively, so all three are far better than nothing and only one of them is useful. Full tables, including the four way ablation and the routing thresholds, are in evals/report-{corpus}-{split}.md.

The finding that changed the project

I built this against federal procurement data first. It measures badly, and not because the pipeline is weak.

The Description field on a federal award is an internal memo, not a product description. The eval surfaces this the moment you look at the failures:

• "EXPRESS REPORT: QUARTER 4" is labelled G007, government health insurance programs
• "AWARD TASK ORDER IGF::OT::IGF" is labelled R706, logistics support
• "VA MICROSOFT ENTERPRISE LICENSE AGREEMENT" is labelled DA01, application development. The pipeline returned W070, lease or rental of IT software, which fits the words better than the label does

A single bad result cannot tell you whether the pipeline is weak or the task is hard, so I added a second corpus as a control. arXiv abstracts into the arXiv subject taxonomy is the same engineering against text that genuinely determines its label, and the identical pipeline gets recall@20 of 0.85 on 155 classes.

That is the useful conclusion, and it is worth more than a flattering number: federal award descriptions do not contain enough signal to predict procurement codes, and the right product here is shortlist plus human review, not automation. Everything in the routing table is built around that.

Things I tried that did not work

Reported because the negative results were more informative than the positive one, and because a list of only the things that worked is not a record of what happened.

Things that did work

The last two rows are worth reading together. I spent real effort making the reranking stage better, and the thing that improved it most was deleting it from the serving path. Both facts are in the repo because the second one only became true after the embedder upgrade, and a record that showed only the final state would make the decision look obvious when it was not.

Protocol

Configuration is chosen on the dev split and reported on test. The split is assigned by hashing the record id rather than by shuffling, so it survives regenerating the data files and a parameter tuned on dev cannot leak into a test number.

Two things I deliberately did not tune. Dense alone beats fusion on NAICS by 0.011 MRR, and fusion beats dense alone on PSC and arXiv by similar margins. On a dev split of 150 rows those differences are noise, so hybrid stays everywhere. The reranking differences are three to six times larger and point the same way on all three corpora, so those I acted on. Knowing which gaps are real at a given sample size is most of what an eval suite is for.

Confidence is weaker than it looks

The retrieval only baseline reports a confidence derived from the margin between the top two candidates. My first version squashed the raw margin through a sigmoid, which works for cross encoder logits and is meaningless for RRF scores where the entire shortlist spans about 0.0003. The calibration table showed all 257 predictions sitting in one bucket at exactly 0.500, which is what a broken confidence signal looks like from the outside. Dividing by the spread across the shortlist makes it scale free and dropped expected calibration error from 0.263 to 0.149.

It is still not predictive. Accuracy is roughly flat across the confidence buckets, which means retrieval margin is close to useless as a signal about correctness. That is the honest argument for the LLM adjudication step rather than a reason to hide the baseline.

The model step

Off by default. PCAGENT_DECIDER=claude turns it on and it needs ANTHROPIC_API_KEY.

The model picks from the shortlist, it does not generate a code. A model asked for "the PSC for this" will happily return a well formed code that does not exist, and constraining it to a list turns a hallucination into a wrong but checkable answer. The tool then verifies the choice against the list anyway, because a prompt is a request and not a guarantee. Quoted evidence is checked against the input text and any span that is not really there is dropped, since a paraphrase presented as a quote is worse than no quote.

Everything outbound is bounded: a per call timeout, one retry and only for an off list answer, and a documented fallback to the retrieval baseline that labels itself in decided_by so a degraded answer is never mistaken for a model answer.

The baseline exists so the model has something to beat. Reporting them side by side is the only way to know whether the model is adding judgement or just adding latency.

Layout

src/pcagent/
  domain.py          pydantic contracts, shared by everything
  auth.py            Principal, Scope, TokenRegistry
  errors.py          the error vocabulary, with stable codes
  config.py          settings from the environment
  bootstrap.py       builds the service graph once
  taxonomy/loader.py loads the committed corpora
  retrieval/         lexical (hand written BM25), dense, rerank, hybrid
  llm/               Decider protocol, baseline, Claude, the prompt
  store/             SQLite and the idempotency protocol
  tools/             the typed core: validation and authorization live here
  api.py             FastAPI, thin
  mcp_server.py      FastMCP, thin
  agent/             hand written batch loop, plus a LangGraph adapter
data/                committed corpora and labelled eval sets, with provenance
evals/               runner, metrics, generated reports
tests/               96 tests, no network, no model downloads
scripts/             one off data builders, stdlib only

Read in this order if you have ten minutes: tools/review.py, store/db.py, retrieval/hybrid.py, tests/test_auth_contract.py.

Tests

pytest                      # 96 tests, under 10 seconds
ruff check . && mypy        # lint and types
python evals/run_eval.py --taxonomy arxiv --split test

The suite stubs the ONNX models. That is not a shortcut, it is a boundary: these tests are about the tool contract, the authorization rules and the idempotency protocol, and none of those get more true with a real embedding model behind them. Model quality is measured in evals/, which is where it belongs. The stubs are also how the degradation paths get tested at all.

What I would do next

• The labels deserve a second opinion. Agreement with a noisy label is a floor, not an estimate. A few hundred hand adjudicated rows would turn that floor into a real number and is the single highest value thing left.
• Hierarchical scoring. Getting PSC group 70 right and the fourth character wrong is not the same failure as returning a weapons code, and the current metric cannot tell them apart.
• Fine tune the bi-encoder on the taxonomy. The off the shelf models have never seen a PSC title. A few thousand synthetic pairs would likely beat everything tried here.
• Postgres and pgvector if the corpus ever outgrows an in memory index. Today it is 3,400 rows and reaching for a database would be weight without benefit.

What this is not

Not a framework. Not production infrastructure. The token registry is a stand in for an identity provider, the store is SQLite, and there is no rate limiting, tracing or multi tenancy. Those are deliberate omissions in something meant to be read in ten minutes, and each one is a line in config.py or store/db.py rather than an assumption spread through the codebase.

Licence

MIT. The data is United States government public domain and arXiv public metadata. See data/README.md.