EvoMem-MCP

Memory for AI agents that knows when you changed your mind.

English · Español

You tell your coding agent: "we're using Postgres for the ledger." Two weeks later: "we moved the ledger to SQLite, Postgres was overkill."

A month after that you ask "what database does the ledger use?" Your agent's memory holds both notes, and the old one matches the question better. So the agent recommends Postgres again.

This is the default failure mode of agent memory: a pile of facts with no idea which ones are still true. EvoMem-MCP is a local MCP memory server that stores decisions as a history. Every new decision is linked to the one it replaces, the old one is kept but marked superseded, and recall always resolves to the version currently in force.

recall("ledger postgres")          # the query matches the OLD note best...
→ change: "We moved the ledger from Postgres to SQLite."     status: active
  └─ supersedes: "We picked Postgres for the ledger."        status: superseded

The result

We built a benchmark to test one hypothesis: does a memory that links decisions to the ones they replace stop an agent from acting on obsolete decisions?

The metric is ODR (Obsolete Decision Rate): the share of questions where the agent acts on a decision that was already replaced. We used 120 frozen scenarios, with 2 to 6 versions of a decision per topic plus distractors. The judge is a local 27B LLM (Bonsai 2 27B, temperature 0). It reads what memory returns and names the option it would act on.

Paired per scenario against flat with dates, EvoMem wins 24 scenarios and loses 0 (exact McNemar, p ≈ 1e-7).

The surprising part: dates are not enough

The memory was flat with dates, and the current version was right there in the context with its date in 111 of 120 scenarios. The 27B model still acted on an obsolete decision in 15% of them. Once the old note was explicitly marked status: superseded, that error went to 0.

So "just show timestamps and let the model figure it out" doesn't work reliably. An explicit "this was replaced" marker does.

How it works

flowchart LR
    A[Agent] -- "remember(text, supersedes?)" --> S[EvoMem server]
    S --> C["Cronista<br/>extracts trigger · change · consequences"]
    S --> L["Linker (optional)<br/>which active decision does this replace?"]
    C --> DB[("SQLite<br/>nodes + raw text + FTS5/BM25")]
    L --> DB
    A -- "recall(query)" --> S
    DB -- "every hit resolved to the<br/>head of its chain" --> A

• Every note becomes a MemoryNode. Its narrative fields are trigger (why), change (what was decided), alternatives_tried, consequences and summary. The raw text is always kept and indexed, so recall never depends on the extraction being perfect.
• supersede never deletes. The old node is kept with status: superseded and a link to its replacement. recall(include_history=True) returns the whole chain, newest to oldest, so the agent can see why things changed.
• recall resolves to the chain head. A query that only matches an old version still returns the current one. This is what the benchmark measures.
• Who creates the links?
  • The agent: remember(text, supersedes=id). This is the most reliable option.
  • The server, with --auto-supersede, when the agent doesn't pass an id:
    • llm uses any OpenAI-compatible endpoint, local or remote;
    • heuristic needs a change word plus overlapping vocabulary. It has no extra cost, but it is not safe on its own (see below).
• Everything is local: one SQLite file, no cloud, no embeddings service.

Quickstart

git clone https://github.com/agunet/evomem-mcp.git && cd evomem-mcp
pip install -e .                      # or: uvx --from . evomem-mcp --db ~/.evomem/evomem.db
evomem-mcp --db ~/.evomem/evomem.db   # lite mode: no GPU, ~7 ms per remember

Claude Code:

claude mcp add evomem -- evomem-mcp --db ~/.evomem/evomem.db

Claude Desktop / any MCP client:

{ "mcpServers": { "evomem": {
    "command": "evomem-mcp",
    "args": ["--db", "/home/you/.evomem/evomem.db"] } } }

Automatic supersession with a local model (here llama.cpp; any OpenAI-compatible API works):

EVOMEM_LINKER_API_KEY=... evomem-mcp --db ~/.evomem/evomem.db \
    --auto-supersede llm --linker-model <model> --linker-base-url http://localhost:8080/v1

Tools

Reproduce it

The scenarios are generated from a fixed seed, so you get exactly the same 120. The deterministic judge runs in about a minute on CPU:

python make_odr_scenarios.py --out data/odr_scenarios.jsonl --n 120
python benchmark_decisiones.py --scenarios data/odr_scenarios.jsonl --auto heuristic

The LLM-judge numbers above need a judge model behind an OpenAI-compatible endpoint:

python benchmark_decisiones.py --scenarios data/odr_scenarios.jsonl --judge llm \
    --judge-model <model> --judge-base-url http://localhost:8080/v1 --auto heuristic,llm

There is also a second benchmark, VDR, which measures the opposite risk: does automatic linking hide decisions that are still valid?

python make_odr_scenarios.py --vdr --out data/odr_vdr.jsonl --n 80 --seed 11

The cost of automatic linking: VDR

A linker that is too eager is dangerous. Suppose you note "we switched the staging ledger to SQLite". If the linker treats that as replacing the production ledger decision, the production decision disappears from recall, even though it is still valid.

The VDR set has 80 scenarios where each component has a decision in another scope (staging, analytics, EU region…) that stays valid the whole time. Half of them use change words on purpose to fool the linker.

Verdict: in these scenarios the LLM linker works without a human in the loop, at the cost of about 1.5 s per remember with a local 27B. The heuristic linker should not run unsupervised.

What this does not prove (yet)

We'd rather you hear the limits from us:

• The scenarios are synthetic, with one judge model and one seed. Real agent logs are messier. Evaluating on real agent history is on the roadmap.
• The "story" part is still untested. With history_depth=0 (only the current version) ODR is already 0, so the benefit measured here comes from filtering what is obsolete. It does not yet show that seeing the why behind each change helps. That needs scenarios where the reason a decision was abandoned matters ("should we try Kafka?" when Kafka was already dropped for a reason that still holds).
• Scope confusion is a separate problem. Even with perfect links, the judge applies a production decision to analytics in 21–30% of VDR queries. An explicit scope field is planned.
• The heuristic linker misses paraphrases. It relies on overlapping words. Pass supersedes yourself, or use the LLM linker.

Side quest: a tiny model as the note-taker

The "cronista" (note-taker) that turns text into fields comes in three versions:

What we learned, including the negative results:

• Fine-tuning for generic summarization doesn't help. On XSum the LoRA does not beat the base model (ROUGE-1 34.3 vs 34.9). For general summaries the model is already at its ceiling, so fine-tuning is only worth it for structured extraction.
• Structured extraction works. The trained LoRA reaches ROUGE 88–93 on trigger / change / consequences. On a validation set with different wording that it never saw it scores almost the same, so it generalizes.
• The generated summary invents things. It gets the service and the numbers right only 26–37% of the time. We fixed two dataset bugs and retrained, and it still invents. The likely cause is that the model's discrete decoding corrupts literal names and numbers.
• So we stopped generating the summary. By default it is assembled from the extracted fields, after matching each one to the closest sentence in the source. That summary is correct 88–93% of the time and is also the fastest option.
• The goal of under 100 ms per remember on the GPU path is not met yet (359 ms).

The full experiment log, with every table, ablation, dead end and dataset bug, is in docs/LAB_NOTEBOOK.md (in Spanish).

Roadmap

• Linker that can abstain: when unsure, return the candidates to the agent instead of guessing.
• A cheaper linker: a small model, called only when the note contains a change word.
• An explicit scope field, to fix production vs staging vs analytics mix-ups.
• Hybrid search (BM25 + embeddings) for notes that say the same thing in other words.
• Trajectory scenarios, to test whether seeing why a decision was dropped helps, beyond filtering what is obsolete.
• Evaluation on real agent histories.
• PyPI release.

Status

v0.2.1, a research prototype. The MCP server works end to end and the tests pass (python test_smoke.py, python smoke_mcp.py). The API may still change.

License

MIT

Credits

The NCM (Next-Concept Modeling) base model is by HazyResearch. EvoMem's history design is inspired by hierarchical, versioned memory (CoMeT-style supersede / merge).