Building a Librarian Agent: Automated Curation for AI Memory

A conversational memory store crosses a threshold somewhere around ten thousand entries where top-k retrieval starts returning five phrasings of the same fact. Recall metrics look fine. The store contains the answer, and it returns the answer, five times, consuming the entire context budget to deliver one piece of information. Nothing is broken in a way a test would catch.

The fix is a background process with no user-facing surface. Call it a librarian: a periodic pass over the store that merges duplicates, reconciles contradictions, demotes entries nobody has ever needed, and writes back structure the write path was too rushed to compute.

What the write path is allowed to do

Curation only works if the write path stays cheap, so define the boundary first. On insert:

• Embed the candidate text. One API call, unavoidable.
• Nearest-neighbor lookup against the caller's existing memories. If cosine exceeds 0.97, treat it as a restatement: update the existing row's timestamp and access count, return that row's id, insert nothing.
• Otherwise insert, with written_at, source client, and a curated_at of null.

That is it. Two round trips, no LLM in the loop, single-digit-to-low-hundreds milliseconds. Everything harder happens later. The 0.97 threshold at write time is deliberately conservative, higher than the merge threshold below, because a false merge on the write path is unrecoverable: the caller gets an id for a row that does not say what they wrote.

The pass itself

for palace in palaces_with_new_writes_since(last_run):
    cands = select_memories(palace, curated_at IS NULL
                                    OR written_at > curated_at)

    # 1. exact-duplicate collapse (cheap, deterministic, no LLM)
    for group in cluster_by_cosine(cands, threshold=0.95):
        keep = longest_text(group)          # superset text, not newest
        for other in group - {keep}:
            keep.access_count += other.access_count
            keep.written_at = max(keep.written_at, other.written_at)
            mark_merged(other, into=keep)   # soft delete, keeps provenance
        write(keep)

    # 2. contradiction detection (LLM, narrow window)
    for a, b in pairs_about_same_entity(cands, cosine_range=(0.75, 0.95)):
        verdict = llm_judge(a.text, b.text)   # SAME | CONTRADICTS | UNRELATED
        if verdict == "CONTRADICTS":
            older, newer = sort_by_written_at(a, b)
            older.superseded_by = newer.id     # never deleted
            write(older)

    # 3. demotion
    for m in cands:
        if m.access_count == 0 and age_days(m) > 90 and m.importance < 0.3:
            m.archived = True
            write(m)

    stamp_curated_at(cands, now())

Step 1 does most of the work and it never calls a model. Cosine at 0.95 over Voyage-class embeddings is a reliable near-paraphrase detector, and the merge rule is mechanical: keep the longest text, sum the access counts, take the latest timestamp, soft-delete the rest with a pointer to the survivor. Keeping the longest rather than the newest matters. Newer restatements are frequently shorter and drop qualifiers, so "keep newest" loses information on every merge.

Step 2 is the only place an LLM is required, and note the narrow cosine band. Below 0.75 the entries are about different things and cannot contradict. Above 0.95 they are the same statement and step 1 already handled it. The interesting zone is the middle, which on a real store is a small fraction of candidate pairs. That gating is what makes the LLM cost survivable.

Why async, and why idempotent

Doing any of this on the write path is a mistake with a specific shape. A remember call is made by a model mid-conversation while the user waits. Adding a clustering query and an LLM judgment call pushes that from 150ms to several seconds, and now a curation-service outage means the user cannot store anything. Curation is a quality improvement, not a correctness requirement, and it should never be able to fail a write.

Idempotence is what lets you run it aggressively. The pass must be safe to kill halfway through and restart, because it will be: deploys, OOMs, rate limits. Two properties get you there. Every mutation is a state transition that is a no-op when reapplied (setting superseded_by to the same id twice changes nothing; a row already marked merged is filtered out of the next candidate set). And curated_at is stamped per row rather than per run, so a crash means the unstamped rows are picked up next time and the stamped ones are skipped. Never keep the watermark in a single global row.

The failure modes, honestly

Over-merging distinct facts."The staging database is on port 5432" and "the production database is on port 5432" sit around 0.96 cosine. Merge them and you have destroyed a distinction that mattered, with no way to recover the original text unless you soft-deleted. Always soft-delete. Additionally, block merges where the two texts differ on a named entity or a numeric literal, regardless of cosine. A cheap regex guard on numbers and capitalized tokens prevents a large share of the damaging cases.

The curator hallucinating a merged fact. This is the worst outcome available. If you ask an LLM to write a unified summary of two memories, it will occasionally produce a synthesis that neither source stated, and that synthesis is now indistinguishable from user-provided ground truth. The mitigation is a hard rule: the curator selects text, it does not generate text. Merges pick one of the existing strings. If you truly need synthesis, store it as a derived entry with a pointer to its sources and a flag, so it can be audited and rolled back.

Re-embedding cost. Any pass that rewrites text has to re-embed it, and a naive implementation re-embeds everything on every run. Only embed when the stored text actually changed, and hash the text to check. On a store of 100k memories with 2% daily churn, this is the difference between 2,000 embedding calls a day and 100,000.

A curator that selects and links is safe. A curator that writes prose is a hallucination generator pointed at your source of truth.

Run frequency is a tuning decision rather than a fixed answer, but hourly per active workspace works well: recent enough that duplicates from a single long session get collapsed before the next session retrieves them, infrequent enough that the LLM judgment calls stay cheap.

Unimatrix runs this as a separate worker rather than inside the MCP request path, so a slow curation pass can never make remember time out. Storage limits per tier, including how many memories a workspace can hold before archival matters, are listed on pricing. The supersession semantics that the pass depends on are drawn from the ranking approach in the Generative Agents paper, which scores memories on recency, importance, and relevance together.