Why Vector Search Alone Can't Give LLMs a Memory

Cosine similarity answers one question: which stored chunk points in roughly the same direction in embedding space as this query? That is a genuinely useful question. It is also not the question a memory system needs to answer, which is closer to: what does this user need me to know right now, including the things they did not ask about.

The gap between those two questions is where most retrieval-augmented systems quietly fail. You can watch it happen with a single test. Store two facts about the same person:

store("Marc prefers TypeScript over Python for new services")   # written 2025-01-14
store("Marc moved the ingestion service to Go last quarter")     # written 2026-06-02

recall("what language should I use for Marc's new service?")

A pure vector store ranks the first entry higher. It contains "prefers," "TypeScript," "Python," and "services", lexically and semantically dense against the query. The second entry is the one that actually changed the answer, and it loses because "moved the ingestion service to Go" is a weaker semantic match for a question about preference. Similarity has no concept of supersession.

Three specific failure modes

Identifiers get smeared. Embedding models compress text into a fixed-dimensional vector, 1024 dimensions for a typical retrieval model. That compression is lossy in a particular way: rare tokens with low semantic content but high discriminative value get averaged into the surrounding context. Search for error code ERR_MEM_4412 and a dense retriever will happily return chunks about ERR_MEM_4413, because those two strings are nearly identical in embedding space and completely different in meaning. BM25, a bag-of-words scoring function from 1994, gets this right and dense retrieval gets it wrong. This is the single strongest argument for hybrid search.

Similarity is not sufficiency. Top-k retrieval returns k chunks that each resemble the query. It does not return a set that jointly answers it. If the answer requires facts from three documents, and only two of them are individually similar to the question, the third never enters the context window. The model then answers confidently from an incomplete set, which reads to the user as a hallucination but is actually a retrieval bug.

Everything is equally present, forever.A vector index has no time dimension unless you add one. A note from eighteen months ago competes on equal footing with one from this morning. For durable facts (a person's name, a system's architecture) that is correct. For volatile facts (current sprint, current employer, current stack) it is actively wrong, and there is no way to tell the two apart from the embedding alone.

What you have to put on top

The fix is not a better embedding model. It is treating vector similarity as one input to a ranking function rather than as the ranking function. A workable score looks something like this:

score = w1 * cosine(query_vec, mem_vec)
      + w2 * bm25(query_terms, mem_text)
      + w3 * exp(-age_days / half_life)      # only for volatile memories
      + w4 * access_frequency_normalized
      - w5 * superseded_flag

Each term fixes a specific failure above. BM25 recovers exact identifiers. The exponential decay term handles volatility. With a half-life around 30 days for a "current state" memory, a fact from 60 days ago retains 25% of its recency weight, enough to surface if nothing newer exists but not enough to beat a fresh contradicting entry. The supersession flag requires an explicit write-time or curation-time decision: when a new memory contradicts an old one about the same entity, mark the old one rather than deleting it, so provenance survives.

Fusing keyword and vector rankings is a solved problem. Reciprocal Rank Fusion combines ranked lists with 1 / (k + rank) per list, typically with k = 60, and it requires no score normalization between systems that produce incomparable scales. It is used in production in Elasticsearch and it is roughly twenty lines of code to implement yourself.

The retrieval step is not the whole system

Even with perfect ranking, there is a layer above retrieval that pure vector systems skip entirely: a write policy. Every conversation produces text. Almost none of it is worth remembering. If you embed and store everything, the index fills with restatements of the same three facts, and top-k returns five near-duplicates that consume the context budget without adding information.

Deduplication at write time is cheap and effective. Before inserting, run the candidate against the existing index; if the nearest neighbor exceeds a cosine threshold around 0.95, merge rather than insert. This one rule typically cuts store size by a large fraction on conversational input, and it improves recall quality directly, because the k slots go to k distinct facts instead of k paraphrases.

A memory system is a ranking problem plus a write policy plus a conflict-resolution rule. Vector search is a component of the first one.

Where this lands

The practical takeaway is a design constraint rather than a tool recommendation. If your retrieval layer cannot express "this fact replaced that one on this date," you do not have memory, you have semantic search over a transcript. Adding a supersession column, a written-at timestamp, and a keyword index to an existing pgvector table is a day of work and it changes the behavior more than swapping embedding models will.

This is the layer Unimatrix is built around: retrieval is hybrid and time-aware by default, and the supersession rule is enforced at write time rather than left to the calling model. The tool surface for that is documented in the MCP reference, and the storage and encryption details are in security.