How Embedding Drift Quietly Breaks Retrieval Over Time

Swap your embedding model and every vector you wrote before the swap becomes noise. Not degraded, not slightly worse: noise. Two models produce vectors in two unrelated coordinate systems, and cosine similarity between them is a meaningless number that nevertheless computes cleanly, sorts cleanly, and returns a top-k list that looks exactly like a working result.

Nothing throws. There is no dimension mismatch if the two models happen to share an output width, no constraint violation, no log line. Postgres computes the distance, pgvector orders by it, your API returns 200. Recall quality collapses and the only symptom is users saying the assistant "feels dumber lately."

Three unrelated problems share one word

"Embedding drift" gets used for three failure modes with wildly different severity and completely different fixes. Conflating them is how teams end up applying a monitoring solution to a migration problem.

1. Model version change (catastrophic, silent)

You move from voyage-3 to voyage-3.5, or from text-embedding-ada-002 to text-embedding-3-small. The learned projection changed. There is no rotation matrix you can apply to reconcile them, no alignment trick that recovers the old vectors, because the two models did not learn the same basis and were never trained to agree. A query vector from the new model compared against a corpus embedded by the old one produces similarity scores concentrated near the distribution mean, which is to say approximately random ranking with a plausible-looking score column.

The falsifiable version of this claim, and one worth running yourself before you trust anyone on it: embed 1,000 documents and 100 known-relevant queries with model A. Compute recall@10. Now embed only the queries with model B, leave the corpus on A, and recompute. Recall@10 will land near the random baseline of 10/1000 = 1% rather than anywhere near the 60 to 90% you had. If it does not, your two models share a lineage and a frozen output head, which is the exception and not something to design around.

2. Dimension change (loud, and therefore harmless)

1024 dimensions to 1536 dimensions. This one is a gift, because it breaks at the type level. A pgvector column is declared vector(1024) and an insert of a 1536-element array fails outright. The pgvector README documents both the fixed-dimension typing and the index-build constraints (HNSW and IVFFlat indexes are built for a specific column, and there is a hard 2,000-dimension ceiling for indexed columns). You get a hard error at write time and you fix it during a migration you planned. Nobody has ever been quietly ruined by a dimension change.

3. Genuine distribution drift (gradual, actually subtle)

The model is unchanged. Your corpus moved. A memory store that started as notes about Kubernetes and Terraform in 2024 and is now full of MCP tool schemas, agent trace vocabulary, and internal product nouns coined last quarter is asking the encoder about tokens it saw rarely or never. Out-of-distribution text embeds toward the center of the learned manifold, which compresses the pairwise distances between exactly the documents you most need to distinguish.

This is measurable without ground-truth labels. Track the mean pairwise cosine similarity of a random sample of stored vectors, weekly, and the mean cosine between each query and its top-1 result. When corpus vectors start bunching (mean pairwise similarity climbing) while top-1 query similarity flattens, the index is losing discriminative power. That is the signal to consider a newer or domain-adapted encoder, and it is a quarters-long trend, not an incident.

The schema fix is two columns

Version the vectors. Refuse to compare across versions. That is the entire discipline, and it has to be enforced in the query, not in a comment.

ALTER TABLE memories
  ADD COLUMN embedding_model_id  text NOT NULL,   -- 'voyage-3.5'
  ADD COLUMN embedding_version   int  NOT NULL,   -- monotonic, ours not theirs
  ADD COLUMN embedding_dim        int  NOT NULL;

-- every read is scoped. no exceptions, no default.
SELECT id, content, 1 - (embedding <=> $1::vector) AS score
FROM memories
WHERE user_id = $2
  AND embedding_model_id = $3
  AND embedding_version  = $4
ORDER BY embedding <=> $1::vector
LIMIT 20;

CREATE INDEX ON memories USING hnsw (embedding vector_cosine_ops)
  WHERE embedding_version = 4;   -- partial index per live version

Two details matter here. embedding_model_id is the provider string, and embedding_version is yours, incremented whenever anything upstream of the vector changes: the model, the chunker, the normalization, the instruction prefix some models prepend for retrieval-vs-document asymmetry. That last one bites people. Embedding a document with the query-side prefix produces a vector that is subtly wrong in a way no model-name check will catch.

The partial index is what makes dual-write survivable. During a migration you have two live versions in one table, and a single HNSW index over both would be traversing a mixture of two incompatible spaces. Partial indexes per version keep each graph internally coherent.

Migration procedure

Assume one million rows, a 1024-dimension model, and a requirement of zero read downtime.

• Add embedding_v_next as a nullable column of the new dimension, plus the version columns above. Backfill the version columns for existing rows with the current model id. Do this first and deploy it alone.
• Turn on dual-write. Every new or updated memory gets embedded twice and both vectors are stored. Reads still serve exclusively from the old column. Dual-write doubles your embedding spend for the duration of the migration and is the price of not having a cutover window.
• Backfill in batches ordered by primary key, with a resumable cursor persisted outside the job. Batch size tuned to the provider's tokens-per-minute limit, not to what your database can take. Rate limits are the binding constraint.
• Build the new partial HNSW index only after the backfill completes. Building it during the backfill means paying insert-time graph maintenance on every row and getting a worse graph.
• Shadow-evaluate. Run your held-out query set against both versions and compare recall@10 and mean reciprocal rank. This is the step that gets skipped, and it is the only one that tells you the new model is actually better on your corpus rather than on MTEB.
• Flip reads behind a flag, per-tenant if you can. Keep the old column and index for a week. Then drop them, and only then stop dual-writing.

What a million rows costs

Do the arithmetic before you commit to a model change. One million memories averaging 200 tokens is 200 million tokens. At roughly $0.02 to $0.13 per million tokens across current retrieval-model pricing tiers, that is $4 to $26 for the corpus pass, doubled if you count the dual-write period. The API cost is not the problem.

The wall-clock is. At a 1M tokens-per-minute rate limit, 200M tokens is 200 minutes of pure throughput at perfect utilization, so realistically a day with retries and backoff. Then the HNSW build: one million 1024-dimension float32 vectors is about 4 GB of raw vector data, and the graph wants to fit in maintenance_work_mem or the build degrades badly. Budget the engineering time for the resumable backfill job and the shadow evaluation harness, because those, not the invoice, are what make people postpone a migration they should have done.

An unversioned vector column is a silent-failure generator with a one-line fix that only works if you apply it before you need it.

Unimatrix stores a model id and version on every vector and scopes every recall query to a single version, which is less a feature than a refusal to ship a system that can return confident garbage. The retrieval path and the tool surface that sits on it are in the MCP reference.