Zero-Knowledge Techniques for Private AI Memory

Here is the uncomfortable fact at the center of every private memory design: the embedding is the leak. You can encrypt the text perfectly, hold the keys entirely on the client, and hand the server nothing but ciphertext, and the moment you also hand it a plaintext 1024-dimensional float vector so it can run similarity search, you have given away most of the meaning. The vector is not metadata. It is a lossy but recoverable projection of the sentence.

What follows is an accounting of the options and what each actually costs in latency. None are free.

Option A: encrypt the text, keep plaintext vectors

This is what nearly everyone ships, including us. Content is encrypted at the application boundary; the embedding sits in a pgvector column in the clear so the database can run an approximate nearest neighbor index over it. Latency is normal: an HNSW index over a few million vectors returns in single-digit milliseconds, and the only added cost is decrypting the k rows you return.

The leak is precise and it is worse than intuition suggests. Morris et al., Text Embeddings Reveal (Almost) As Much As Text, train an iterative model that inverts embeddings back to text and recover 32-token inputs exactly in a substantial fraction of cases, with high similarity on the rest. The attack needs black-box query access to the same embedding model, which for a commercial API is a credit card. So "the vectors are just numbers" is not a defensible claim in a threat model. An attacker with your vector table has an approximate copy of your corpus.

What Option A does buy is real: it defeats the actual common breach. A leaked database dump, a stolen backup, a misconfigured bucket, a subpoena served on the host. Those get ciphertext plus vectors, and turning vectors into text requires a deliberate, model-matched inversion effort rather than a SELECT. Encrypting each ciphertext under its own scrypt-derived key also means there is no single unwrapping step that yields the whole store.

Option B: client-side embedding plus an encrypted index

Next rung up. The client runs the embedding model locally, so the raw vector never leaves the device, and uploads only an encrypted representation for the server to search.

Two problems. The first is mundane and often fatal: you now have to ship and version an embedding model on every client, and a browser, an iPhone, and a Linux desktop must produce comparable vectors for the same input, forever, or your index silently splits into incompatible halves. The second is cryptographic. To search, the server needs an operation on the encrypted vectors that correlates with plaintext distance, which is exactly what semantic security forbids. So schemes in this family use order-preserving or distance-preserving transforms, and those leak the thing they preserve.

Leaked distances are more damaging than they sound. Given the pairwise distance matrix of a corpus, an adversary can cluster it and mount known-plaintext attacks: insert a document whose plaintext you control, observe its distances to everything else, and you have a ruler for the entire space. The literature on encrypted-database inference attacks has repeatedly shown that access-pattern and distance leakage compose into near-full recovery given modest auxiliary knowledge.

Option C: PIR and homomorphic encryption

The cryptographically honest answers. Homomorphic encryption lets the server compute an inner product on ciphertexts without decrypting, so it can rank encrypted vectors against an encrypted query. Private information retrieval lets a client fetch record i without the server learning i. Combine them and the server genuinely learns nothing: not the content, not the query, not which rows matched.

The cost is the reason you are not using this. CKKS-style schemes handle the arithmetic you need (approximate real-number addition and multiplication with SIMD batching), but ciphertext expansion runs one to two orders of magnitude and per-operation cost is thousands of times a plaintext float op. Worse for this use case: PIR and HE search are fundamentally linear scans. The whole point of an ANN index is to avoid touching most of the corpus, and "which nodes did you traverse" is precisely the access pattern you were trying to hide. You cannot have a sublinear encrypted index and full access-pattern privacy at the same time.

Concretely, single-digit millisecond search becomes seconds to tens of seconds at corpus sizes where plaintext ANN does not break a sweat. For a batch analytics job over a few thousand records, that is survivable. For a memory lookup on the critical path of a conversation, it is not.

Option D: trusted execution environments

Run the search inside an enclave. Data arrives encrypted and is decrypted only inside a hardware-isolated region the host OS and hypervisor cannot read. Remote attestation lets the client verify which code image is running before releasing keys. Performance is within a small factor of plaintext, which makes this the only strong-privacy option that keeps a normal ANN index and a normal latency budget.

The catch is the trust assumption. You are trusting a silicon vendor's implementation, its firmware update path, and its resistance to side channels. That trust has been broken repeatedly in published work: speculative execution leaks, voltage fault injection, and cache-timing attacks have each produced enclave key extraction, and each was patched. TEEs are a strong control against a malicious cloud operator and a weak one against a well-resourced attacker with physical access.

What is actually deployable

Option A, with the encryption done properly, and Option D if your threat model includes the host and your latency budget can absorb the operational complexity. Everything else is either a research artifact or a false sense of security.

option  search latency   server learns          shippable
A       ~5ms             semantics (via inv.)   yes
B       ~10ms            pairwise distances     technically
C       seconds-to-min   nothing                no (latency)
D       ~5ms             nothing (if silicon    yes, with a
                         holds)                 trust assumption

The question to ask any vendor claiming zero-knowledge semantic search is short: where is the vector computed, and what exactly does your server see when it ranks? If the answer is that the server compares numbers it can read, the search is not zero-knowledge, whatever the text encryption looks like.

There is also a lever easier than any of the above and often more effective: store less. A system that distills a conversation into the three facts worth keeping has a far smaller leak surface than one embedding every turn, whatever crypto sits underneath. Reducing what exists costs no latency.

Unimatrix runs Option A today: AES-256-GCM at rest with a per-ciphertext scrypt-derived key, and vectors readable by the search path because that is what makes recall fast enough to be worth using. Self-hosting via Docker is the answer for anyone whose threat model includes us specifically. The full breakdown, including what we can and cannot see, is on the security page.