The Real Difference Between Context Windows and Memory

Inference is stateless. That single fact explains most of the confusion about what a context window is. When you send turn 40 of a conversation, the model has no recollection of turns 1 through 39. Your client library re-sent all of them, as text, in the same request. The model read them for the first time, every time. There is no session on the other end holding your history.

A context window is a buffer you refill on every request. Memory is state that survives the request. Those are different systems with different failure modes, and vendors advertising million-token windows have made it easy to believe the first one subsumes the second.

Three reasons a bigger buffer is not memory

Attention cost scales badly

Self-attention is quadratic in sequence length: each token attends to every other token, so cost grows as O(n²). Modern serving stacks soften the constant with FlashAttention, paged KV caches, and sliding window layers, but the asymptotic shape holds during prefill. Going from 20k to 200k tokens of context is not ten times the work, it is closer to a hundred times the attention work. Somebody pays for that, in latency or in price per token, usually both.

Retrieval accuracy is not uniform across the window

This is the finding that should change how you build. Liu et al. measured retrieval accuracy as a function of where the answer sits in a long input and found a U-shaped curve: models are accurate when the relevant information is at the beginning or the end of the context, and measurably worse when it is in the middle. In some configurations, performance on a mid-context fact dropped below the same model's performance with no context document at all. The paper is Lost in the Middle and it is worth reading the figures directly.

The practical consequence: filling a 200k window with everything you have does not guarantee the model uses any given piece of it. Position within the buffer is a real variable. If you are stuffing context, you are implicitly gambling that the fact that matters landed near an edge.

The cost arithmetic is brutal

Take a 200k-token context resent on every turn of a 50-turn conversation. That is 10 million input tokens for one session. At a representative $3 per million input tokens, $30 for a single conversation, and the majority of those tokens are the same text sent forty-nine times.

Now retrieve 2k tokens of the right material per turn instead. 100k tokens across the session, $0.30, a hundredfold reduction. And by the lost-in-the-middle result, the 2k version is often more accurate, not less, because everything in the buffer is relevant and nothing is buried at position 90k. Cheaper and better is a rare combination and it exists here.

KV caching helps, and it is not memory

The obvious objection is prompt caching. Providers let you mark a prefix as cacheable so the key and value tensors for those tokens are retained server-side and reused instead of recomputed. Cache reads typically bill at roughly a tenth of the input rate, and time to first token on a long stable prefix can drop by a large factor. It is a genuine and significant optimization. Use it.

It is still not memory, for four reasons that matter architecturally:

• It is a computation cache, not a data store. A KV cache holds intermediate attention tensors for a specific token sequence under a specific model. It is not queryable. You cannot ask it what it contains.
• It expires fast. Typical TTLs are minutes. Cross-device continuity is measured in hours and days.
• It is keyed on an exact prefix. Change one token near the start and the whole cache downstream of that point is invalidated. This is why you order stable content first.
• It lives on the provider's infrastructure and dies with the model version.Your data being retrievable is now a property of someone else's eviction policy. Switch from Claude to Gemini and there is nothing to carry over.

That last point is the one that decides the architecture. Any state that lives inside a provider's serving layer is by definition not portable across providers. If continuing a conversation on a different model with full context is a requirement, the state has to live somewhere you control.

The context window is working memory: fast, expensive, and gone at the end of the request. Memory is the disk behind it. Confusing the two is how you end up paying long-context prices for amnesia.

What the split looks like in practice

The division of labor is not subtle once you name it. Anything the model needs for the current reasoning step goes in the window. Anything it might need later goes in the store, and gets retrieved when a query indicates it is relevant.

# Per request, assembled fresh:
[ system prompt        ]  ~500 tokens   stable, cache this prefix
[ retrieved memories   ]  ~2k tokens    top-k from the store, changes per turn
[ recent turns         ]  ~4k tokens    last few exchanges verbatim
[ current user message ]  ~200 tokens

# Persisted outside the request:
memories table  ->  embedding, text, written_at, superseded_by, palace_id

Note the ordering. The stable system prompt sits first so the cache prefix stays valid across turns. Retrieved memories sit next, followed by recent turns and the user's message, which puts the highest-value material near the edges of the buffer where the U-curve says attention is strongest.

The reason to care about this distinction is not elegance. A system built on window-stuffing degrades in a specific pattern: it works in demos with short histories, gets expensive around turn 20, and starts dropping facts that are demonstrably present in the prompt somewhere around the point where the buffer fills. A system built on retrieval has flat cost per turn and its failures are retrieval failures, which you can measure, log, and fix. One of those is debuggable.

Unimatrix sits on the persisted side of that line. Memories live in Postgres with pgvector, encrypted at rest, so the context that follows you from ChatGPT on a phone to Claude on a tablet is not dependent on any single provider's cache surviving. The storage and encryption details are written up under security.