A context window is not a memory system
A context window is temporary working memory. It is the space the model can see on one call. It does not decide what is worth keeping, it does not resolve contradictions, and it resets when the session ends.
When teams say “we will just use a bigger window”, they usually mean “we will send more history”. That works until the history is larger than the window, the token bill matters, or two old lines contradict each other.
The three costs of replaying history
| Cost | What happens | What memory changes |
|---|---|---|
| Tokens | Every turn pays for the same transcript again. | Retrieve a small ranked set of durable memories. |
| Latency | A longer prompt takes longer to process. | Keep the context short and relevant. |
| Contradictions | Old lines compete with new truth. | Merge and supersede facts, then rank the current truth first. |
The memory types worth modeling
| Type | What it answers | Example |
|---|---|---|
| Fact | What is true about this entity? | The account is on the Scale plan. |
| Preference | How does this person want to be treated? | Friday summaries, no calls before 10:00. |
| Decision | What did we decide, and why? | Escalate to the on-call engineer. |
| Constraint | What limits the solution? | The EU migration must finish before the audit. |
| Episode | What happened last time? | The CSV import produced duplicate rows. |
The four parts of a memory architecture
A durable memory system has a write path, a consolidation step, a read path and a deletion path. Most demos only build the write path and the read path.
- Write: accept events with user, agent, session and tenant scope.
- Consolidate: extract, deduplicate, merge, supersede and compress.
- Read: score similarity, recency, kind and graph relations, then rerank under a token budget.
- Delete: remove by memory, user, tenant or TTL, with an audit trail.
Evaluate memory like a product feature
Do not evaluate memory by asking whether the answer “feels better”. Measure the behaviors memory is supposed to change.
| Metric | Definition | Target direction |
|---|---|---|
| Recall of earlier turns | Correct use of a fact that appeared at least ten turns earlier. | Up |
| Repeat-question rate | How often the agent asks for information it already has. | Down |
| Contradiction rate | How often the agent repeats a superseded fact. | Down |
| Tokens per turn | Context tokens handed to the model per turn. | Down or flat |
| Retrieval precision | Share of retrieved memories that were relevant. | Up |
| Deletion correctness | Deleted data no longer influences answers. | 100% |
An implementation checklist
- Start with one user-facing workflow, not every agent in the company.
- Define the memory kinds and scopes before the first write.
- Keep the source and timestamp for every memory.
- Set a token budget and a maximum number of retrieved memories.
- Test a changed fact, a deleted user and a cross-tenant query.
- Add an export path before you need it.
Frequently asked questions
How much memory should an LLM app keep?
Keep what changes future behavior: facts, preferences, decisions, constraints and meaningful episodes. Drop small talk and transient state unless it affects a later decision.
Is long-term memory the same as a vector database?
No. A vector database is a retrieval index. Long-term memory also needs extraction, merging, scopes, timestamps, versioning and deletion.
How do I stop the memory from becoming stale?
Supersede facts when they change, keep a timestamp and source, and rank the current truth first. Test a fact that changes twice in a week.
Can I use a larger context window instead?
You can postpone the problem, but you still pay for the history on every turn and you still risk contradictions. A larger window is useful; it is not a memory policy.
What is the smallest useful memory system?
A scoped store with add and search, a merge rule for changed facts, a token limit and a delete path. You can add graph retrieval and reranking later.
How do I evaluate memory without a large labeled set?
Use a small set of real conversations with known facts and changes. Measure recall, repeat questions, contradictions and tokens per turn. Twenty well-chosen cases are more useful than a generic leaderboard.