Two different kinds of context
Retrieval-augmented generation, or RAG, retrieves passages from a knowledge base and puts them in the prompt. The knowledge is usually external, shared and document-shaped: policies, manuals, product docs, tickets, contracts or code.
Agent memory stores what the agent learned while interacting with a person, an account or a project. The knowledge is usually internal to the relationship: preferences, decisions, constraints, past attempts and the current truth about an entity.
A RAG question sounds like “What does the refund policy say?” A memory question sounds like “Does this customer prefer a call or an email, and who signs off on a refund?”
RAG vs agent memory at a glance
| Dimension | RAG | Agent memory |
|---|---|---|
| Source | Documents, tickets, code, web pages | Events, conversations, tool results, decisions |
| Question answered | What does the knowledge base say? | What do we know about this user, account or project? |
| Update behavior | Re-index when the document changes | Merge, supersede and version when the truth changes |
| Temporal reasoning | Usually limited to document dates | Needs timestamps and current-truth resolution |
| Scope | Often shared across users | Must be filtered by user, agent, session and tenant |
| Failure mode | Retrieves the right document for the wrong person | Remembers an old fact that is no longer true |
| Token cost | Grows with the number of passages | Grows with the number of memories unless compressed |
Why RAG alone still forgets the conversation
A document index does not know that the customer said “never call before 10:00” three weeks ago. That sentence is not a policy document. It is a preference attached to a person, and it should change the next response.
Teams often try to solve this by stuffing the transcript into the prompt or by summarizing it into a rolling note. Both approaches work for a while. Then the transcript grows, the summary loses the reason behind a decision, and two notes contradict each other.
Memory is the part that decides what deserves to persist, merges it with what is already known and returns only the lines that matter. RAG is the part that finds the policy or the manual. You usually need both.
When to use both
Use both when the answer depends on external knowledge and on the state of the relationship. That describes most production agents: support, sales, onboarding, healthcare navigation, coding assistants and research copilots.
- Support: retrieve the policy, remember the customer’s plan, open issue and escalation rule.
- Sales: retrieve the product brief, remember the objection history and the last commitment.
- Coding: retrieve the repository docs, remember the architecture decision and the review feedback.
- Research: retrieve the sources, remember the hypotheses and the paths already ruled out.
A simple architecture that keeps the prompt small
Run the two retrievals in parallel, then merge them under one token budget. Give the model a clear boundary between “what the knowledge base says” and “what we know about this user”.
- Keep provenance: the model should be able to cite the document or the memory behind a claim.
- Keep the token budget explicit: documents and memories should not compete without a limit.
- Keep the scopes separate: a document is not a fact about a person, and a memory is not a policy.
const [documents, memories] = await Promise.all([
knowledge.search({ query, limit: 4 }),
memory.search({ query, userId, tenantId, limit: 5 }),
]);
const prompt = [
'Answer from the policy excerpts below.',
'Use the customer memories only when they are relevant.",
formatDocuments(documents),
formatMemories(memories),
].join('
');How to evaluate the combination
| Metric | What it catches | How to measure |
|---|---|---|
| Recall of earlier turns | The agent forgot a preference or decision. | Ask questions whose answers appeared 10+ turns earlier. |
| Contradiction rate | The agent repeats an old fact after a change. | Introduce a change and ask a question that depends on it. |
| Tokens per turn | The prompt is growing without control. | Measure the context handed to the model, not the whole transcript. |
| Retrieval precision | The agent receives irrelevant memories. | Review the top five memories for a sample of real questions. |
| Deletion correctness | A deleted user still appears in context. | Delete a test user and search again, including derived memories. |
The short version
- Use RAG for shared knowledge that lives in documents.
- Use memory for state that belongs to a user, account, agent or project.
- Use both when the answer depends on policy and on the relationship.
- Do not use a longer prompt as a substitute for either one.
Frequently asked questions
Is agent memory a replacement for RAG?
No. RAG retrieves shared knowledge; memory stores state about a user, account or project. Most production agents need both.
Can I store documents in AI Memory?
AI Memory accepts text or JSON events, so you can send document excerpts as events. For large shared knowledge bases, a dedicated RAG index is usually the better tool; use memory for the facts and decisions that belong to a specific scope.
Why not just put the transcript in the prompt?
A longer prompt costs more on every turn, eventually exceeds the window and lets old lines contradict new ones. Memory extracts and merges the signal instead of replaying the transcript.
Does memory help with hallucinations?
It can reduce a specific kind of error: forgetting or contradicting known facts. It does not replace grounding, citations or evaluation for external knowledge.
Which one should I build first?
If users repeat themselves and the agent forgets, build memory first. If the agent cannot answer from your knowledge base, build RAG first. If both are true, build the smaller path that unlocks the most user value.
How do I keep the combined prompt small?
Set a token budget for documents and memories separately, retrieve a small ranked set from each, and drop low-confidence items instead of filling the budget.