AI memory poisoning: how agents get corrupted and how to stop it

What is AI memory poisoning?

AI memory poisoning is the persistence of adversarial or incorrect information in a memory system so that it changes future agent behavior. The input may arrive through a user message, retrieved document, webpage, email, tool result or another agent. The attack succeeds when the system promotes that input into durable memory without enough source, scope or trust validation.

A single bad response is a transient error. A poisoned memory is repeated automatically in later sessions. That persistence increases impact because the agent may retrieve the false fact long after the attacker has left, and other users or agents may inherit the same contaminated summary.

How does an attacker poison agent memory?

The usual chain is source contamination, unsafe extraction, durable write, later retrieval and finally a consequential model or tool action. Attackers look for a write path that treats untrusted content as a fact or instruction and a read path that cannot distinguish that record from a verified preference.

How is memory poisoning different from prompt injection?

Prompt injection tries to make the current model follow attacker-controlled instructions. Memory poisoning tries to persist content that changes future runs. The two often combine: an injected document asks the agent to remember a false rule, and the memory system later reintroduces that rule as context.

Removing the malicious document after one request does not remove the extracted memory or its derived summaries. Defenses must therefore cover both immediate instruction hierarchy and long-term write, retrieval and deletion paths.

• Prompt injection affects the current context; memory poisoning affects later contexts.
• Retrieved memory must never become system policy or authorization.
• Source trust and provenance should survive extraction and summarization.
• High-impact actions require fresh confirmation outside the model.
• Incident response must remove derived records and embeddings, not only the source.

What are the warning signs of memory poisoning?

The strongest signals are unexplained behavior changes that persist across sessions, retrieved facts with missing provenance, contradictory preferences appearing without a correction event and tool calls that cite remembered instructions rather than current user intent. A sudden cluster of bulk writes from one source is another warning sign.

Do not wait for a visibly wrong answer. Monitor extraction categories, source distributions, trust changes, write volume, correction rates and high-impact fact changes. Store enough security metadata to investigate without logging the full sensitive memory content.

• A durable fact has no source or trust level.
• A retrieved record contains instructions rather than information.
• One document produces a burst of unrelated user or tenant memories.
• A high-impact preference changes immediately after untrusted tool output.
• The same false statement appears in summaries across several scopes.
• Corrections do not remove older derived retrieval artifacts.

Which defenses actually reduce memory poisoning risk?

Use defense in depth. Authenticate every write principal, minimize what can become memory, label source and trust, validate extracted fields and require stronger evidence for durable or high-impact facts. Retrieval should preserve those signals instead of flattening every record into one relevance score.

How should retrieval treat untrusted memories?

Retrieve trust alongside content and mark the memory block as data, not instructions. Exclude quarantined records from normal prompts, down-rank unverifiable claims and prefer current facts from authoritative systems. A memory record should never grant a permission or override a policy because the language inside it sounds authoritative.

When two facts conflict, preserve both with validity times and source trust. Return the newest verified value for ordinary tasks and explicitly disclose uncertainty when the system cannot resolve the conflict. For consequential actions, retrieve from the source system instead of trusting a remembered representation.

const memories = await memory.search({
  userId,
  agentId,
  tenantId,
  query,
  excludeTrust: ['quarantined'],
  requireProvenance: true,
  limit: 6,
});

const context = memory.renderAsUntrustedData(memories, {
  includeSource: true,
  includeObservedAt: true,
  maxTokens: 1200,
});

// Recheck permissions and account settings with the source system.
const account = await accounts.getCurrent(tenantId, userId);

How do you recover after memory has been poisoned?

Recovery starts by freezing the affected write path and identifying the source event and every record derived from it. Quarantine the lineage, invalidate caches and summaries, rotate any exposed credentials and notify the incident owner. Do not simply overwrite one visible memory while leaving embeddings or graph relationships active.

After removal, replay representative queries and high-impact tool checks to prove the false context no longer influences behavior. Record the correction reason and add a regression case so the same source pattern cannot silently return.

• Preserve an incident-safe audit trail without copying poisoned content into logs.
• Trace source event to extracted memory, summary, embedding and graph edge.
• Quarantine the full lineage before deleting confirmed malicious records.
• Recheck consequential systems for unauthorized changes.
• Replay retrieval and tool-decision tests after cleanup.
• Add the attack pattern to extraction and evaluation suites.

How do you test memory poisoning defenses?

Test poisoning as a write-to-action attack, not only as a prompt-injection string. Seed an untrusted document or tool response, allow the system to process it and verify that it cannot create a broadly scoped verified memory. Then test conflict, retrieval, correction and deletion across every derived record.

What is the right security model for durable AI memory?

Treat memory as untrusted, scoped data with a complete lineage. Authenticate the writer, validate the shape of what can be remembered, preserve provenance and trust, exclude quarantined records and reconfirm high-impact facts before action. Recovery must remove every derived copy, not just the first record an operator can see.