How to test agent memory

What should an agent memory test measure first?

Start with direct facts and isolation, then move to harder temporal and relevance cases. A memory system that cannot retrieve a user’s stated plan or keep one tenant out of another should not proceed to a large answer-quality test. After the fundamentals pass, measure whether the right memories improve the next model response without stuffing the context window.

Use a frozen evaluation set and preserve the retrieved records for each run. A final answer alone cannot show whether the model succeeded because of memory, luck or a long prompt. Record the write event, extracted memories, retrieval score, prompt tokens and final response.

• Exact fact recall across sessions
• Precision and irrelevant recall
• Contradiction and temporal resolution
• User, agent and tenant isolation
• Correction, TTL and deletion
• Latency, token count and answer improvement

What does a practical memory evaluation matrix look like?

Create 30–50 realistic cases with known expected memories. Each case should include the write sequence, query, allowed scopes and grading rubric. Keep separate scores for retrieval and answer quality so a model upgrade does not hide a retrieval regression.

Use a five-point rubric for retrieval relevance, factual correctness, temporal correctness, scope safety and response usefulness.

Are public AI memory benchmarks enough?

Public benchmarks such as LoCoMo and LongMemEval are useful for comparing models on common tasks, but they do not encode your customer vocabulary, retention rules or tenant model. Vendors also choose prompts, retrieval budgets and answer models, so a headline accuracy number is not directly portable to production.

Use a benchmark as a screening signal, then replay your own conversations. A memory layer should report what was retrieved, how it was ranked and why an old fact was invalidated. The audit path matters more than a single benchmark chart.

How do I build a small test harness?

Keep the harness boring and deterministic. Each case writes events, waits for extraction, performs a scoped search and stores the returned memories alongside the final model answer. Use the same LLM and prompt across vendors so the memory layer is the variable.

• Freeze the test conversations and expected answers in version control.
• Wait for asynchronous extraction before testing retrieval.
• Run each case at least three times to expose instability.
• Keep a baseline with full transcript replay for answer quality.
• Store raw retrieved records for debugging and regression review.

for (const test of memoryTests) {
  await memory.add({
    userId: test.userId,
    agentId: test.agentId,
    messages: test.events,
  });

  const result = await memory.search({
    userId: test.userId,
    agentId: test.agentId,
    query: test.query,
    limit: 8,
  });

  record({
    testId: test.id,
    memories: result.memories,
    latencyMs: result.metrics.latencyMs,
    promptTokens: result.metrics.tokenCount,
  });
}

Which memory metrics should appear in the report?

Report retrieval and business-facing outcomes separately. Precision at the prompt, recall of required facts, contradiction resolution, scope violations, deletion failures, p50 and p95 latency, prompt tokens per turn and task success are enough to expose the important tradeoffs without creating a vanity dashboard.

For each query, compare three modes: no memory, full transcript replay and memory retrieval. The memory layer is valuable when it improves task success while using less context. If it only improves a benchmark score but increases hallucinated facts or privacy failures, the implementation is not ready.

When is agent memory ready for production?

Production readiness starts when the hard gates pass: no cross-tenant retrieval, no resurrection after deletion, a documented export and an alert for extraction or retrieval failures. Soft quality metrics then guide optimization without blocking every release.

Run the suite after prompt changes, model changes, embedding changes, schema changes and retention-policy changes. Memory regressions often come from a changed extraction prompt rather than the retrieval database itself.

• Hard gate: zero user and tenant boundary failures.
• Hard gate: deleted data is absent from retrieval and export.
• Hard gate: a complete backup and restore test has passed.
• Quality gate: the priority conversation set meets its recall target.
• Operations gate: p95 retrieval stays inside the response budget.
• Business gate: memory improves the next action enough to justify its cost.