AI memory benchmarks: what LoCoMo and LongMemEval prove

What do AI memory benchmarks measure?

A memory benchmark usually measures whether a system can answer questions after a sequence of interactions or documents has been processed. The score may combine retrieval recall, temporal reasoning, multi-session aggregation, knowledge updates and final answer accuracy.

That combination makes headline scores difficult to compare. Two systems can report the same accuracy while using different reader models, retrieval budgets, context windows and grading prompts. A production decision needs the benchmark construction and methodology, not only the final number.

What does the LoCoMo benchmark test?

LoCoMo, introduced in “Evaluating Very Long-Term Conversational Memory of LLM Agents,” tests memory over long multi-session conversations. It covers tasks such as recalling facts, understanding events across sessions and reasoning about temporal information.

The benchmark is useful because the conversation extends far beyond a short chat. Its constraints still differ from a production assistant that must retrieve account data, obey tenant boundaries, call tools and delete user records. Use LoCoMo to compare long-term conversational memory, not as proof of enterprise governance.

• Multi-session conversational history.
• Single-hop and multi-hop recall.
• Temporal understanding of events.
• Answer quality after long histories.
• Less direct coverage of tool authorization and tenant isolation.

What does LongMemEval measure?

LongMemEval is designed to evaluate long-term interactive memory in chat assistants. Its tasks include information extraction, multi-session reasoning, temporal reasoning, knowledge updates and abstention when the answer is not supported.

Those constructs are closer to a product memory lifecycle than a static long-context test. Knowledge updates and abstention matter because production systems must prefer current truth and say when memory does not contain an answer instead of inventing one.

How should you compare LoCoMo and LongMemEval scores?

Compare the task construction first: what history was supplied, which questions were asked, how retrieval was configured and which model generated the final answer. Then compare whether both systems used the same context budget and whether the benchmark counted retrieval accuracy separately from answer accuracy.

A vendor can improve the headline result by changing the reader model, reranker, prompt or number of retrieved tokens. That may be a legitimate product improvement, but it is not an apples-to-apples memory-only comparison. Reproduce the run with your own controlled variables.

Why do benchmark scores not predict production performance?

Production memory has properties that generic benchmarks rarely enforce: authenticated user and tenant scopes, source-of-truth policy, correction history, deletion, data residency, rate limits, latency and tool permissions. A benchmark can reward retrieving an old fact that production policy says should already be erased.

Your vocabulary is also different. A customer’s plan, an internal project code or a repository convention may never appear in a public dataset. The benchmark tests general capability; production tests whether memory improves the actual next action without violating business rules.

• Tenant and user isolation are usually hard gates, not averages.
• Current billing or permission data must come from source systems.
• Deletion must propagate to summaries and vector indexes.
• The reader model may change independently from memory quality.
• Latency and token budgets differ by product.
• Private customer conversations cannot be replaced by public examples.

How do you build a production AI memory benchmark?

Create a frozen set of representative write sequences, questions and expected outcomes. Include direct recall, temporal updates, multi-session reasoning, irrelevant recall, cross-tenant attempts, deletion and a no-memory baseline. Separate retrieval metrics from final answer quality.

Use the same reader model and prompt across providers. Preserve the retrieved records for every run so an answer can be diagnosed. Treat scope violations and deletion failures as binary gates rather than blending them into an overall quality score.

for (const test of benchmarkCases) {
  await memory.add(test.events, {
    tenantId: test.tenantId,
    userId: test.userId,
  });

  const retrieved = await memory.search({
    tenantId: test.tenantId,
    userId: test.userId,
    query: test.question,
    limit: 8,
  });

  const answer = await fixedReaderModel.generate({
    question: test.question,
    memory: retrieved,
  });

  record({
    caseId: test.id,
    retrievedIds: retrieved.map((item) => item.id),
    answer,
    promptTokens: retrieved.tokenCount,
    latencyMs: retrieved.latencyMs,
  });
}

Which metrics should a memory benchmark report?

Report retrieval and product outcomes separately. Required-fact recall, precision, temporal correctness, contradiction resolution, scope violations, deletion failures, latency, tokens and task success cover the important tradeoffs without collapsing safety into one score.

Run each case multiple times when generation is nondeterministic and report sample size. Keep a no-memory and full-history baseline. Memory should improve the next action while using less context than replaying the entire history.

How should benchmark results influence a vendor decision?

Use public benchmarks to eliminate implausible options, then replay your own cases. A lower benchmark score can still be the better product if it offers reliable export, stronger isolation, predictable cost or better integration with your runtime.

Ask for methodology, dataset version, prompts, model versions, retrieval budget and raw per-category results. Re-run the important claims where licenses and tooling permit. Keep the decision focused on production behavior rather than marketing rank.

How can AI memory benchmark claims mislead buyers?

A headline score can hide which variables changed. A vendor may use a stronger reader model, retrieve more tokens, tune prompts to the dataset or omit failed runs. If the comparison is not controlled, the result may demonstrate product quality overall while saying little about the memory layer itself.

Another problem is selective ability reporting. A system can perform well on direct recall but fail temporal updates, abstention or multi-session aggregation. One aggregate number makes those failures invisible. Ask for per-category results, raw methodology and confidence intervals when the sample is small.

• Different models or prompts were used across systems.
• Retrieval budgets were unequal.
• Benchmark examples leaked into training or tuning.
• Failed runs and unsupported questions were omitted.
• Safety and deletion tests were left outside the score.
• One aggregate number hid category-level regressions.

What is the right way to evaluate AI memory?

Use LoCoMo and LongMemEval as screening tests for long-term conversational abilities. Then evaluate your own product with fixed cases, equal context budgets and binary safety gates. Combine public evidence with scope, correction, deletion, latency, portability and cost.