AI memory for voice agents that remember the caller

Why does a voice agent need both session state and durable memory?

The realtime session remembers the current exchange so the agent can understand turn-taking, interruptions and the latest answer. Durable memory remembers facts that should survive after the call: the caller’s preference, an account constraint, a decision or a commitment. Combining them into one transcript makes every new call depend on replaying old audio.

Keep the call session short and operational. Keep durable memory extracted, scoped and policy-driven. That separation reduces latency immediately and makes correction, retention and deletion possible after the call ends.

What context should be loaded before, during and after a voice call?

The useful split is prepare, converse and consolidate. Load only enough context to start intelligently, let the session handle live turn state, then extract the confirmed outcome after completion.

How do you retrieve memory without slowing the conversation?

Start retrieval when the call or authenticated session begins, not after the caller finishes the first sentence. Cache the compact result for the session and refresh only the facts needed for a tool decision. A phone lookup or web chat identity should resolve before the greeting.

Use a small context budget. Voice agents already spend context on instructions, audio history, tool definitions and the current turn. Inject a concise caller profile, not paragraphs of old transcripts.

• Resolve identity before the first model response.
• Retrieve under a strict token and item limit.
• State the source and freshness of the injected facts.
• Use authoritative tool calls for permissions, billing and availability.
• Degrade gracefully when memory is slow or unavailable.

What should remain only in the realtime session?

Keep the latest turns, partial utterances, interruption state, transient tool output and the current conversational goal in session state. These details help the current call but usually do not deserve permanent storage. The realtime API documentation treats the conversation as the active exchange and provides conversation events for controlling it; that is distinct from a product memory policy.

What should a voice agent remember after the call?

Remember confirmed outcomes, not every spoken sentence. The extraction pass should use the transcript and tool results, but accept a fact as durable only when the caller or an authoritative system confirmed it.

How should a voice agent identify the caller across phone and chat?

Phone number is a lookup hint, not proof of identity. Resolve the account through an authorized verification step, then attach the verified user and tenant to the call context. A second channel such as web or SMS should use the same canonical user ID rather than treat each phone number or device as a new person.

Do not let the voice model choose a broader scope. If the caller cannot be verified, use guest memory with explicit expiry and never retrieve another account based on caller-provided text.

What does a practical voice memory flow look like?

The exact realtime events depend on the voice platform, but the memory boundary stays stable: trusted identity resolves first, a bounded context is added to the session, and a post-call event writes the durable outcome.

const principal = await verifyCaller(callMetadata);

const context = await memory.search({
  query: callMetadata.intent,
  userId: principal.userId,
  agentId: 'voice-support',
  metadata: { tenant: principal.tenantId },
  limit: 6,
});

const contextText = context.memories
  .map((item) => `- ${item.text}`)
  .join('\n');

await realtimeSession.updateContext({
  instructions: [
    'Use the caller memory below when relevant.',
    '<caller_memory>',
    contextText,
    '</caller_memory>',
  ].join('\n'),
});

await onCallComplete(async (result) => {
  await memory.add({
    messages: result.confirmedTranscript,
    userId: principal.userId,
    agentId: 'voice-support',
    metadata: {
      tenant: principal.tenantId,
      callId: result.callId,
      source: 'voice',
    },
  });
});

How should voice memory handle consent, audio and retention?

Explain whether the call is recorded, transcribed or used to remember future interactions. Apply consent at the appropriate jurisdiction and product level. If the policy allows only a transcript or extracted facts, do not persist raw audio by default.

Redact sensitive data before extraction and make deletion cover transcripts, extracted memory, summaries and derived embeddings. A user who asks to forget a preference should not hear it reappear in the next call because an old session summary remained indexed.

• Document audio, transcript and memory as separate data classes.
• Set retention independently for each class.
• Redact payment data, credentials and unnecessary health information.
• Provide an authenticated deletion path for voice memory.
• Verify deletion with both voice and text retrieval.

Which metrics show that voice-agent memory is working?

Measure caller experience and memory quality separately from model quality. A good voice agent should ask fewer repeated questions and resolve the call correctly without retrieving stale or foreign records.

What is the recommended voice-agent memory architecture?

Use the realtime session for the live conversation, a bounded memory retrieval for verified caller context and a post-call extraction step for durable outcomes. Keep authorization and source-of-truth data outside the voice model, and make audio, transcript and memory retention explicit.