AI agents deployed to observe and document human conversations are hitting a major wall: they cannot reliably remember what they hear, see, or reason about across multiple participants and sessions. Researchers at the University of Toronto and collaborating institutions have built the first comprehensive benchmark to measure this problem, revealing that even advanced large language models (LLMs) fall short when asked to track information across multimodal conversations involving multiple people.

What Makes Human-to-Human Conversations So Hard for AI?

Unlike traditional chatbots that interact one-on-one with users, a new class of AI agents is being deployed as silent observers in real-world settings. These systems need to watch meetings, doctor-patient conversations, and group discussions, then answer questions about what happened. The challenge is far more complex than it sounds. Human conversations are messy: people reference things said earlier using pronouns like "he" or "that," they share photos and documents, multiple voices contribute conflicting information, and important details emerge across different sessions over time.

Existing benchmarks for AI memory focus almost entirely on single-user, text-only interactions. They miss the multimodal chaos of real human communication. To fill this gap, researchers introduced H2HMem, a Human-to-Human Multimodal Memory Benchmark that tests AI agents on three critical dimensions: memory recall, reasoning, and application.

How Are Researchers Testing AI Memory?

The team built a large-scale dataset using a privacy-preserving approach. Rather than recording actual conversations (which raises serious privacy concerns), they used LLMs to generate realistic multimodal, multi-session dialogues involving two or more participants. The benchmark then evaluates agents across several specific tasks:

• Memory Recall: Can the agent retrieve specific facts from conversations and resolve conflicting information across sessions?
• Memory Reasoning: Can the agent infer causal relationships, track how knowledge evolves, and understand temporal sequences?
• Memory Application: Can the agent learn new information during a conversation, detect when participants contradict each other, and refuse to answer when it lacks sufficient information?

When researchers tested advanced multimodal LLMs (models that process both text and images) on these tasks, the results exposed substantial limitations. Agents struggled to align information across different modalities, failed at structured reasoning about complex relationships, and could not reliably apply memories in dynamic settings.

Why This Matters for Real-World AI Deployment

The stakes are high. Clinical documentation systems already use AI to generate patient notes from doctor-patient conversations. Meeting assistants powered by AI are being integrated into platforms like Zoom. These systems need to track who said what, remember context across multiple visits or meetings, and reason about relationships between facts. If they fail at memory and reasoning, they risk missing critical information, generating inaccurate summaries, or making dangerous errors in medical settings.

The research reveals that current memory mechanisms for LLM agents fall into three categories, each with limitations. Some models simply extend the conversation history fed into the model, but this approach becomes computationally expensive and loses information over long interactions. Others use retrieval-augmented generation (RAG), maintaining an external memory store, but these systems excel at factual recall and struggle with understanding cause-and-effect relationships. A third approach uses specialized memory modules with explicit operations like writing and forgetting, but these are primarily designed and tested in human-assistant settings, not human-to-human scenarios.

What Steps Can Organizations Take to Improve AI Memory Systems?

• Evaluate Against Realistic Benchmarks: Organizations deploying AI agents in human-to-human settings should test systems against multimodal, multi-participant benchmarks like H2HMem rather than relying on single-user benchmarks that do not capture real-world complexity.
• Design for Cross-Modal Integration: Build memory systems that explicitly handle information from multiple sources (text, images, audio) and can align facts across modalities without losing coherence.
• Test Reasoning and Application, Not Just Recall: Memory systems must be evaluated on their ability to infer relationships, track how information changes over time, and apply memories in dynamic contexts, not merely retrieve isolated facts.

The H2HMem benchmark represents a significant step toward understanding where current AI agents fall short. By systematically evaluating memory across recall, reasoning, and application, researchers have created a framework that exposes the gap between laboratory performance and real-world deployment requirements. As AI agents move from research settings into hospitals, boardrooms, and clinics, this kind of rigorous evaluation becomes essential.

The research team emphasized that substantial room for improvement exists in next-generation LLM agents. The findings suggest that future progress will require not just larger models or more training data, but fundamentally better approaches to how AI systems construct, retain, and utilize memories across the complexity of human interaction.