Rather than trying to decode the billions of parameters inside AI systems, researchers suggest we should treat them like conversation partners, testing their consistency and challenging their answers the way we do with other people. This shift in thinking could reshape how organizations build trust in artificial intelligence without waiting for scientists to fully understand how these systems actually work.

Why Opening the AI Black Box Might Be Impossible?

The challenge of understanding how large language models (LLMs) work has become increasingly urgent as these systems grow more powerful and more widely deployed. Mechanistic interpretability, the research field focused on mapping out exactly what happens inside neural networks, has produced genuine insights into AI behavior. However, this approach faces a fundamental problem: the function of a modern deep neural network is not concentrated in neat, human-readable modules that researchers can easily inspect.

Instead, the computation is distributed across vast stretches of the network, with complex interactions among billions of parameters allowing the system to combine patterns and exceptions in ways that resist straightforward inspection. That distributed structure is precisely what makes these systems so powerful, but it also makes them nearly impossible to fully decode through traditional analysis alone.

What Can Human Interaction Teach Us About Trusting AI?

A more practical approach may come from an unexpected source: how humans interact with each other. Nick Chater, a professor of behavioral science at Warwick Business School, notes that humans face a similar opacity problem with one another. Each person carries around a brain with roughly 100 billion neurons and in the order of 100 trillion synaptic connections, yet we cannot inspect one another's neural circuitry. Despite this fundamental mystery, humans manage to interact successfully, explain each other's behavior, predict responses, and coordinate socially well enough to sustain complex societies.

"Explanation is something that happens between agents, not something extracted directly from neural tissue," explained Chater.

Nick Chater, Professor of Behavioral Science at Warwick Business School

The parallel has real limits. Humans trust other humans partly because they share a common cognitive architecture, evolutionary history, and embodied experience. When a person offers a justification for their behavior, we can reasonably assume it reflects something about their actual reasoning process. With current AI systems, that assumption is far less secure. When an LLM produces a justification, it may be generating text that sounds like a faithful account of its reasoning but bears no reliable relationship to the computational process that generated the original output.

How to Interact With AI Systems More Effectively

However, decades of research in psychology suggest that human introspection and AI explanations may not be as different as they first appear. Humans are prolific confabulators, meaning our brains unconsciously fill gaps in our memories with fabricated or distorted information. The justifications people offer for their actions are often plausible-sounding narratives constructed after the fact, because they have no conscious access to the cognitive process that drove their behavior.

What makes human interaction work despite this pervasive confabulation is not that we take people's justifications at face value. Instead, we test them through structured probing and cross-examination. Interactive explainability applies these same insights to AI systems by treating them as partners in dialogue rather than demanding full transparency of internal mechanisms.

• Counterfactual Probing: Ask the AI what it would say if a key assumption changed, testing whether its reasoning adapts logically to new conditions.
• Consistency Checks: Evaluate whether the system maintains coherent positions across related responses and over time, catching contradictions that might indicate unreliability.
• Cross-Examination: Challenge the AI directly, pointing out when it contradicts itself or when its stated beliefs conflict with its recommendations.
• Extended Interaction: Assess coherence and performance across multiple queries and conversations, not just single isolated requests.
• Justification Requests: Ask the AI to explain its reasoning for specific words and actions, then evaluate whether those explanations hold up under scrutiny.

In this view, understanding is not a static property of a model's architecture. Instead, it is an emergent property of structured interaction. A system that is coherent and consistent in dialogue is not necessarily trustworthy, however. It may actively be attempting to deceive. AI safety researchers have long interrogated AI systems using methods such as adversarial red teaming, a form of stress-test which attempts to lure the AI into forbidden or inappropriate behavior.

What Does This Mean for Building Trust in AI Systems?

Interactive explainability adds the recognition that this kind of structured probing should not be confined to pre-deployment safety testing. Instead, it should be a continuous feature of how organizations engage with AI systems. This does not mean that internal analysis of AI is without value. If the motivating concern is the safety of increasingly powerful systems, then understanding internal mechanisms still matters a great deal for safety, debugging, and scientific understanding.

But for practical trust, what matters most may be something more familiar: whether the system behaves coherently across contexts, responds to challenges, corrects itself when prompted, and integrates feedback over time. Human social life already runs on this principle. We trust people not because we understand their neurons, but because they can justify themselves, respond to criticism, and maintain consistency across situations.

The core shift is this: explanation need not require the unveiling of hidden circuitry. It is the achievement of mutual intelligibility through interaction, the approach that has sustained human social coordination for as long as humans have existed. If AI systems can do something analogous, and if organizations can build infrastructure that better ensures that they do, then black-box opacity may not be as severe a problem as it first appears.