Financial regulators worldwide are cracking down on the use of opaque artificial intelligence systems in banking and capital markets, demanding that firms deploy AI they can actually understand and explain. A new supervisory toolkit from the International Organization of Securities Commissions (IOSCO) reveals that explainability, interpretability, and transparency are no longer optional features for AI in finance; they are regulatory expectations that firms must meet to avoid enforcement action.

What Are Regulators Most Worried About With AI in Finance?

The complexity and opacity of modern AI systems pose a significant challenge for financial supervisors. Generative AI (GenAI) systems, which power many new financial applications, are particularly difficult to predict and evaluate. According to IOSCO's analysis, the non-deterministic nature and technological complexity of GenAI systems make them hard to predict, evaluate, understand, explain, and test. One of the most pressing concerns is hallucination risk, where AI systems generate outputs that sound plausible but are factually incorrect.

Beyond hallucinations, regulators are concerned about several interconnected risks that could destabilize markets or harm investors:

• Concentration Risk: When multiple financial firms rely on the same third-party AI provider, a failure in that shared infrastructure could trigger cascading effects across the entire financial system.
• Automation Bias: Human overseers may over-rely on AI recommendations and fail to conduct adequate checks, reducing the effectiveness of human oversight as a safeguard.
• Autonomous Capabilities: Frontier AI models are developing the ability to independently discover, chain, and exploit security vulnerabilities, requiring firms to implement continuous risk assessment and shorter remediation cycles.
• Data Quality Issues: AI systems trained on historical data may not perform well when processing real-time updates that differ from training data, leading to unexpected failures.

How Should Financial Firms Implement Interpretable AI Systems?

IOSCO's supervisory toolkit provides a framework that firms should use to design and deploy AI responsibly. The approach centers on three key risk factors that regulators will evaluate when assessing a firm's AI use.

• Assess System Complexity: Evaluate the nature and complexity of the AI system in question. More complex or difficult-to-interpret systems limit the ability to identify unintended behavior and diagnose root causes of incidents. Firms should prioritize simpler, more explainable models where possible, or implement additional safeguards for complex systems.
• Define Human Oversight Levels: Establish clear governance around how humans interact with AI recommendations. Options range from human-in-control (where humans decide whether to use AI output) to human-in-the-loop (where humans must approve before AI acts) to human-out-of-the-loop (where AI acts autonomously). Regulators expect firms to justify their choice based on risk level.
• Measure Potential Harm: Quantify the severity of harm that could result from an AI failure, including impacts on individual clients, the broader market, and the firm's operations. Larger, systemically important institutions face heightened expectations even for medium or low-risk AI applications.

Beyond these three factors, IOSCO expects firms to demonstrate robust governance and risk management frameworks that include adequate testing, maintenance, and monitoring capabilities. Data quality measures are essential, as are appropriate levels of transparency for key stakeholders about how AI is being used.

What Technical Safeguards Can Reduce Hallucination Risk?

Regulators acknowledge that hallucination risk cannot be eliminated entirely, but several technical approaches can reduce it. These include Retrieval-Augmented Generation (RAG), which grounds AI responses in factual documents; Chain of Verification (CoVe), which has the AI verify its own outputs; and Multi-Agent Debate, where multiple AI systems discuss and challenge each other's conclusions. However, IOSCO notes that these techniques do not fully eliminate hallucination risk, and firms remain responsible for the accuracy of financial products and services they provide, regardless of the technology used.

The challenge is that human overseers, while essential, are not a perfect solution. Humans can suffer from automation bias, may tend to over-rely on AI, and may lack the technical knowledge to adequately oversee certain processes. Regulators therefore expect firms to ensure that human overseers have the requisite capabilities and knowledge of the system's design and limitations, and that there is clear accountability for human oversight decisions.

Why Does Interpretability Matter More Than Raw Accuracy?

Interpretability and explainability are now core regulatory principles because they enable both human oversight and regulatory supervision. A highly accurate AI system that no one can understand is riskier than a slightly less accurate system that can be audited and explained. This shift reflects a broader recognition that in financial services, trust and transparency are as important as performance.

IOSCO's framework emphasizes four key principles for AI deployment in capital markets: proportionality, accountability, transparency, and reliability. Firms must ensure that their AI systems are deployed in a manner consistent with fairness, explainability, transparency, interpretability, and operational resilience. This means documenting how AI systems make decisions, maintaining clear records of AI use, and being prepared to explain those decisions to regulators and clients.

The regulatory message is clear: the era of deploying black-box AI systems in finance is ending. Firms that invest now in interpretable AI, robust governance, and human oversight will be better positioned to meet evolving regulatory expectations and avoid enforcement action. Those that delay risk facing heightened scrutiny and potential penalties as regulators worldwide implement these supervisory standards.