Financial regulators worldwide are demanding that banks implement strict safeguards against AI failures, including hallucinations and algorithmic bias, according to a new supervisory framework released by the International Organization of Securities Commissions (IOSCO). The toolkit provides the first comprehensive blueprint for how regulators will assess AI risk in capital markets, signaling that firms can no longer treat artificial intelligence as a black box technology.

What Are Regulators Most Concerned About With AI in Finance?

IOSCO's "Supervisory Toolkit for AI Use in Capital Markets" reveals that regulators are worried about several interconnected risks. The most pressing concern is the opacity of complex AI systems, particularly generative AI models that can produce convincing but entirely false outputs, known as hallucinations. Because these systems operate in non-deterministic ways, meaning they don't follow predictable logical paths, they become difficult to test, evaluate, and explain.

Beyond hallucinations, regulators are tracking concentration risk, where multiple financial firms rely on the same third-party AI providers. If a shared AI system fails at a systemically important institution, the damage could cascade across the entire market. Regulators are also concerned about data quality issues and algorithmic bias, which can lead to discriminatory lending decisions or unfair investment advice without anyone noticing.

The speed of AI evolution itself poses a regulatory challenge. Frontier AI models are developing autonomous capabilities, including the ability to independently discover and exploit security vulnerabilities. This rapid advancement means regulators must conduct continuous risk assessments rather than one-time audits.

How Should Banks Structure Human Oversight of AI Systems?

IOSCO identifies four distinct levels of human involvement in AI decision-making, each with different risk profiles. Understanding these frameworks is critical for banks seeking to satisfy regulatory expectations and protect customers.

• Human-in-Control: The AI system cannot act independently. A human must actively choose whether to use the AI's recommendation or ignore it entirely. This model provides maximum oversight but requires significant human resources.
• Human-in-the-Loop: The AI system evaluates data and makes a recommendation, but a human must explicitly approve the action before it executes. This balances efficiency with accountability.
• Human-on-the-Loop: The AI system acts automatically unless a human actively disapproves. This model assumes the AI is usually correct but allows human intervention if something looks wrong.
• Human-out-of-the-Loop: The AI system operates entirely independently without human involvement. Regulators view this as the highest-risk scenario and expect it only in low-risk applications.

However, IOSCO warns that human oversight alone is not a cure-all. Humans suffer from automation bias, meaning they tend to over-rely on AI systems and fail to conduct adequate checks. Regulators now expect firms to ensure that human overseers actually understand the AI system's design, limitations, and failure modes before granting them oversight authority.

What Three-Factor Risk Assessment Framework Are Regulators Using?

IOSCO has established a structured approach to evaluating AI risk that banks should understand and prepare for. The framework considers the nature of the AI system, the level of human oversight, and the potential harm to clients and markets.

The first factor examines system complexity. More complex AI systems limit a firm's ability to identify unintended behavior or diagnose what went wrong when incidents occur. AI systems that process real-time data updates face additional risk if that data differs significantly from the training data the model learned from. Generative AI systems that produce hallucinations represent a particularly thorny challenge because their incorrect outputs can be persuasive and difficult to catch.

The second factor assesses human oversight capacity. Can a human actually detect and resolve problems before they cause real harm? This depends on the oversight model chosen and whether the human has sufficient knowledge and resources.

The third factor evaluates potential impact. Regulators consider whether the AI system could harm individual clients through discrimination or unsuitable investment advice, affect large numbers of customers, disrupt core business operations, or trigger systemic market effects. Larger, systemically important institutions face heightened expectations even for medium or low-risk AI applications because their failures have market-wide consequences.

How Can Banks Address the Hallucination Problem?

IOSCO acknowledges that hallucinations pose a genuine technical challenge without a perfect solution. Several technical approaches can reduce but not eliminate hallucination risk. Retrieval-Augmented Generation (RAG) grounds AI responses in factual documents rather than letting the model generate answers from memory alone. Chain of Verification (CoVe) asks the AI to verify its own outputs step-by-step. Multi-Agent Debate involves multiple AI systems checking each other's work.

Despite these techniques, IOSCO emphasizes that firms remain ultimately responsible for the accuracy of their AI outputs and the financial products they deliver. The primary responsibility for managing hallucination risk cannot be outsourced to technology vendors. Banks must combine technical safeguards with appropriate human oversight, robust governance frameworks, and clear disclosures to clients about AI limitations.

What Governance and Risk Management Framework Do Regulators Expect?

Beyond specific AI safeguards, regulators are examining whether firms have adequate governance structures calibrated to their actual use cases and potential outcomes. This includes having sufficient knowledge and resources for testing, maintaining, and monitoring AI systems over time. Firms must demonstrate data quality measures, appropriate transparency for stakeholders, and accountability mechanisms when things go wrong.

Regulators will also scrutinize outsourcing arrangements and third-party dependencies. If a firm relies on external AI providers, the firm remains responsible for that provider's performance. This creates pressure for banks to conduct thorough due diligence on vendors and maintain contractual safeguards.

The regulatory focus on governance reflects a broader shift in how financial services firms must approach AI. Rather than deploying cutting-edge models and hoping for the best, firms now need comprehensive frameworks that address data quality, bias detection, cybersecurity integration, recordkeeping, and business continuity. Regulators expect firms to demonstrate that they understand their AI systems well enough to explain them to regulators, clients, and courts if necessary.

For financial services firms, the message is clear: AI adoption is not a technology decision alone. It requires governance, accountability, and transparency frameworks that satisfy regulators and protect customers from discrimination, unsuitable advice, and market-destabilizing failures.