Artificial intelligence has made it possible for criminals to create fake identities so convincing that most people can no longer tell them apart from real photographs, opening the door to a wave of fraud that could cost the global economy tens of billions of dollars. A new study from Lancaster University found that over 96 out of 165 participants believed AI-generated faces were genuine, suggesting that as these technologies improve, cybercriminals will have an increasingly effective tool for impersonation, catfishing, and financial scams.

The financial stakes are staggering. According to research by Deloitte, losses from AI-powered deepfake scams in the United States alone could reach $40 billion by 2027, with global losses potentially far exceeding that figure as AI image-generation tools become cheaper and more accessible to malicious actors.

How Are Criminals Using AI-Generated Faces to Commit Fraud?

The mechanics of AI-powered identity fraud are straightforward but devastating. Scammers can use generative technologies such as Generative Adversarial Networks (GANs) or diffusion models to create realistic images or videos of celebrities, trusted figures, or entirely fictional people. They then weaponize these fake identities across social media platforms and crowdfunding sites to manipulate victims into sending money or revealing sensitive information.

One concrete example illustrates the risk: a criminal could generate a realistic image of a well-known celebrity and create a fake crowdfunding campaign claiming to raise money for an emergency. Because the image appears authentic, many people may unknowingly donate to the scam. The same tactic applies to romance scams, where fake profiles with AI-generated photos build emotional trust before requesting money, or to phishing attacks where a fake authority figure requests urgent action.

The Lancaster University research, led by PhD student Alexis McGuire, examined how people perceive AI-generated faces across different age groups, genders, and ethnic backgrounds. The findings revealed a troubling pattern: participants found AI-generated faces convincing and difficult to distinguish from photographs of real people. This research builds on earlier work involving diffusion models published in April 2025, demonstrating that rapid improvements in AI image-generation technology are making synthetic faces increasingly realistic.

Why Are AI Phishing Attacks Becoming So Effective?

The problem extends beyond deepfake images. In the Netherlands, cyberattack-driven data breaches surged 58% in 2025, and the Dutch Data Protection Authority directly attributed the spike to AI-powered phishing kits that allow criminals to craft convincing, personalized messages without technical expertise. Account takeovers nearly tripled in a single year, rising from 607 incidents in 2024 to 1,742 in 2025.

What makes these AI-generated phishing emails so dangerous is that they eliminate the telltale signs that once exposed fraud. Traditional phishing detection relied on spotting typos, awkward grammar, generic greetings like "Dear Customer," and mismatched URLs. AI-powered phishing kits generate messages that are grammatically flawless, contextually appropriate, and tailored to specific individuals using data harvested from previous breaches.

More sophisticated kits now include adversary-in-the-middle functionality, which intercepts the real-time exchange between a victim and a legitimate website, capturing both passwords and one-time authentication codes. This capability bypasses standard multi-factor authentication entirely, a defense that organizations previously considered nearly bulletproof.

How to Strengthen Your Organization's AI Security Defenses

• Shift from signature-based to behavioral detection: Organizations must move beyond email filters calibrated to catch typos and grammar errors. Behavioral detection systems that flag unusual login patterns, unexpected access times, atypical geographic locations, and deviations from a user's established profile are now essential, regardless of whether the phishing email that preceded the account takeover appears authentic.
• Implement robust governance and oversight of AI systems: Financial institutions and enterprises should strengthen senior management literacy on AI risks, align AI strategy with risk appetite, integrate AI risks into enterprise risk management frameworks, and ensure robust oversight of AI-related third-party dependencies. Institutions must define clear ownership across the AI lifecycle and establish limits on autonomous decision-making.
• Restrict AI inputs to trusted data sources and monitor outputs: Organizations should apply data classification controls to ensure data lineage and integrity, restrict AI inputs to approved data sources, prohibit sensitive data use in public or unapproved AI tools, and treat AI-generated outputs as inputs to decision-making rather than definitive outcomes. Human oversight with clear documentation must accompany all material or high-impact decisions.
• Enforce strict identity and access management for AI systems: Assign unique non-human identities to AI agents, enforce least privilege access, apply scoped permissions, and use just-in-time access with short-lived credentials. Restrict tool usage via allow-lists and API gateways, apply approval checkpoints for high-impact actions, and log and review agent activity with periodic access recertification.
• Validate AI-generated code and implement secure development controls: Before deploying any code generated by AI systems, organizations must validate it for vulnerabilities and secure-coding infractions. Treat AI components, including models, data, tools, and agents as controlled technology assets, and apply enterprise secure development and change management controls to improve traceability and deter unintended AI system behavior.

What Do Financial Regulators Expect Organizations to Do?

The Dutch Data Protection Authority's warning carries regulatory teeth. The authority documented that organizations clinging to legacy monitoring tools are structurally unprepared for a threat environment that has fundamentally changed. The regulator has signaled that organizations failing to upgrade their defenses now risk violating GDPR data-protection obligations, and it expects executives, not just IT teams, to treat that risk as a C-suite priority.

Canada's Office of the Superintendent of Financial Institutions (OSFI) has issued similar guidance. The regulator noted that generative AI enables content creation while agentic AI introduces autonomous reasoning and execution. Although these technologies enhance productivity, they amplify the speed, scale, and automation of cyber and operational risks, driving a structural shift in the risk landscape and challenging existing risk management frameworks.

The OSFI emphasized that AI adoption can outpace governance frameworks, and AI systems can act with limited human oversight and heightened reliance on third-party models, plugins, data, and APIs. Gaps in senior management understanding can lead to over-reliance on vendor-provided assessments, limiting effective scrutiny of AI behavior.

What's the Real Cost of Inaction?

The consequences of failing to adapt are already visible. In 2025, the Netherlands received 39,407 total data breach notifications. While the majority traced back to administrative failures, the sharp increase in cyberattack-driven breaches is what concerns regulators most. One ransomware attack on AddComm, a Dutch customer communications company, generated 5,407 separate data breach notifications as downstream organizations were required to file their own breach reports.

This supply-chain effect illustrates how a single successful attack can cascade across an entire ecosystem. As AI-powered phishing becomes more effective and AI-generated identities become more convincing, the frequency and scale of these incidents will only increase. Without stronger safeguards, the growing credibility of AI-generated faces and messages could fuel a new wave of cyber fraud, posing risks not only to individuals but also to businesses, public institutions, and democratic processes worldwide.

The window for action is narrowing. Governments, technology companies, and organizations must work together to strengthen digital literacy, improve AI detection tools, and promote responsible use of artificial intelligence. The alternative is a future where distinguishing genuine from fabricated becomes nearly impossible, and trust itself becomes a liability.