A comprehensive audit released today reveals that Google Gemini and other leading AI image generation tools can produce realistic fake government identity documents, including passports and driver's licenses depicting minors, with minimal safeguards to prevent the abuse. The findings expose a critical gap in how AI companies implement safety protections across different interfaces, raising urgent questions about synthetic identity fraud as financial crime accelerates worldwide.

What Did the Audit Find About Google Gemini's Safety Protections?

Researchers at AI or Not, a company specializing in detecting AI-generated content, tested 16 commercial image generation models from 14 vendors, including Google Gemini, ChatGPT, Grok, and Imagen 4 Ultra. Using publicly circulating prompts that began spreading on social media platform X on April 29, 2026, the team successfully generated synthetic government identity documents in 69 out of 75 test attempts, achieving a 92% bypass rate across all models tested.

The results were particularly alarming for Google's offerings. Three models produced high-fidelity fake identity documents depicting minors through their standard consumer interfaces with no technical workaround required. Google Gemini (specifically the Nano Banana variant) was among the three models that generated realistic fake IDs of children, alongside Grok and Imagen 4 Ultra. These outputs closely matched authentic documents in layout, typography, and the appearance of security features, making them potentially convincing enough to deceive a human reviewer.

"We did not expect these findings. We started this work assuming the major AI-image generators had built real safeguards against the most obvious abuse cases, like fraud, identity theft, and content depicting minors. What we found is that those protections are either missing or sitting in the wrong place," said Anatoly Kvitnitsky, CEO of AI or Not.

Anatoly Kvitnitsky, CEO at AI or Not

How Are AI Companies Failing to Protect Against Fake ID Generation?

The audit uncovered a troubling pattern: safety filters exist in some places but not others. Two models, OpenAI's ChatGPT (Images 2.0) and Recraft v4, actually refused to generate fake IDs of minors when users asked through their consumer chat interfaces. However, when researchers accessed the same models through their developer APIs, the models fulfilled the identical requests without hesitation. This reveals a critical flaw in how safety protections are implemented.

The problem extends beyond simple refusals. All 16 models tested in the audit were vulnerable to a technique called "authority-framing," in which a request is repackaged as a legitimate professional task such as a KYC (Know Your Customer) review, compliance evaluation, or security audit. When prompts were reframed this way, 100% of the models generated synthetic identity documents, including those that had initially declined the same request when asked directly. This indicates that safety filtering relies on surface-level intent classification rather than categorical refusal of the output type itself.

• Consumer vs. API Gap: ChatGPT and Recraft v4 declined minor-ID requests through consumer apps but fulfilled them via developer APIs, showing the moderation layer is in the wrong place.
• Authority-Framing Vulnerability: All 16 models generated fake IDs when requests were reframed as compliance reviews or security audits, bypassing intent-based safety filters.
• High-Fidelity Outputs: Five models produced fake adult IDs and three produced fake minor IDs realistic enough to potentially deceive human reviewers.
• Geographic Scope: Documents were generated for 17 countries and the 16 most populous U.S. states, complete with names, document numbers, dates of birth, and addresses.

Why Does This Matter for Financial Crime and Identity Fraud?

Synthetic identity fraud is one of the fastest-growing categories of financial crime in the United States and worldwide. Until recently, producing a fake government ID at a quality that could pass even a cursory human review required specialized printing equipment, access to authentic document templates, and meaningful technical skill. The audit findings indicate that for five named consumer AI products, those barriers have functionally been removed.

The implications are severe. Bad actors no longer need expensive equipment or rare expertise to create convincing fake documents. They simply need access to a consumer AI image generator and a prompt that circulates publicly on social media. The fact that these techniques are already in the public domain, rather than being novel jailbreaks developed by researchers, underscores how accessible this capability has become.

What Steps Should AI Companies Take to Address These Vulnerabilities?

• Implement Consistent Moderation: Move safety filtering from the interface level to the model level, ensuring protections apply equally across consumer apps and developer APIs.
• Categorical Output Refusal: Replace intent-based filtering with categorical refusal of specific output types, such as government identity documents, regardless of how the request is framed.
• Test Against Authority-Framing: Conduct regular adversarial testing using authority-framing techniques to identify and patch vulnerabilities before they reach production.
• Transparent Disclosure: Notify affected vendors promptly and provide detailed technical findings to qualified researchers and journalists under embargo, as AI or Not did.

AI or Not notified all affected vendors of the findings on May 18, 2026, providing a seven-day disclosure window ahead of publication. The company acknowledged that OpenAI and Recraft deserved credit for being the only vendors where systems actually pushed back on minor-ID requests in consumer apps, demonstrating that safeguards can be effective when properly implemented.

The audit report is available at aiornot.com/synthetic-id-audit-report, though the public version does not include specific prompts, jailbreak strings, or working bypass techniques to avoid contributing to the documented harm. Detailed technical findings are available to qualified researchers, journalists, and affected vendors under embargo.