Canada's Privacy Commissioner has determined that xAI's Grok chatbot and its parent company X Corp. violated federal privacy law by launching an AI image-generation tool without sufficient safeguards to prevent the creation and distribution of non-consensual sexualized deepfakes. The investigation, initiated in January 2026, found that the system enabled the generation of millions of harmful images targeting women and children worldwide.

What Exactly Did Grok's Image Tool Allow?

According to the report released on June 11, 2026, Grok's image-generation feature was deployed without proper risk assessment or technical controls to prevent exploitation. Users globally were able to create and circulate sexualized deepfakes without consent, raising significant concerns about privacy violations and online harm. The Office of the Privacy Commissioner of Canada said the case highlights serious privacy risks linked to emerging AI technologies and the need for stronger regulatory safeguards at the design stage of such tools.

The investigation concluded that both X Corp. and xAI breached Canada's federal private-sector privacy law, marking one of the most high-profile regulatory actions yet involving generative AI systems. This represents a significant moment in AI regulation, as it demonstrates that even well-funded companies backed by prominent figures like Elon Musk are not exempt from privacy enforcement.

How Are xAI and X Corp. Being Required to Comply?

• Detection Systems: Following public scrutiny and during the investigation, the companies introduced additional safeguards, including detection systems aimed at identifying and removing abusive content from the platform.
• Usage Restrictions: New restrictions were implemented to reduce misuse of the image-generation tool and prevent further creation of non-consensual deepfakes.
• Regular Reporting: The companies have committed to submitting regular compliance reports until issues involving sexualized deepfakes are fully resolved, with quarterly reporting required to demonstrate effectiveness in preventing harm.
• Independent Audits: The Privacy Commissioner called for ongoing independent audits to verify that the safeguards are working as intended and protecting users from exploitation.

"The case underscores the urgent need to modernize Canada's federal privacy framework, arguing that current legislation does not provide sufficient enforcement powers, including the ability to issue binding orders or impose meaningful financial penalties," stated Philippe Dufresne, Privacy Commissioner of Canada.

Philippe Dufresne, Privacy Commissioner of Canada

While acknowledging the remedial steps taken by xAI and X Corp., the Privacy Commissioner emphasized that further improvements are required. The Office of the Privacy Commissioner said it will continue monitoring implementation of these measures to ensure compliance with Canadian privacy law.

What Does This Mean for AI Regulation Going Forward?

The findings add to growing global scrutiny of generative AI systems and their potential misuse in creating non-consensual synthetic content. Commissioner Dufresne has recommended introducing stronger "privacy by design" obligations, which means building privacy protections into AI systems from the earliest stages of development rather than adding them later. He also called for mandatory privacy impact assessments for high-risk AI systems before they are deployed to the public.

The Canadian government has recently introduced legislation aimed at strengthening online safety rules for AI chatbot services and social media platforms, which the commissioner welcomed as a step toward improving protections for users, particularly minors. This regulatory action against Grok signals that governments are taking a more aggressive stance on holding AI companies accountable for the harms their tools can enable, even when those harms are generated by users rather than the companies themselves.

The case demonstrates that as AI image-generation technology becomes more accessible and powerful, the responsibility falls on companies to anticipate misuse and build safeguards into their systems from day one. For Grok and xAI, the path forward involves not just fixing the current problems but proving through ongoing audits and reporting that they have fundamentally changed how they approach privacy and safety in AI development.