Generative AI has transformed from a productivity tool into a serious cybersecurity threat, enabling attackers to scale phishing campaigns, create convincing deepfakes, and automate malicious code generation at unprecedented speed. The real challenge for security teams is not AI itself, but AI misuse: the deliberate weaponization of large language models (LLMs), synthetic media, and automation to make cyber threats faster, cheaper, and harder to detect.

How Are Attackers Using Generative AI to Launch Cyberattacks?

Industry reports confirm the shift is already underway. Verizon's 2026 Data Breach Investigations Report notes that threat actors are using AI to work faster across activities such as identifying security gaps and writing malware. Microsoft's Digital Defense Report 2025 also highlights how attackers are using AI to scale phishing campaigns, automate intrusions, and misuse deepfakes or AI-generated synthetic identities to bypass verification systems.

The concern is not that AI creates entirely new categories of cybercrime. Rather, it makes existing attacks more convincing, scalable, and accessible to low-skilled attackers who previously lacked the technical expertise to launch sophisticated campaigns.

• AI-Powered Phishing: Attackers can now create polished emails, LinkedIn messages, WhatsApp texts, and business email compromise (BEC) messages that are personalized, emotionally persuasive, and context-aware by leveraging job roles, company news, leaked data, and social media activity.
• Deepfake Impersonation: AI-generated audio, video, and images enable attackers to impersonate CEOs, CFOs, HR teams, vendors, and employees, creating serious risks for finance teams, helpdesks, and identity verification workflows.
• Malicious Code Generation: Threat actors use AI to generate scripts, improve malware logic, write obfuscated code, automate reconnaissance, create phishing kits, and troubleshoot malicious payloads, significantly reducing the time required to build and scale attacks.
• Prompt Injection Attacks: Malicious inputs can manipulate AI models into ignoring instructions, revealing sensitive data, performing unintended actions, or misusing connected tools, especially when AI systems are linked to email inboxes, databases, APIs, and autonomous agents.

What Data Risks Are Employees Creating Without Realizing It?

One of the fastest-growing vulnerabilities is "shadow AI," the unsanctioned use of generative AI tools by employees without approval from security, IT, legal, privacy, or compliance teams. Employees often use AI tools to save time by pasting confidential information into public or unmanaged AI systems, exposing customer data, source code, API keys, internal policies, financial reports, legal documents, credentials, and personally identifiable information.

This data leakage creates multiple downstream risks. Attackers can use exposed information to craft more convincing phishing campaigns, identify security gaps, or develop targeted exploits. Additionally, when employees unknowingly paste sensitive data into AI systems, they may violate regulatory requirements around data protection and compliance.

How to Reduce Generative AI Security Risks in Your Organization

• Implement AI Governance Policies: Establish clear acceptable use policies for generative AI tools, including which systems employees can use, what data they can input, and how to report shadow AI usage.
• Conduct AI Red Teaming: Regularly test your organization's AI systems and defenses by simulating attacks, prompt injection attempts, and data poisoning scenarios to identify vulnerabilities before attackers do.
• Provide Practical AI Security Training: Move beyond traditional "spot the typo" phishing awareness training to teach employees how AI-generated content differs from human-written messages and how to recognize AI-assisted social engineering.
• Monitor AI System Outputs: Deploy monitoring tools to detect when AI systems produce hallucinations (fabricated information), biased decisions, or suspicious outputs that could indicate data poisoning or model manipulation.
• Include AI Risk in Compliance Frameworks: Integrate AI risk assessments into vendor due diligence, data protection impact assessments, incident response plans, and third-party risk management processes aligned with NIST's AI Risk Management Framework.

Why Is AI Model Theft Becoming a National Security Concern?

Beyond internal misuse, a new threat has emerged: the large-scale theft of proprietary AI models through a technique called adversarial distillation. The White House Office of Science and Technology Policy recently accused Moonshot AI, a Beijing-based company, of distilling Anthropic's Fable model to create its own K3 model.

"To do this they developed a sophisticated internal platform to conduct large scale distillation against U.S. models, allowing them to quickly switch between multiple methods of access to avoid detection," stated Michael Kratsios, who leads the White House Office of Science and Technology Policy.

Michael Kratsios, White House Office of Science and Technology Policy

Model distillation is a technique where developers extract the core capabilities of a larger AI model to create a smaller, more efficient version. While legitimate distillation plays a vital role in AI innovation, covert industrial distillation aimed at stealing proprietary U.S. technology represents a significant national security risk.

The concern is particularly acute because when an AI model has learned to reason through software weaknesses, security gaps, and attack paths, copying its behavior also copies that analytical capability. Piyush Sharma, CEO of Tuskira, an AI cybersecurity detection and response company, noted that Anthropic earlier this year accused Chinese company Alibaba of distilling their Claude AI model using 25,000 fraudulent accounts to run 28.8 million interactions over six weeks.

"When a model has learned to reason through software weaknesses, security gaps, and attack paths, copying its behavior also copies that analytical capability," explained Piyush Sharma, CEO of Tuskira.

Piyush Sharma, CEO at Tuskira

In response, the U.S. House Homeland Security Committee and the Select Committee on China announced a joint investigation into the integration of Chinese AI models, examining "a pattern of conduct by Chinese-based AI laboratories involving the large-scale theft of proprietary capabilities from American frontier AI systems through adversarial distillation".

What Should Security Teams Prioritize Right Now?

For security teams, generative AI misuse has become a board-level, security operations center (SOC)-level, and governance-level concern. The challenge is that traditional defenses were designed to catch obvious threats like typos in phishing emails or suspicious file attachments. AI-generated content is often indistinguishable from legitimate communication, making detection significantly harder.

Organizations must move beyond reactive incident response and adopt proactive AI governance. This includes mapping where generative AI is being used across the organization, establishing clear data handling policies, training employees to recognize AI-assisted social engineering, and integrating AI risk into compliance and audit frameworks. The organizations that act now will have a significant advantage over those that wait until AI-powered attacks become unavoidable.