AI Is Now Weaponizing Its Own Hallucinations Against Software Supply Chains
Attackers are exploiting a fundamental weakness in how AI systems work: their tendency to generate plausible-sounding but entirely fictional web addresses that defenders have never seen before. Security researchers at Palo Alto Networks Unit 42 discovered that large language models (LLMs), which are AI systems trained on vast amounts of text data, consistently hallucinate web domains for legitimate brands. Adversaries are now actively registering these nonexistent domains before defenders even know they exist, creating what researchers call "phantom squatting".
This represents a fundamentally new attack vector that existing security tools were never designed to catch. Unlike traditional phishing or malware distribution, phantom squatting exploits the gap between what an AI model generates and what actually exists in the real world.
How Does This Attack Actually Work?
The attack unfolds in a predictable sequence. First, adversaries systematically query AI models to map out which fake domains they tend to generate for target companies. Then, they register the most valuable of these hallucinated domains before defenders even realize they're at risk. When developers or AI systems later consult those same AI models for information, they receive URLs that point to attacker-controlled servers instead of legitimate services.
The speed advantage is staggering. In one real-world case, researchers identified a phishing kit called Montana Empire that targeted a hallucinated domain they had predicted 23 days before the attacker even registered it. The attacker had staged the entire phishing infrastructure in advance, waiting for the domain registration window to open.
Why Traditional Security Defenses Miss This Threat?
Conventional cybersecurity tools rely on reputation scoring and threat intelligence databases. When a domain is first registered, it has no history of malicious activity, no blocklist entries, and no accumulated security signals. By the time threat intelligence systems catch up, users have already been directed to the malicious site by an AI system they trusted.
Palo Alto Networks researchers analyzed 913 global brands and executed over 685,000 URL queries across different AI model configurations. They discovered approximately 250,000 hallucinated domains that remain unregistered, representing a massive opportunity for adversaries to exploit the software supply chain through preemptive registration.
Steps Organizations Can Take to Defend Against Phantom Squatting
- Monitor AI-Generated Outputs: Track and validate any URLs, API endpoints, or service recommendations that AI systems generate before developers integrate them into production code or CI/CD pipelines.
- Implement Domain Verification Protocols: Require independent verification of any third-party service endpoints recommended by AI coding assistants, rather than trusting the model's output directly.
- Establish Hallucination Detection Systems: Deploy monitoring systems that can predict which domains a target brand's AI models are likely to hallucinate, then preemptively register or monitor those domains before attackers do.
The research team was able to predict use of high-risk hallucinated domains 18 to 51 days ahead of actual adversary registration, demonstrating that proactive detection is possible.
What Makes This Attack Different From Previous Supply Chain Threats?
For decades, supply chain attacks targeted predictable artifacts like tampered build tools, malicious dependencies, and compromised update servers. Defenders built protections around these known attack surfaces using package integrity checks, signed binaries, and dependency auditing tools. But phantom squatting operates on a different principle entirely.
LLMs are no longer peripheral utilities in software development; they are active participants in the development lifecycle. Developers consult AI coding assistants for documentation links, and those assistants perform autonomous web research on behalf of developers, then formulate and execute HTTP requests against URLs the models themselves generate. Enterprise CI/CD pipelines, which automatically build and deploy software, now integrate AI assistants that recommend third-party service endpoints. This fundamentally alters the attack surface.
"The software supply chain threat landscape is shifting," the Unit 42 research team noted. "LLMs are no longer peripheral utilities, they are active participants in the software development lifecycle."
Palo Alto Networks Unit 42 Research Team
The broader context makes this threat even more urgent. Agentic AI, which refers to AI systems that can take autonomous actions to accomplish goals, is accelerating attack timelines across the entire cybersecurity landscape. According to Palo Alto Networks Unit 42, the average time through the cyber kill chain, from discovery to exploitation of a weakness, has moved from weeks to hours. AI-assisted workflows remove the human capital constraint, allowing attackers to seek out vulnerabilities and work through attack sequences across hundreds of targets in parallel.
The effects are measurable. IBM X-Force research demonstrated that AI can generate highly convincing phishing emails in five minutes compared to the 16 hours typically required by experienced human operators, a 192-fold improvement in efficiency. According to DarkTrace, AI-powered phishing attacks are 67 percent more successful than traditional ones.
What Are Security Teams Doing to Catch Up?
Security experts recognize that traditional human-led defense is no longer viable at the scale and speed of AI-driven attacks. The volume and velocity of threats have outpaced the cognitive capacity of even elite security operations center teams. This has led to a critical realization: defenders must deploy agentic AI themselves to remove the human bottleneck and handle the heavy lifting.
The defensive strategy centers on three key areas. First, architectural resilience means building new systems from the ground up with significantly fewer software defects and inherent resistance against long-established security risks like access control failures and memory errors. Second, autonomous testing and hardening involves identifying and patching weaknesses with new thoroughness before a system is deployed. Third, continuous adaptive monitoring means training defensive AI to meet adversarial agents at hyper-speed, blow-for-blow, at the tactical edge.
However, experts acknowledge this is a multiyear catch-up effort at best. Kevin Mandia, founder and CEO of AI security company Armadin, stated at the 2026 RSA Conference that "It's a perfect storm for offense over the next year or two" as defenders try to catch up. "The scale and scope and total recall of an AI agent compromising you and swarming you is not humanly comprehensible," he explained.
The challenge is not just technical; it is organizational. Resistance to change, bureaucratic inertia, and the need to focus on reliable autonomous agents to write code represent the true vulnerabilities. The threat is artificial, but the solution requires collective human will to evolve.