The Open AI Paradox: Why Keeping Models Secret Might Not Keep Us Safe
The battle between open and closed artificial intelligence models has become a proxy war over safety, national security, and innovation itself, but economists say both sides may be asking the wrong question. Rather than debating whether openness is inherently dangerous, the real issue is how diffusion of AI technology shapes where capabilities develop and who can defend against misuse.
Why Are Tech Leaders So Divided on Open AI Models?
The tension between open-weights and closed AI models has intensified as companies like Anthropic and OpenAI warn that competitors can reverse-engineer their models through "distillation attacks," where rival labs use a company's public outputs to train their own systems. Anthropic has even asked Congress to take action against what it calls brazen intellectual property theft by Chinese companies.
The closed-model camp argues that without strict gatekeeping, the United States will lose its competitive edge and funding advantage. If Chinese labs can replicate months of expensive training work in weeks at a fraction of the cost, they reason, American companies won't be able to afford the next generation of models. Beyond economics, Anthropic's leadership has made a safety argument: once AI systems cross a certain intelligence threshold, open-weights models become impossible to recall or control, creating existential risk.
Open-weights advocates counter that concentrated AI power is itself a danger. They argue that without open models, the market for artificial intelligence would become dangerously concentrated among a few labs, and that closed systems create a false sense of security. Security researchers regularly jailbreak protections on closed models anyway, they note, and Anthropic and OpenAI's own systems have already been exploited by hackers to breach government infrastructure.
What Does Economic History Tell Us About This Debate?
Economist Petra Moser studied the Great Exhibition of 1851, when thousands of inventions from around the world were displayed in London's Crystal Palace for anyone to examine and reverse-engineer. Nations sent their brightest engineers to study rival technologies, yet Moser's analysis of nearly 15,000 inventions from that fair and its 1876 American sequel revealed something surprising: patent protection didn't increase the total amount of innovation. Instead, it simply redirected where inventors focused their efforts.
Switzerland, which had no patent system at the time, saw innovators concentrate in fields like scientific instruments and food production, where trade secrets and first-mover advantages provided protection. Countries with strong intellectual property laws saw innovation spread more widely across industries. The total amount of innovation remained constant; only its distribution changed.
Catalini argues this historical pattern applies directly to today's AI debate.
Closed labs may continue pushing the frontier of general artificial intelligence, while open-weights models enable experimentation across companies and industries that can combine machine intelligence with proprietary data, distribution networks, and specialized knowledge."Open weights are unlikely to change the level of investment in AI, only its direction: what gets built, who builds it, and who captures the returns," he explained.
Christian Catalini, Economist
How Can Companies Actually Prevent Model Distillation?
The practical challenge facing Anthropic and OpenAI reveals why enforcement may be nearly impossible. Distillation doesn't involve stealing secret model weights; instead, attackers simply query a public model repeatedly, using its outputs to train competing systems. Each individual request looks like legitimate customer use, and organized operations can scatter their volume across farmed accounts, aggregators, and different jurisdictions.
Anthropic has attempted to restrict customers from using model outputs to train competing systems, but this creates a business problem. The company's restrictions on assisting with frontier AI research frustrated users enough that Anthropic had to adjust them within days. Anti-fraud controls aggressive enough to meaningfully prevent distillation would inevitably harm legitimate work and push power users toward open-weights alternatives.
There's also an irony at the heart of the closed-model argument. The top AI labs themselves rely heavily on distillation and fair use arguments to justify training their models on massive amounts of internet material, media, and books. Singling out distillation as illegitimate while depending on it for their own progress puts these companies in a difficult position.
What Are the Real Safety Implications of Open Versus Closed Models?
The safety debate becomes more nuanced when examined through an economic lens. Rather than asking whether openness is dangerous in the abstract, the relevant question is when diffusion strengthens defenders and where harmful capabilities can actually be constrained.
Consider the practical implications of each approach:
- Concentrated Control: Closed models may create an illusion of safety by limiting who can access frontier capabilities, but they also concentrate power in a few organizations and may not prevent misuse by well-resourced actors or insiders.
- Distributed Oversight: Open-weights models distribute the responsibility for safety across many organizations and researchers, potentially enabling faster identification of vulnerabilities and broader scrutiny of how systems behave.
- Recall and Containment: Closed models theoretically allow companies to update or restrict access if problems emerge, but open-weights models cannot be recalled once released, creating irreversible risk if dangerous capabilities are discovered later.
Both sides honestly believe their approach is the only path to genuine safety. But Catalini's analysis suggests the economics doesn't favor either position absolutely. Instead, the outcome depends on specific circumstances: what capabilities are being developed, who has the resources to build them, and where the greatest risks actually lie.
Why Does This Matter for AI's Future?
The open versus closed debate will shape not just which companies succeed, but how artificial intelligence develops globally. If the United States pursues aggressive enforcement against distillation, it may slow Chinese progress but also reduce the competitive pressure that drives innovation. If it embraces open-weights, it may accelerate global AI development but lose some ability to control where capabilities concentrate.
The historical lesson from the Great Exhibition suggests that neither pure openness nor pure closure is optimal. Instead, the question is how to structure incentives so that innovation happens where it's most valuable, safety oversight is distributed appropriately, and no single actor gains unchecked power over transformative technology. That requires moving beyond the current binary debate and asking harder questions about what specific risks we're trying to prevent and whether the proposed solutions actually address them.