A frontier artificial intelligence company should voluntarily shut down and publicly announce that it cannot safely build powerful AI systems, according to a detailed argument from AI safety researcher Michael Dickens. The proposal, published on LessWrong, challenges the prevailing assumption among AI developers that unilateral safety efforts are pointless because competitors will simply continue building dangerous technology regardless.

Why Would a Company Voluntarily Shut Down?

Dickens argues that if a multi-billion-dollar AI company announced it was closing operations specifically because the risks of artificial general intelligence (AGI), or AI systems matching human-level intelligence across all domains, were too severe to manage safely, the signal would be impossible for policymakers to ignore. "Shutting down would make people say, holy shit, they are serious about this extinction risk thing," Dickens explained in his analysis. The move would demonstrate that coordination among AI companies might be possible, contradicting the common industry refrain that competitive pressures make safety compromises inevitable.

Dickens

The argument hinges on a simple premise: if the most safety-conscious company in the space actually stopped building frontier models rather than continuing to race ahead with incremental safety research, it would send a credible signal that the risks are genuine and that the industry's current trajectory is unsustainable. This could encourage other companies to take safety more seriously and prompt governments to implement stronger AI governance frameworks.

What Are the Main Objections to This Idea?

Critics raise several practical and legal concerns about such a shutdown. Some worry that if the most safety-minded company exits the field, valuable safety research would be lost. Others question whether a unilateral shutdown would actually change industry behavior or government policy, or whether it would simply be dismissed as a competitive failure. There are also investor protection concerns; shareholders might sue a for-profit company for abandoning its business model, though Dickens notes that some AI companies, like Anthropic, which operates as a public benefit corporation, may have legal protections for such decisions.

Dickens acknowledges these counterarguments but proposes an alternative: rather than a complete shutdown, a frontier AI company could reallocate its entire budget toward safety research and global coordination efforts aimed at making AI development safer across the industry. This approach would preserve the company's ability to conduct research while signaling a fundamental shift in priorities away from competitive model development.

How Could Companies Signal Commitment to AI Safety?

• Voluntary Shutdown: A frontier AI company could close operations entirely and announce that existential risks from advanced AI are too severe to responsibly continue building frontier models, sending an unmistakable signal to governments and competitors.
• Budget Reallocation: Instead of shutting down completely, a company could redirect all resources toward safety research and international coordination efforts, abandoning competitive model development until alignment problems are solved.
• Governance Protections: AI companies planning to go public could establish shareholder protections similar to those used in other industries, allowing leadership to prioritize safety over profit maximization if existential risks become apparent.
• Schelling Point Creation: A unilateral safety move by one company could establish a coordination focal point that other companies could rally around, breaking the prisoner's dilemma dynamic that currently pressures all firms to race ahead.

The core tension Dickens identifies is that AI companies themselves admit competitive pressures prevent them from slowing down, yet alignment of advanced AI systems may require exactly that kind of deceleration. "By AI companies' own admission, competitive pressures don't allow them to slow down. Why would things change in the future?" he asked. If alignment is genuinely difficult, as many researchers believe, then the current strategy of racing ahead while conducting safety research on the side may be fundamentally insufficient.

Dickens also challenges the assumption that safety research requires access to frontier models. He notes that many types of AI safety research do not require the latest models, and companies could conduct substantial work on existing systems or negotiate access to competitors' models for research purposes.

What Conditions Would Justify Such a Drastic Move?

Dickens raises a critical question that remains unanswered: under what specific conditions should a safety-minded AI company actually shut down? Without clear thresholds or decision criteria, the proposal remains theoretical. He suggests that companies should establish these conditions in advance, before they face pressure to make such decisions, and commit to following through if those conditions are met. This would require a level of governance foresight that most AI companies have not yet demonstrated.

The debate reflects a deeper disagreement about how to manage existential risks from advanced AI. Some researchers and safety advocates believe that market competition and incremental safety improvements are insufficient to address the scale of potential harms. Others argue that continued development, combined with robust safety research, is the best path forward. Dickens' proposal sits at the extreme end of the safety-focused spectrum, but it forces a confrontation with uncomfortable questions about whether the current industry structure can actually solve the alignment problem before building systems that pose existential risks to humanity.