The debate over whether artificial intelligence will destroy humanity has moved from academic conferences to living rooms and protest marches, but experts remain deeply divided on what the real danger actually is. Some warn of existential threats from superintelligent AI systems spiraling beyond human control, while others argue the focus on doomsday scenarios distracts from concrete harms already happening today, including environmental damage, worker exploitation, and threats to child safety.

The disagreement doesn't follow traditional political or economic lines. Instead, it reflects fundamentally different worldviews about technology, risk, and humanity's future. What unites many of these camps is a sense of urgency, even if they disagree sharply on what needs to happen next.

What Do AI Safety Experts Fear Most?

For some researchers, the concern centers on a specific technical scenario: autonomous AI agents that can build smarter versions of themselves in a recursive loop that spirals beyond human control.

"Our biggest fear is that, in the very near future, one of these companies will set up an autonomous agent that is capable of developing a smarter agent that is capable of developing an even smarter agent, and the whole thing will spiral out of control," stated Malo Bourgon, CEO of the Machine Intelligence Research Institute.

Malo Bourgon, CEO of the Machine Intelligence Research Institute

Bourgon emphasized that once artificial general intelligence (AGI), or AI systems matching human-level reasoning across all domains, becomes significantly smarter than humans, "the game is essentially over".

This perspective reflects a view held by some in the effective altruism movement, a philosophy focused on using evidence and reason to help others as much as possible. These researchers argue that companies building advanced AI systems understand the danger but feel trapped in a competitive race where stopping development would only allow less-responsible competitors to surge ahead.

However, other experts reject the existential risk framing entirely. Shreeharsh Kelkar, an interpretive social scientist at UC Berkeley, noted that while accidents involving complex systems do happen, the idea that AI will kill all of humanity is exaggerated. "Humanity is pretty tough," Kelkar observed. "We've survived worse, including two world wars, plagues, and much else".

Are Current AI Harms Being Overlooked?

A growing chorus of researchers and advocates argue that focusing on speculative existential risks obscures the immediate, measurable harms of AI systems already deployed in the world.

"These kinds of conversations about an AI 'kill switch' or existential risk position all of the harms of AI in some speculative future, and it distracts from the harms of AI that are currently happening," explained Dylan Baker, lead research engineer at the Distributed AI Research Institute.

Dylan Baker, lead research engineer at the Distributed AI Research Institute

Baker pointed to a range of documented problems that deserve immediate regulatory attention:

• Environmental Impact: Data centers powering AI systems consume enormous amounts of electricity and water, contributing to climate change and resource depletion.
• Labor Harms: Workers training AI models often face low wages, poor working conditions, and exposure to traumatic content without adequate support.
• Child and Teen Safety: AI systems are being deployed in ways that threaten young people's mental health and expose them to harmful content.
• Intellectual Property Theft: AI models trained on artists' work without permission or compensation raise serious questions about creative rights.

Baker argued that treating AI as an exceptional technology requiring special rules actually shields companies from the kind of rigorous oversight applied to other industries. "The real way for us to ensure we have a safe future is to stop treating AI as exceptional and impose regulations and safeguards, like any other industry," Baker stated.

Baker

What Policy Solutions Are Experts Proposing?

The path forward remains contested, but several concrete proposals have emerged from different camps. State Senator Scott Wiener, who has been advocating for AI safety legislation since 2023, shared his experience pushing for regulatory action in California. His first proposal, which included an AI "kill switch" and liability for reckless model releases, was vetoed. A second version focused on transparency and incident reporting was signed into law, but Wiener emphasized that more action is needed.

Baker outlined a regulatory framework modeled on how other industries are governed:

• Surveillance Infrastructure Regulation: Controlling how data is collected and used before surveillance systems become entrenched in society.
• Data Center Oversight: Regulating where and how AI training facilities are built to address environmental and resource concerns.
• Multi-Level Governance: Creating rules at federal, state, and local levels similar to food safety systems, which combine FDA oversight with restaurant inspections and industry standards.
• Government Leadership: Rejecting industry self-regulation in favor of mandatory government enforcement and standards.

Baker noted that California is positioned to lead on this issue, having pioneered earlier AI guardrails for the country. "We have the expertise and the political will to take action here, and we can set strong safety standards while Washington gets it together," Baker said.

Baker

How Should AI Development Proceed?

Accelerationists, who believe rapid AI development is necessary for human flourishing and defense against geopolitical threats, propose a different approach. Rather than slowing overall development, they argue for careful controls on deployment and specific narrow restrictions on the most dangerous applications.

An accelerationist perspective presented in the expert roundup emphasized that slowing AI development globally is impractical, since neither China nor open-source communities can be controlled. "There's no way to slow down China or open source, so having the strongest AI is likely the best defense against other nations' AI, quantum decryption, and cyberattacks," the accelerationist expert noted.

This camp supports stronger prerelease safety monitoring and evaluation, and potentially narrow bans on creating AI systems specifically designed for catastrophic purposes, such as designing biological weapons. However, they warn that broad slowdowns risk "hobbling the defenders of democracy and civilization".

Notably, this debate has created unusual political alignments. U.S. President Donald Trump and Chinese President Xi Jinping appear to agree on accelerating AI development, alongside Nvidia CEO Jensen Huang. Meanwhile, leaders from companies like OpenAI and Anthropic are pushing for slower, more cautious development.

The fundamental disagreement reflects competing visions of what AI safety actually means. For some, it means preventing existential catastrophe through development controls. For others, it means protecting people from harms happening right now through regulation and accountability. As AI systems become more powerful and integrated into society, these competing frameworks will likely shape policy debates for years to come.