The leaders building the world's most powerful AI systems are openly assigning odds to human extinction. Dario Amodei, CEO of Anthropic, estimates a 25% chance that AI development "goes really, really badly" for humanity. Elon Musk puts the figure around 20%, while Geoffrey Hinton, the Nobel laureate known as the "Godfather of AI," has suggested the risk could be as high as 50%. These aren't fringe doomsayers; they're the architects of the technology itself.

The AI community has developed a shorthand for this calculation: "p(doom)," or probability of doom. It's become a standard question posed to executives and engineers, asking them to quantify the likelihood that advanced AI development could trigger societal collapse, widespread death, or human extinction. The fact that leaders of major AI companies are willing to assign such high probabilities to catastrophic outcomes suggests the industry is grappling with genuine existential concerns.

What Is the 'Probability of Doom' and Why Does It Matter?

The p(doom) metric emerged from within AI circles as a way to discuss the existential risks posed by increasingly powerful AI systems. When these leaders talk about "doom," they're generally referring to scenarios in which AI models become capable of causing events that lead to societal collapse or the extinction of humanity itself. The variation in their estimates is striking: some see a 10% risk, others 25%, and a few suggest it could be closer to 50%.

This uncertainty reflects a deeper problem. No one truly knows how advanced AI systems will behave once they reach a certain level of capability. The technology is moving faster than our ability to predict its outcomes, and even the people designing these systems acknowledge they're heading into uncharted territory.

How Are AI Leaders Divided on the Path Forward?

The AI community has split into two camps with fundamentally different philosophies. On one side are the "Doomers," who believe the risks are severe enough to warrant slowing down AI development and prioritizing safety research. On the other side are the "Boomers," tech optimists who argue that accelerating AI adoption will unlock transformative benefits and that excessive caution could be counterproductive.

Sam Altman, CEO of OpenAI, exemplifies the accelerationist position. In a September 2024 blog post, he wrote that "astounding triumphs, fixing the climate, establishing a space colony, and the discovery of all of physics, will eventually become commonplace" if AI stays on its current trajectory. He has even suggested that increased computing power could make a cure for cancer possible. This optimism stands in sharp contrast to the warnings from Doomers who see the same technological path as potentially catastrophic.

Despite their disagreements about risk tolerance, both camps agree on one thing: the next major milestone in AI development, called Artificial General Intelligence (AGI), is coming soon. AGI refers to an AI system that can outperform humans across a broad range of tasks and uses reasoning, understanding, and adaptability rather than narrow, specialized functions. Altman has recently referred to this as "superintelligence".

What Specific Risks Are AI Companies Warning About?

On the same day this article was published, Anthropic released an essay calling for the option to "pause" AI development. The company warned about a phenomenon called "recursive self-improvement," in which an AI model becomes capable of designing and developing more powerful versions of itself without human intervention. Anthropic stated that full recursive self-improvement could significantly increase the risks of "humans losing control over AI systems".

Anthropic

This concern reflects a fundamental shift in how the industry views its own creations. The models being developed today are beginning to show capabilities that earlier systems did not possess, and the trajectory suggests that future systems could operate in ways that are difficult for humans to predict or control. The fact that one of the major AI companies is publicly calling for a development pause indicates that internal safety concerns have reached a critical level.

• Recursive Self-Improvement: AI systems that can design and build more advanced versions of themselves without human oversight, potentially accelerating capability growth beyond human control.
• Loss of Control: As AI systems become more capable, the risk increases that humans may no longer be able to understand or direct their actions, leading to unintended consequences.
• Societal Collapse Scenarios: Worst-case outcomes in which AI-driven events trigger widespread disruption, economic failure, or loss of human life on a massive scale.

How Does This Compare to Historical Precedent?

The parallels to the development of nuclear weapons are difficult to ignore. When J. Robert Oppenheimer was asked by Major General Leslie R. Groves whether there was a chance that detonating the first atomic bomb might "destroy the world," Oppenheimer replied that the chances were "near zero." The general responded, "Zero would be nice." Today's AI leaders face a similar dilemma: they are developing technology whose full consequences remain unknown, yet they are proceeding with development anyway.

The comparison is not lost on the AI industry itself. In 2023, Sam Altman rented out a cinema in San Francisco for OpenAI employees to watch the film "Oppenheimer," which explores the physicist's internal struggles with having created a weapon of mass destruction. Altman later posted on X that he found the film underwhelming, suggesting he may have been uncomfortable with the parallels being drawn between Oppenheimer's moral dilemmas and the existential questions facing AI developers.

What Role Is the Religious Community Playing?

In May 2026, Pope Leo XIV issued an encyclical addressing the dangers of artificial intelligence. The Pope called on AI developers to "disarm" AI, clarifying that this did not mean rejecting technology but rather "preventing it from dominating humanity." He addressed developers directly, saying they bore an "ethical and spiritual responsibility" to ensure that what they are creating is a "genuine good".

The Pope's intervention reflects a broader recognition that AI development is not merely a technical problem but a moral and spiritual one. The religious framing adds another dimension to the debate, suggesting that the stakes extend beyond economic or political concerns to fundamental questions about human dignity and the future of civilization itself.

Steps to Understanding the AI Risk Debate

• Learn the Terminology: Familiarize yourself with key terms like AGI (Artificial General Intelligence), p(doom) (probability of doom), and recursive self-improvement to follow industry discussions and news coverage.
• Follow the Key Players: Pay attention to statements from major AI company leaders like Sam Altman, Dario Amodei, and Elon Musk, as well as researchers like Geoffrey Hinton, to understand where the industry consensus is shifting.
• Track Policy Developments: Monitor regulatory and governance initiatives from governments, international bodies, and institutions that are attempting to establish safety standards before AGI is achieved.
• Examine the Evidence: Look for published research and essays from AI safety organizations like Anthropic that detail specific technical risks and proposed solutions.

The debate over AI risk is not abstract or distant. The decisions made by a small group of executives and engineers over the next few years could determine the trajectory of human civilization. Whether one assigns a 10% probability of doom or a 50% probability, the stakes are undeniably high, and the uncertainty is profound. As Pope Leo XIV noted, the future of humanity may ultimately depend on whether those building AI systems choose to prioritize safety alongside capability.