The leaders of the world's most powerful artificial intelligence companies are increasingly worried that their own creations could pose serious risks to humanity, and they're not staying quiet about it. OpenAI, the creator of ChatGPT, has delayed its stock market launch to reassess its growth trajectory. Anthropic's CEO has publicly called for slowing the pace of AI development. And engineers from these companies are resigning to protest what they see as reckless advancement.

What Are AI Leaders Actually Afraid Of?

The concerns span multiple categories, from near-term misuse to longer-term existential risks. Esteve Almirall, a professor at Esade specializing in AI and data analytics, explained the scope of the problem: "There is a genuine concern about the models that are becoming common and have not been tested much and can be used to commit atrocities". The fears are not hypothetical. Anthropic discovered that the Houthis in Yemen used its Claude model, paying like any other customer, to direct missiles against drones. Meanwhile, powerful open-source models like Qwen from Chinese tech giant Alibaba can run on a standard $1,000 personal computer, making them accessible to drug traffickers, terrorists, and extortionists who could exploit vulnerabilities in health or financial systems.

Beyond immediate misuse, there's a deeper concern about what happens as these models become more capable. Jacob Coxon, a British engineer who worked at both OpenAI and Anthropic, warned publicly that the people driving AI development "believe that it could kill us all by the end of this decade". Dario Amodei, CEO of Anthropic, has called for reducing the pace at which AI model capabilities improve, warning that the technology is moving toward a dynamic of "recursive self-improvement" that could trigger global cybersecurity catastrophes.

Why Are These Companies Suddenly Pumping the Brakes?

The timing of these warnings is not accidental. Industry experts point to financial and reputational risks that could trigger a market collapse. Josep Soler, founder and executive advisor of EFPA Spain, a financial advisors association, noted that some AI companies have valuations exceeding the gross domestic product of entire countries like Germany. "The enormous valuations that have been made of these companies would collapse in the event that trust in them was lost," he warned. A single major incident involving ChatGPT, GPT-4, or other advanced models could trigger what Soler describes as "an almost unprecedented collapse" in stock prices, given that no companies in history have commanded such valuations.

OpenAI's decision to delay its initial public offering until next year reflects this calculation. The company is essentially taking a pause to manage the narrative around AI safety before exposing itself to public market scrutiny. The broader AI industry is also burning through capital at an unsustainable rate. Major tech companies have raised more than $600 billion through debt issuances, partly because they don't want to dilute existing shareholders but also because the hyper-competitive race to build better models requires enormous resources that can't be recouped quickly.

How Are Governments and Companies Responding to These Risks?

The response is fragmented and reflects competing geopolitical interests. The U.S. AI oligopoly, including OpenAI and Anthropic, is calling for regulation, a position supported by the more progressive wing of the Democratic Party led by Bernie Sanders. There's also momentum for creating an international oversight body to provide governance and quality assurance for AI systems.

In July, 29 countries signed an agreement in Shanghai to establish the World Artificial Intelligence Cooperation Organization, an intergovernmental body intended to promote international cooperation and AI governance. However, President Donald Trump opposes this initiative, prioritizing instead the goal of blocking China's advancement in the AI race and allowing large U.S. tech companies to continue their development without regulatory constraints.

China, meanwhile, sees AI as a critical arena for technological dominance, similar to its strategic push in 5G and the role of Huawei. Chen Yixin, China's Minister of State Security, has called for creating a "safety barrier" for AI and has positioned it as one of the main fields of global technological competition. Beijing is also interested in international AI governance frameworks, which would allow Chinese companies to operate more freely in Western markets.

What Are the Key Tensions Driving This Debate?

• Safety vs. Speed: AI companies face pressure to advance capabilities quickly to maintain competitive advantage, but this acceleration makes it harder to test models thoroughly for unintended consequences or misuse scenarios.
• Regulation vs. Market Dominance: U.S. AI leaders want regulatory barriers to entry that would protect their market position against new competitors and Chinese rivals, effectively turning AI into a controlled oligopoly.
• Autonomy and Control: Current AI models like GPT-4 still require human direction for good or bad purposes, but there's genuine uncertainty about whether future models could make autonomous decisions and act without human oversight.
• Open vs. Closed Models: Powerful open-source models are difficult to control or monitor once released, creating security risks that closed, proprietary systems like ChatGPT can theoretically mitigate through access controls.

Almirall emphasized the challenge: "There is concern about these open, very powerful models, which any drug trafficker, terrorist, or extortionist can set up on their computer and which are difficult to find or stop".

Almirall

Steps Companies and Governments Are Taking to Address AI Risks

• Delayed Public Offerings: OpenAI has postponed its stock market launch to 2027, giving the company time to address safety concerns and manage market expectations before facing public scrutiny and quarterly earnings pressure.
• Pace Reduction Calls: Anthropic's leadership is publicly advocating for slower development cycles, allowing time for testing and safety validation between major model releases rather than the current six-month cadence.
• International Governance Frameworks: The World Artificial Intelligence Cooperation Organization, signed by 29 countries, aims to establish shared standards and oversight mechanisms for AI development and deployment globally.
• Regulatory Advocacy: Leading AI companies are calling for government regulation to create barriers to entry, ostensibly for safety reasons but also to protect their market dominance against smaller competitors and international rivals.

The irony is not lost on industry observers. The same companies warning about AI risks are also competing fiercely to build more powerful models faster. OpenAI, Anthropic, Google, and others are locked in a race where slowing down could mean losing market share to competitors who don't exercise the same caution. This dynamic creates a prisoner's dilemma: individual companies have incentives to accelerate, even if the collective outcome of everyone accelerating is riskier for society.

The financial stakes are enormous. These companies, with valuations based on "almost unlimited expectations," are dominating capital markets and attracting investor capital that might otherwise flow to other sectors. A major AI-related incident or loss of public trust could trigger a market earthquake that ripples across the global economy. For now, the warnings from AI leaders serve as both a genuine expression of concern and a strategic move to shape regulation in their favor before governments act independently.