The United States and China have agreed to create a formal AI safety dialogue and notification system to alert each other about artificial intelligence incidents that could pose national security risks. US Treasury Secretary Scott Bessent and China's Vice-Premier He Lifeng met at J.P. Morgan headquarters in New York on Sunday to discuss the framework, which is expected to be a focal point when Chinese President Xi Jinping visits Washington from September 23 to 25 for talks with President Trump.

Why Are the World's Two AI Superpowers Suddenly Talking About Safety?

The agreement comes as major technology leaders have issued increasingly urgent warnings about existential risks from artificial intelligence. Microsoft CEO Satya Nadella, Anthropic CEO Dario Amodei, and OpenAI CEO Sam Altman have all recently sounded alarms about the dangers posed by rapid AI advancement. These concerns intensified following reports of recursive self-improvement in AI systems, model misalignment incidents, and a warning from former Anthropic researcher Jacob Coxon that there is more than a 10% chance AI could pose an existential threat to humanity.

The geopolitical stakes are particularly high. Both the US and China are locked in an intense race to develop artificial general intelligence, or AGI, which refers to AI systems with human-level reasoning across any domain. Without strong international guardrails, experts worry this competitive dynamic could push both nations to prioritize speed over safety. Max Tegmark, an MIT professor and president of the Future of Life Institute, has called an AGI race "a suicide race," arguing that competitive pressures leave no room to solve critical technical problems around AI control and alignment.

What Exactly Will This New AI Safety Mechanism Do?

The notification system is designed to raise alerts when AI-related incidents reach a national security level. Treasury Secretary Bessent emphasized that the mechanism represents an important first step toward reducing risks from the rapidly developing technology. "The most important thing is just to start talking and so we have set up the AI Dialogue. We will likely meet again in two months," Bessent told the Financial Times.

The Carnegie Endowment for International Peace, which has advised on the framework, recommended that such a crisis channel should include several key features to be effective:

• Communication Method: The channel should be text-based to ensure clear, documented exchanges between the two countries.
• Tiered Response System: Different levels of alerts should trigger proportional responses, preventing both overreaction and dangerous delays.
• Specific Triggers: Clear criteria should define what constitutes an AI safety incident worthy of notification.
• Continuous Staffing: The channel must be staffed 24/7 by technically proficient personnel who understand both AI systems and national security implications.

Scott Singer, co-director of the China AI Initiative at the Carnegie Endowment for International Peace, noted that the timing reflects growing urgency. "The US and China are expected to hold AI safety talks later this month. With AI capabilities improving fast, lots of people are calling for a grand bargain to prevent catastrophe," Singer observed.

How to Understand the Competing Visions of AI Risk

Despite the agreement on dialogue, the two leaders bring starkly different perspectives on AI danger. President Trump has expressed skepticism about existential AI risks, calling concerns about the technology "a hoax." NVIDIA CEO Jensen Huang echoed this view, stating: "I completely disagree that AI is going to be the end of the world in 2030. There is a 0% chance that's going to be the end of the world".

Yet even Trump has announced plans to form an "AI Force" and appoint an AI tsar, suggesting that despite his public skepticism, the administration recognizes the need for some oversight. Meanwhile, Chinese President Xi has laid out a more cautious vision. In comments released by China's Ministry of Foreign Affairs, Xi stated that "global AI technology innovation is entering an unprecedented period of activity" and called for countries to "attach great importance to the various inherent and derivative risks arising from AI".

Foreign Affairs

"We must advocate for concerted efforts and improve global governance. AI is a shared intellectual achievement and a valuable asset of all humankind," Xi said, stressing that countries should "establish a broadly consensus-based global governance framework as soon as possible."

President Xi Jinping, President of China

Industry leaders have largely sided with the more cautious perspective. Anthropic CEO Dario Amodei has argued that a competitive "race to the bottom, spurred by commercial incentives, can make these risks from AI more acute." He advocates for "pacing" development to give researchers time to solve alignment problems, the technical challenge of ensuring AI systems behave as intended. OpenAI CEO Sam Altman and Elon Musk of xAI have echoed similar sentiments, with Altman emphasizing that losing control of humanity's future to AI is "unacceptable".

What Does This Mean for the Future of AI Development?

The establishment of a US-China AI safety dialogue represents a significant diplomatic shift, even if both nations continue to compete fiercely on AI capabilities. The notification system could help prevent misunderstandings that might otherwise escalate tensions during a critical period of AI advancement. However, experts caution that dialogue alone cannot solve the underlying problem: the structural incentive for both nations to move fast and take risks in pursuit of AI leadership.

The coming weeks will reveal whether this framework leads to meaningful cooperation or remains largely symbolic. The September 23 to 25 summit between Trump and Xi will be closely watched by AI researchers, policymakers, and industry leaders who see international coordination as essential to managing existential risks from advanced AI systems.