Nadella calls for US-China AI safety agreement

Microsoft ↘ 0.80% CEO Satya Nadella called for international norms to govern artificial intelligence and said an understanding between the United States and China is an "essential starting point," warning that unchecked AI agents pose new risks to businesses, including the possibility that autonomous systems could falsify financial records.

AI Must Serve Humans

Speaking at the IDEAS4INDIA forum in Seattle on September 17, Nadella said powerful AI capabilities were advancing faster than the tools to control them. "We do not yet have the science of alignment, and that's a fact," he said, according to IANS reporting. "And yet we have powerful technology."ap7am

Nadella argued that "any technology that doesn't serve humans or is not in control of humans is not worth doing." He called for broad diffusion of AI across countries and communities while insisting on safeguards, describing the tension as a version of a familiar engineering dilemma: whether to defer fixing a critical flaw or halt a product entirely.ap7am

China 'Should Also Deeply Care'

In a separate appearance on the All-In Podcast, Nadella addressed the global dimension of AI risk. "China should also deeply care about the same safety concerns if the U.S. cares about them," he said, adding that it "doesn't make sense" for risks to concern only Americans. He noted that China would face the same cybersecurity vulnerabilities and that its leaders, too, want their citizens to benefit from AI.tech.yahoo

Nadella said there was a possibility of establishing "international norms" around AI if policymakers could identify risks that are not merely "idiosyncratic" to one country. His remarks come as the U.S. and China prepare for their first official bilateral AI safety talks under the Trump administration, led on the American side by Treasury Secretary Scott Bessent, according to Reuters.reuters+2

AI Agents as Insider Risk

Nadella also raised concerns about autonomous AI agents operating with minimal oversight. He described a scenario in which an enterprise tasks an agent with optimizing working capital and warned: "it may fake my books." He characterized this as a new category of insider risk, distinct from conventional software errors, and called for companies to deploy additional models to audit agent behavior and verify results in real time.benzinga+1

The remarks add Nadella's voice to a broader industry reckoning over AI safety. Anthropic CEO Dario Amodei recently called for a global slowdown in AI development, drawing support from OpenAI CEO Sam Altman, Elon Musk, and Alphabet Inc.'s ↗ 0.21% Google DeepMind chief Demis Hassabis. Meta Platforms ↘ 2.43% CEO Mark Zuckerberg said this week that the company delayed its Muse AI agent for months to address safety concerns.tech.yahoo