US and China Experts Push for New AI Safeguards as Autonomous Systems Raise Global Risks

US and China Experts Push for New AI Safeguards as Autonomous Systems Raise Global Risks

Artificial intelligence is rapidly becoming more capable, but a growing international debate is moving beyond what AI can do to a more difficult question: what happens when an autonomous system makes a mistake during a geopolitical crisis?

Security specialists from the United States and China are proposing new safeguards designed to reduce the possibility that artificial intelligence could contribute to an unintended military or cyber escalation between the world's two largest economies.

The recommendations include maintaining human control over critical decisions, establishing red lines around nuclear systems and creating a dedicated communication channel for incidents involving autonomous AI.

The proposals come from experts involved in a long-running US-China dialogue convened by the Brookings Institution and Tsinghua University's Center for International Security and Strategy. Reuters reported on the recommendations on September 17.

The underlying concern is straightforward but potentially serious.

As AI systems gain greater autonomy, machines can analyse information, identify threats and initiate actions at speeds far beyond normal human decision-making.

That speed can be useful.

During an international crisis, however, it can also become a risk.

When an AI Mistake Looks Like an Attack

Imagine an autonomous cybersecurity system detecting suspicious activity on a critical computer network.

The system decides to respond immediately.

An AI system operated by another country detects that response and interprets it as a deliberate cyberattack. It automatically launches its own countermeasure.

Within seconds, two automated systems could potentially begin reacting to each other before human officials fully understand what started the incident.

That possibility is one reason experts are discussing whether traditional crisis-management systems are sufficient for the AI era.

The danger is not necessarily that a machine deliberately decides to begin a conflict.

A malfunction, misunderstood instruction, compromised system or incorrectly classified threat could potentially create an action that another government interprets as intentional.

When nuclear-armed countries are involved, distinguishing between an accident and a deliberate attack becomes particularly important.

Human Control Is Becoming a Global Question

One of the central recommendations is that humans should retain authority over the most consequential military decisions.

Experts have proposed explicit restrictions preventing AI from independently deciding to use nuclear weapons or autonomously attacking nuclear command-and-control systems.

The concept has some precedent.

In November 2024, the United States and China endorsed the principle that humans, rather than artificial intelligence, should retain control over decisions involving nuclear weapons.

The challenge now is defining what human control actually means as AI becomes more deeply integrated into defence and cybersecurity systems.

A human technically approving an action is not necessarily meaningful oversight if the machine has already made thousands of decisions leading to that recommendation and the person has only seconds to respond.

This is why researchers are increasingly discussing the idea of meaningful human control rather than simply having a person somewhere in the decision chain.

Why an AI Hotline Is Being Discussed

Another proposal involves creating a dedicated US-China communication channel for incidents involving autonomous artificial intelligence.

Traditional military hotlines are designed to give governments a way to communicate during dangerous situations.

AI creates a new complication because automated systems can operate extremely quickly.

A dedicated channel could theoretically allow one country to tell another that an unusual AI-driven operation was accidental, unauthorised or still under investigation before the event is interpreted as an intentional hostile act.

Whether such a mechanism would work reliably is another question.

Existing US-China crisis communication systems have faced criticism because communication between the two sides has not always functioned effectively during previous disputes.

Technology alone cannot solve that problem. A hotline only works when governments are willing and able to use it.

The AI Race Makes Cooperation Difficult

There is a major obstacle to international AI safeguards: the countries discussing safety are simultaneously competing for technological leadership.

The United States and China are central players in artificial intelligence, advanced computing and semiconductor technology.

Both have strategic reasons to avoid restrictions that could slow their own development while allowing the other side to advance.

That creates a difficult balance.

Governments may recognise that powerful autonomous AI introduces shared risks while simultaneously disagreeing about technology controls, access to advanced chips, national security and industrial competition.

Associated Press reporting this week similarly highlighted the tension between the need for US-China cooperation on AI safety and the broader strategic competition between the two countries.

This makes AI governance different from a purely technical problem.

It is increasingly a diplomatic one.

AI Safety Is Moving Into Mainstream Debate

Concerns about powerful AI systems are also expanding beyond military applications.

On Thursday, King Charles hosted technology executives and other specialists at Dumfries House in Scotland for discussions about AI safety.

Participants included representatives from major AI companies such as Nvidia, Google DeepMind, OpenAI and Anthropic, according to Reuters. The meeting focused on how increasingly powerful systems can be developed while reducing potentially severe risks.

Separately, OpenAI has disclosed examples of unexpected behaviour observed during model development and evaluation and introduced a framework for monitoring such incidents more systematically.

These developments are contributing to a broader shift in the AI conversation.

For several years, much of the public discussion focused on generative AI producing text, images and computer code.

The next stage is increasingly about AI agents — systems capable of performing longer sequences of actions with less direct human involvement.

Greater autonomy can make AI substantially more useful.

It can also make failures more consequential.

Critical Infrastructure Raises the Stakes

The debate becomes particularly important when AI is connected to systems outside a chatbot.

Energy networks, financial infrastructure, telecommunications, healthcare, transportation and government computer systems can all be considered critical infrastructure.

As AI becomes more deeply integrated into these environments, security specialists need to consider not only whether a model provides the correct answer but also what authority it has to act.

An AI assistant incorrectly summarising a document creates one level of risk.

An autonomous system incorrectly identifying activity as a cyberattack and responding without adequate human supervision creates another.

This distinction will become increasingly important as businesses and governments deploy AI agents capable of using software, communicating with other systems and carrying out multi-stage tasks.

There Is No Global AI Rulebook Yet

International AI governance remains fragmented.

Countries and regions have adopted different approaches to regulation, safety testing, data protection and the development of advanced models.

Meanwhile, technological capabilities are moving quickly.

That creates a timing problem for policymakers.

Regulation written around today's AI systems can become outdated as models gain new capabilities.

At the same time, waiting until every risk is fully understood may mean rules arrive only after powerful systems have already been widely deployed.

The US-China proposals do not solve that problem, and neither government has formally adopted the recommendations reported this week.

Their significance lies elsewhere.

They show that discussions about artificial intelligence are beginning to resemble conversations traditionally associated with cybersecurity, arms control and international crisis management.

The Next AI Race May Include Safety

Competition in artificial intelligence is unlikely to disappear.

Companies want more capable models. Governments want technological advantages. Investors want commercial returns. Researchers continue pushing the boundaries of what machines can accomplish.

But another competition may now be emerging alongside the race for capability: the effort to develop credible safeguards before autonomous systems become embedded in increasingly sensitive parts of society.

That will require more than promises from technology companies.

It could require technical standards, independent testing, communication between governments, clearly defined boundaries and agreement about which decisions should always remain under meaningful human control.

Artificial intelligence has already transformed the global technology race.

The debate unfolding now is about whether international rules and safeguards can evolve quickly enough to keep pace with the machines themselves.