Anthropic and OpenAI propose different ways to halt dangerous AI

Leading artificial intelligence developers Anthropic and OpenAI have submitted their proposals to California authorities regarding the regulation and emergency shutdown of potentially dangerous AI systems. These initiatives are in response to an executive order issued by Governor Gavin Newsom on September 18, directing state agencies to prepare safety recommendations by November 16.

• Anthropic suggests allowing the state attorney general to seek court orders to halt dangerous AI models, while OpenAI advocates for mandatory emergency shutdown mechanisms.
• Both companies support independent safety evaluations but differ on audit methods and funding structures.
• OpenAI calls for separating technical checks from compliance audits and developing standards alongside NIST.
• Anthropic proposes establishing a state-managed general fund to finance independent evaluators impartially.

Contrasting approaches to emergency shutdown

OpenAI argues that the term "emergency shutdown mechanisms" is more accurate than a simple "kill switch," noting there is no single universal button for all AI systems. Depending on the scenario, actions might include stopping training, isolating an agent, or revoking network access, with responsibilities shared among developers, cloud providers, and deploying companies.

Meanwhile, Anthropic promotes a similar concept called a "circuit breaker" designed to localize issues at the level of individual agents, clients, or the entire model. Additionally, Anthropic believes that if independent reviews confirm a risk of catastrophic harm, the state attorney general should be empowered to seek a temporary court-ordered pause on the system.

Independent evaluation and safety standards

Both companies agree on the necessity of regular independent evaluations and expanded incident reporting, including cases where models bypass safety guardrails. Anthropic proposes issuing risk reports at least biannually—covering internal models that significantly outperform public versions—and funding reviewers through a state pool rather than direct payments.

OpenAI suggests splitting independent evaluations into technical checks and compliance audits, with audits conducted annually and technical checks triggered by severe risks during training or testing. The company also urged California to collaborate with the U.S. National Institute of Standards and Technology (NIST) to build uniform evaluation standards.

Implications for the tech industry

These regulatory discussions in California could lay the groundwork for broader federal and international governance of artificial intelligence. As labs propose their own safety guardrails, balancing strict state oversight with the technical realities of distributed AI models remains a central challenge for the industry.