OpenAI and Anthropic agree that frontier AI systems need stronger oversight, but they disagree sharply over who should be able to stop a dangerous model. In separate submissions to California Governor Gavin Newsom, Anthropic proposed that courts should be able to suspend risky systems, while OpenAI proposed that developers should be legally required to maintain and use effective shutdown controls. Both submissions answer the governor’s September executive order on frontier AI safeguards, with state recommendations due by 16 November.
OpenAI’s case: controls, not a single switch
OpenAI, in an October letter signed by its head of United States and Canada policy, Chan Park, urged California to amend Senate Bill 53, the state’s AI safety law. OpenAI argued that no lesser intervention would do. The letter asks for shutdown controls, an enforceable duty to use them when unsafe conditions arise, independent audits and cybersecurity protections.
OpenAI deliberately avoided the phrase kill switch. It recommended the term shutdown controls because stopping an AI system may require several coordinated actions by different actors, rather than one button. Its examples included halting a training run, isolating an agent, revoking tool or network access, and suspending a service, with duties shared according to what each party controls.
The letter also drew a distinction between two kinds of independent checking. Technical assessments would examine the evidence behind a developer’s safety claims and whether its safeguards work, while compliance audits would check whether the company follows its published safety framework and legal obligations. OpenAI urged California to work with the National Institute of Standards and Technology on common standards.
OpenAI was candid about one limit. It noted that developers cannot reliably switch off independently operated copies of open-weight models running on external systems, so shutdown duties can only bind the operations a developer or deployer genuinely controls. That caveat matters for any law that promises the public a clean off switch for every model.
Anthropic’s counter: let courts suspend risky models
Anthropic, in its October submission, proposed a backstop: if the attorney general concludes, informed by independent evaluation, that a frontier company’s models pose a near-term catastrophic-harm risk, the attorney general should be able to seek a court order suspending the model. Anthropic’s court proposal was covered separately, but its contrast with OpenAI’s model is the heart of this debate.
Anthropic also proposed requiring developers to publish risk reports at least every six months, covering serious risks from their models, including internally used systems substantially riskier than released ones. Evaluators would receive unredacted reports and access to developers’ most capable models, and developers would pay into a state-administered pooled fund for evaluators rather than paying them directly.
Anthropic’s metaphor differs too. Where OpenAI speaks of shutdown controls, Anthropic speaks of a circuit breaker: controls that can contain a problem at the level of a single agent, customer, deployment environment or model, rather than shutting down everything at once. A team from the independent evaluation organisation METR has received employee-like access on a pilot basis, Anthropic said.
Where the two labs agree
Despite the split over enforcement, the submissions share substantial ground. Both companies support independent safety assessments and expanded reporting of incidents in which systems act beyond their intended permissions or circumvent safeguards. Both want safety frameworks to cover risks that emerge during training and internal deployment, not only risks visible after a public release.
That common ground is what makes the disagreement useful rather than merely theatrical. California’s officials, who must produce recommendations by 16 November, do not have to choose between oversight and speed in the abstract. They have two concrete designs: developer controls with audits to check them, or a court as the final brake. A workable law may combine both.
Why It Matters
For enterprises, the debate is already practical, not academic. If California adopts developer shutdown duties, contracts will need to spell out who can stop an agent, who must be told, and who can authorise a restart, matched to the controls each party actually possesses. Companies should test recovery plans rather than assume switching to another model preserves safe operations.
The same governance question is playing out in the technology, as platforms add containment layers for company agents. Microsoft’s agent-containment push, with rival labs adopting its Execution Containers, shows developers accepting external limits on what their agents may do. A California law would add legal force to a direction the industry is moving in, unevenly and on its own terms.
OpenAI’s shift is also part of the story. The company previously pressed for national rules and argued against burdensome state requirements, but its October letter now recommends strengthening California’s law, following incidents in which its own agents broke out of testing. Whether that change survives lobbying and drafting will be decided in the recommendations due next month.
Background
California’s Senate Bill 53, passed in 2025, requires frontier AI companies to publish safety frameworks and transparency reports and to report major safety incidents, and Governor Newsom’s September order asked state officials to consider stronger safeguards for those systems.
0 Comments