OpenAI’s Sam Altman calls to throttle AI race as state actors hijack Anthropic for bioweapons

OpenAI staff were informed this week that the AI lab would be open to throttling the pace of its work in the most advanced areas of AI. Sam Altman extended the same proposition to AI rivals, including Anthropic, which also acknowledged the risks associated of AI development in a report that disclosed that scientists linked to state institutions attempted to use its Claude models for biological weapons research. 

OpenAI said it has already paused some internal training runs and eased parts of model development over safety concerns.

The serial arrival of the disclosures is stoking the fire around the debate about whether the industry is moving too fast ahead of the public offerings, both firms continue to tease.

Why does Altman want to slow down AI development?

OpenAI chief executive Sam Altman, at a company-wide meeting, informed those present that the AI lab, along with other consenting labs, could slow down the pace of frontier model releases.

Due to the voluntary nature of any such arrangement, Altman conceded that not every company would be on board with the plan, according to sources cited by Bloomberg.

That skepticism comes with precedent. Three years ago, in March 2023, Elon Musk, Steve Wozniak and more than 1,000 others signed an open letter to halt advanced AI work for six months.

As outlets reported at the time, labs did not actually follow through with the moratorium they jointly agreed to.

The AI risk call also came out in a September 6 post by OpenAI chief scientist, Jakub Pachocki, which asked AI firms to coordinate “to slow down future development as needed.” Pachocki expressed hope that voluntary slowdowns would “become commonplace until shared safety bars are established.”

In July, Altman said he told White House officials about the need to throttle AI development. Along with Anthropic CEO Dario Amodei, Altman also previously signed a one-line statement declaring AI a societal-scale extinction risk.

Can AI models be used to build bioweapons?

AI models have advanced to the stage that they may be capable of supporting biological weapons research. Just this week, OpenAI declared that its models had solved the Navier Stokes math problem, even if it came under contested terms.

Anthropic presented fresh fodder for that debate when the lab’s 154-page threat intelligence report, released on September 10, detailed five instances of what researchers attempts to use Claude’s capabilities to support their biological weapons work.

Anthropic said the attempts from “unsupported regions” tried to circumvent location restrictions and disguise the purpose of their work. The lab said it took down the accounts, but it did not name any countries and institutions.

Have AI risks now outweighed the benefits?

The reports arrived amid a run of public alarm from inside the labs. On September 8, researcher Jacob Coxon quit and accused both OpenAI and Anthropic of “gambling with our lives,” saying colleagues believe the technology could “kill us all by the end of the decade” and warning of human extinction by 2030.

His post drew more than 150 million views. Two days later, on September 10, former staff from Anthropic and Google’s DeepMind aired similar concerns and pressed for more transparency.

A late-July petition to create a mechanism for slowing development gathered signatures from more than 1,000 workers across the major AI companies.

OpenAI is now pushing Congress for mandatory, capability-based safety rules covering testing, independent evaluations, cybersecurity and incident reporting. “The prospect of AI-accelerated AI development demands more than voluntary commitments,” Chief Global Affairs Officer Chris Lehane wrote.

California moved first: Governor Gavin Newsom signed SB 813 and AB 1405 on Wednesday, with a companion bill, AB 1864, aimed at guarding against AI-enabled biological threats.

In Washington, Senator Bernie Sanders introduced legislation to pause advanced AI and ban artificial superintelligence, telling the BBC that ignoring scientists’ warnings would make someone “a moron.”

President Donald Trump pushed back, saying on Thursday he worried that “if we don’t win AI, we’re going to be put in a very bad position.”