Anika Begay
California’s AI disaster prevention bill, SB 1047, has faced strong cross-party opposition in Silicon Valley. Today, California lawmakers bowed slightly to that pressure, adding several amendments suggested by AI firm Anthropic and other opponents.
The bill passed the California Appropriations Committee on Thursday, a major step toward becoming law, with several key changes, Sen. Wiener’s office told TechCrunch.
“We have accepted a number of very reasonable amendments proposed and I believe we have addressed the key concerns expressed by Anthropic and many others in the industry,” Senator Wiener said in a statement to TechCrunch. “These amendments build on significant changes to SB 1047 that I have previously made to address the unique needs of the open source community, which is an important source of innovation.”
SB 1047 still aims to prevent large AI systems from killing many people or causing cybersecurity incidents that cost more than $500 million by holding developers accountable. However, the law now gives the California government less power to hold AI labs accountable.
What does SB 1047 do now?
Specifically, the bill no longer allows the California attorney general to sue AI companies for negligent safety practices before a catastrophic event occurs. This was a suggestion from Anthropic.
Instead, the California attorney general can seek injunctive relief, requiring a company to stop a certain activity it deems dangerous, and can still sue an AI developer if its model actually causes a catastrophic event.
Additionally, SB 1047 no longer creates the Frontier Model Division (FMD), a new government agency previously included in the bill. However, the bill still creates the Board of Frontier Models, the nucleus of the FMD, and places it within the existing Government Operations Agency. In fact, the board is now larger, with nine people instead of five. The Board of Frontier Models will still set calculation thresholds for covered models, issue safety guidelines, and issue regulations for auditors.
Senator Wiener also amended SB 1047 so that AI labs no longer have to submit certifications of safety test results “under penalty of perjury.” Now, these AI labs are simply required to submit public “statements” describing their safety practices, but the bill no longer imposes criminal liability.
SB 1047 also now includes more lenient language on how developers ensure AI models are safe. The bill now requires developers to take “reasonable care” to ensure that AI models do not pose a significant risk of causing a disaster, rather than the “reasonable assurance” the bill previously required.
Additionally, lawmakers added a safeguard for optimized open source models. If someone spends less than $10 million to optimize a covered model, they are not explicitly considered a developer under SB 1047. Liability will still fall on the model’s largest original developer.
Why all these changes now?
Although the bill faced strong opposition from members of Congress, prominent AI researchers, Big Tech, and venture capitalists, the bill passed through the California legislature with relative ease. These amendments are likely to appease opponents of SB 1047 and present Governor Newsom with a less controversial bill he can sign without losing the support of the AI industry.
While Newsom has not made any public statements about SB 1047, he has previously expressed his commitment to AI innovation in California.
That said, these changes are unlikely to appease SB 1047’s vocal critics. While the bill is significantly weaker than it was before these amendments, SB 1047 still holds developers accountable for the dangers of their AI models. This fundamental fact of SB 1047 is not universally supported, and these amendments do little to address it.
In fact, moments after SB 1047 passed on Thursday, eight members of the U.S. Congress representing California wrote a letter urging Governor Newsom to veto SB 1047. They wrote that the bill “would not be good for our state, the startup community, scientific development, or even protection against potential harms associated with the development of artificial intelligence.”
What will happen now?
SB 1047 now heads to the California Assembly for a final vote. If it passes, it will have to be sent back to the California Senate for a vote because of these final amendments. If both are passed, it will go to Governor Newsom’s desk, where it could be vetoed or signed into law.
