is The Verge’s senior AI reporter. An AI beat reporter for more than five years, her work has also appeared in CNBC, MIT Technology Review, Wired UK, and other outlets.
Last September, all eyes were on Senate Bill 1047 as it made its way to California Governor Gavin Newsom’s desk — and died there as he vetoed the buzzy piece of legislation.
SB 1047 would have required makers of all large AI models, particularly those that cost $100 million or more to train, to test them for specific dangers. AI industry whistleblowers weren’t happy about the veto, and most large tech companies were. But the story didn’t end there. Newsom, who had felt the legislation was too stringent and one-size-fits-all, tasked a group of leading AI researchers to help propose an alternative plan — one that would support the development and the governance of generative AI in California, along with guardrails for its risks.
On Tuesday, that report was published.
The authors of the 52-page “California Report on Frontier Policy” said that AI capabilities — including models’ chain-of-thought “reasoning” abilities — have “rapidly improved” since Newsom’s decision to veto SB 1047. Using historical case studies, empirical research, modeling, and simulations, they suggested a new framework that would require more transparency and independent scrutiny of AI models. Their report is appearing against the backdrop of a possible 10-year moratorium on states regulating AI, backed by a Republican Congress and companies like OpenAI.
The report — co-led by Fei-Fei Li, Co-Director of the Stanford Institute for Human-Centered Artificial Intelligence; Mariano-Florentino Cuéllar, President of the Carnegie Endowment for International Peace; and Jennifer Tour Chayes, Dean of the UC Berkeley College of Computing, Data Science, and Society — concluded that frontier AI breakthroughs in California could heavily impact agriculture, biotechnology, clean tech, education, finance, medicine and transportation. Its authors agreed it’s important to not stifle innovation and “ensure regulatory burdens are such that organizations have the resources to comply.”
“Without proper safeguards… powerful Al could induce severe and, in some cases, potentially irreversible harms”
But reducing risks is still paramount, they wrote: “Without proper safeguards… powerful Al could induce severe and, in some cases, potentially irreversible harms.”
The group published a draft version of their report in March for public comment. But even since then, they wrote in the final version, evidence that these models contribute to “chemical, biological, radiological, and nuclear (CBRN) weapons risks… has grown.” Leading companies, they added, have self-reported concerning spikes in their models’ capabilities in those areas.
The authors have made several changes to the draft report. They now note that California’s new AI policy will need to navigate quickly-changing “geopolitical realities.” They added more context about the risks that large AI models pose, and they took a harder line on categorizing companies for regulation, saying a focus purely on how much compute their training required was not the best approach.
... continue reading