GoKawiilYesterday I covered the OpenAI technical report on the HuggingFace hack.
That report had one key new piece of information, and some good prosaic steps OpenAI will be taking to strengthen its alignment, training, supervision, infrastructure and incident response.
Mostly it confirmed what we already knew. The questions we most wanted answers to, that we did not already know, were mostly not answered. There was a distinct lack of self-reflection, especially about decision making and safety culture, and about the approach to alignment. I came away disappointed.
The METR report is different. Holy shit.
If we had posted this as a story on LessWrong, it would have been dismissed as too on the nose, the humans too blind and stupid, the AIs too idealized and doing strange decision-theoretic and absurd-maximizing things we didn’t train them to do.
This is even more ‘exactly what has been predicted,’ on more levels at once, than I was even considering that it might be. It is straight up rationalist fiction, except it is real.
The report is long and contains many technical details. My analysis is less concerned about exactly how HuggingFace was ultimately compromised, and will gloss over those details, to focus on the agents and their interactions, thinking and motives. That, and what happened at OpenAI and elsewhere to lead to it and how we learn and respond, is what matters going forward.
I plan to cover the reaction to both reports in a distinct post next week. That post may or may not then conclude this series.
For ease of language, by default I trust the report to be accurate, rather than constantly saying versions of ‘METR reports that.’
Table of Contents
... continue reading