GoKawiilCredit: Art Potts
Last week, I attended a workshop on Chain of Thought (i.e., reasoning chain) monitorability. Many of the most influential researchers and “members of technical staff” working on this topic were present. I felt lucky to be there, since pretty much everything I know about CoT monitoring I learned from Peter Hase. I have, however, worked extensively on interpretability, and there seems to be a general hope that interpretability will improve or complement CoT monitoring, though this relationship is complicated, as I discuss below.
I learned a lot at the workshop and came away with many things to think about. This post seeks to summarize and systematize the main takeaways for me as a sort of outsider to the community:
CoT and monitoring: An uneasy and uneven alliance
The general consensus of the workshop group seemed to be that it is a stroke of luck that CoT provides a basis for monitoring, because it wasn’t invented for that purpose.
CoT reasoning was developed primarily to make models better at hard tasks. This hypothesis traces to the classic paper introducing CoT as a prompting technique: Wei et al. 2022. The usual story is that present-day LLMs have a fixed depth, and so their capacity to perform computations before producing an output is highly limited. By allowing models to produce tokens before their final answer, we enrich them with a more open-ended form of computation. For Wei et al., CoT is purely a prompting technique, but it was elevated to a post-training process with the release of OpenAI’s o1 model in September 2024.
Though Wei et al. focus on performance improvements, they note that CoT “provides an interpretable window into the behavior of the model”, with the added caveat that “fully characterizing a model’s computations that support an answer remains an open question”. This became a full program for monitoring in the agenda paper Korbak, Balesni, et al. 2025, Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety. (For discussion of this paper’s origins and predecessors, see this post by Peter Hase and me.)
The phrase “fragile opportunity” from Korbak et al.’s title seems to capture the spirit of their proposal. CoT has stood the test of time because it leads to capability improvements, not because it increases transparency. Furthermore, the two goals are not necessarily aligned. Some recent papers (Kirchner, Chen, et al. 2024; Guo et al. 2025) have even suggested that transparency can be a tax on the effectiveness of CoTs for models. One does not have to be especially cynical about the business of AI to conclude that, if push comes to shove, progress on capabilities will win out over increased transparency.
On balance, I agree that it is useful to be able to monitor CoT. However, the fragility of this alliance makes me feel that we should reduce our dependence on CoT for safety, to plan for a future in which CoT isn’t useful. The next two sections of this post provide additional motivation for this conclusion.
✳
... continue reading