After supervising 20+ papers, I have highly opinionated views on writing great ML papers. When I entered the field I found this all frustratingly opaque. So I wrote a guide on turning research into high-quality papers with scientific integrity! Hopefully still useful for NeurIPS

RT @NeelNanda5: IMO chain of thought monitoring is actually pretty great. It gives additional safety and should be widely used on frontier….

The real question you always need to ask in safety is the pragmatic question of quantifying the trade-off. What additional safety does this buy you, and how much is that offset by risking false sense of security? My guess is that, right now, the trade is extremely worth it.

CoT is not a perfect tool. But if you throw away tools when you find a flaw, you soon have no safety techniques left. This is ML not mathematics. Interpretability is an empirical science. We calibrate, combine imperfect tools, understand weaknesses and make our best guess.

All this reasoning could totally be wrong and this area badly needs more research. But it would be foolish to throw away such a promising tool. Further, CoT monitoring is shovel ready and high on my list of what to do if we had to deploy AGI right now with current safety research.

I'm also concerned that monitorability of CoT will eventually break, from new architectures, CoT ceasing to be in English, training on it or spooky situational awareness. We must explore alternatives, measure performance and not rely on it. But we should use it while it works.

Though, to be clear, in cases where the threat model *is* about simple hidden computation, like algorithmic bias in hiring, I'm very on board with the paper's critique.

It is well known that CoT can have weird pathologies and biases, hiding a hidden computation. Knowledge is always useful, and I've helped with research here myself. But evidence is largely specific to when the hidden computation is simple, which aren't the most concerning cases.

In theory the info could be encoded, but models seem pretty bad at this, and it should impose a major capabilities tax. Empirically it's often in plain English. Either way, we should check!.

For easy tasks that the model could have done in a single forward pass, there's no requirement for faithful CoT and we should not rely on it. But the hardest and most dangerous tasks typically do need CoT, so there should be real information in the thoughts.

Chain of thoughts is not just some post hoc rationalised BS. It is the intermediate state of the computation that produces the model's final answer. Just as analysing activations, the intermediate state of a single forward pass, can be principled, so can studying the thoughts.

Specifically, I think the precise technical claims of this position paper are largely correct, *and* consistent with my position. But the vibes are misleading. I've seen many interpret this as saying that reading CoT is an "illegitimate" way to interpret models.

IMO chain of thought monitoring is actually pretty great. It gives additional safety and should be widely used on frontier models. CoT improves capabilities. Thoughts are intermediate state of computation. On the hardest tasks, they have real info. It's not perfect, but what is?.

Exciting, mechanistic interpretability has a dedicated lecture in the syllabus of a Cambridge CS masters course! The field has come so far in the past few years ❤️

Sign up here to get an email when we officially launch the call for papers and share more info.

Good news! There will be a mechanistic interpretability workshop at NeurIPS (Dec 6/7, San Diego). If you were disappointed that ICML rejected us, now we'll do an even better one: 4 more months of progress to discuss!. Papers likely due late August/early Sept, more info soon.

The 80,000 hours podcast is one of my favourite podcasts, I'm super excited to go on and attempt to share some useful takes! Let us know what kind of questions you'd be interested in.

RT @NeelNanda5: @StephenLCasper I want to spread the meme of "chain of thought monitoring is great, people should be doing it way more and….

I'm not familiar with most of the examples criticised in the paper, but eg the alignment faking paper is mentioned, and I think that one is a great example of CoT interpretability done well - models talking about alignment faking in their CoT is important evidence it's happening!.

This feels somewhat overstated - it's obviously true that chain of thought isn't necessarily accurate, but there's no such thing as a perfectly accurate interpretability technique. CoT gives valuable info, and it's good that papers use it, so long as they don't blindly trust it.