I'm excited to join @AnthropicAI to continue the superalignment mission!. My new team will work on scalable oversight, weak-to-strong generalization, and automated alignment research. If you're interested in joining, my dms are open.

The authors of this paper say this; I'm just more pessimistic than them about how useful this will be.

Lots of egregious safety failures probably require reasoning, which is often hard for LLMs to do without showing its hand in the CoT. Probably. Often. A lot of caveats.

To be clear: CoT monitoring is useful and can let you discover instances of the model hacking rewards, faking alignment, etc. But absence of bad "thoughts" is not evidence that the model is aligned. There are plenty of examples of prod LLMs having misleading CoTs.

If you don't train your CoTs to look nice, you could get some safety from monitoring them. This seems good to do!. But I'm skeptical this will work reliably enough to be load-bearing in a safety case. Plus as RL is scaled up, I expect CoTs to become less and less legible.

RT @sleepinyourhat: 🧵✨🙏 With the new Claude Opus 4, we conducted what I think is by far the most thorough pre-launch alignment assessment t….

If you want to read more about Sonnet & Opus 4, including a bunch of alignment and reward hacking findings, check out the model card:.

It's also (afaik) the first ever frontier model to come with a safety case – a document laying out a detailed argument why we believe the system is safe enough to deploy, despite the increased risks for misuse.

These mitigations address jailbreaking for catastrophic misuse and are based on the constitutional classifiers we demoed earlier this year 👇.

So many things to love about Claude 4! My favorite is that the model is so strong that we had to turn on additional safety mitigations according to Anthropic's responsible scaling policy.

RT @yanda_chen_: My first paper @AnthropicAI is out!. We show that Chains-of-Thought often don’t reflect models’ true reasoning—posing chal….

Somewhat surprising that faithfulness of chain-of-thought doesn't improve much with outcome-based RL.

Check out the blog post for details 👇.

My impression is the future bottleneck will be fixing misalignments, not discovering them:. There are a few recent examples of badly misaligned models (alignment faking, backdoored code -> evil model, reward hacks); better solutions are needed, especially for superhuman models!.

As models get smarter, they will get better at hiding their misalignments, so this is no guarantee that we'll be able to uncover important misalignments in the future. But our techniques for finding them will improve as well!.

I was quite surprised how successful the "auditing" teams were at finding the hidden misalignments and how many different strategies worked. A promising application of current interpretability techniques!.

The misaligned model is trained to satisfy a list of reward model biases, but hide that it is trying to do so. Some of the reward model biases were only described to it in text and not reinforced with RL. Interestingly, the model still generalized to perform those behaviors.

Could we spot a misaligned model in the wild?. To find out, we trained a model with hidden misalignments and asked other researchers to uncover them in a blind experiment. 3/4 teams succeeded, 1 of them after only 90 min.

More details in the updated blog post:.

We’re very grateful to everyone who helped test our system’s jailbreaking robustness and contributed to our safety research. Y’all are awesome :).

As models get more capable, robustness to jailbreaks becomes a key safety requirement to protect against misuse related to chemical, biological, radiological, and nuclear risks. Our demo showed classifiers can help mitigate these risks, but need to be combined with other methods.