Life update: I’ve joined the UK @AISafetyInst on the Safeguards Analysis team! We’re working on how to conceptually and empirically assess whether a set of safeguards used to reduce risk is sufficient.

🔒How can we prevent harm from AI systems that pursue unintended goals? AI control is a promising research agenda seeking to address this critical question. Today, we’re excited to launch ControlArena – our library for running secure and reproducible AI control experiments🧵

New paper! Turns out we can avoid emergent misalignment and easily steer OOD generalization by adding just one line to training examples! We propose "inoculation prompting" - eliciting unwanted traits during training to suppress them at test-time. 🧵

New @AISecurityInst research with @AnthropicAI + @turinginst: The number of samples needed to backdoor poison LLMs stays nearly CONSTANT as models scale. With 500 samples, we insert backdoors in LLMs from 600m to 13b params, even as data scaled 20x.🧵/11

Well... damn. New paper from Anthropic and UK AISI: A small number of samples can poison LLMs of any size. https://t.co/STAMpLpbck

We’ve conducted the largest investigation of data poisoning to date with @AnthropicAI and @turinginst. Our results show that as little as 250 malicious documents can be used to “poison” a language model, even as model size and training data grow 🧵

Regardless of model size and amount of clean data, backdoor data poisoning attacks on LLMs only need a near-constant number of poison samples! Exciting to put out this research from AISI in Collab with Anthropic and ATI. Lots of work on detection and mitigation to be done here!

We’ll be continuing our work with Anthropic and other frontier developers to strengthen model safeguards as capabilities improve. Read more on our blog:

✨New AI Safety paper on CoT Monitorability✨ We use information theory to answer when Chain-of-Thought monitoring works, and how to make it better.

We at @AISecurityInst recently did our first pre-deployment 𝗮𝗹𝗶𝗴𝗻𝗺𝗲𝗻𝘁 evaluation of @AnthropicAI's Claude Sonnet 4.5! This was a first attempt – and we plan to work on this more! – but we still found some interesting results, and some learnings for next time 🧵

If you're excited by this kind of work, and more broadly ensuring AI systems are secure, AISI are hiring a bunch of roles: https://t.co/SJTVYEjVFQ Incl. a research scientist role on the safeguards team where you can work on red-teaming frontier AI systems

As @sleepinyourhat says, there is a lot more work to do developing the methodology and tooling for alignment evaluations, and we're excited to be pushing forwards these research.

This was a big cross-team collaboration at @AISecurityInst – we also did additional ablations of the alignment evaluations, and evaluated capabilities relevant for misalignment threat models, and it was very cool to see everything come together for this on fairly short notice!

We also took a lot of inspiration from @summerfieldlab's "Lessons from a Chimp" paper when running these evaluations to ensure the results and interpretation were more rigorous and grounded (we now regularly use the phrase "is this eval chimp-proof?").

Learnings: We built lots of automated evals using LLMs, building on a tool shared w/ us by Anthropic. Prompting is crucial, and nothing substitutes for manually reading transcripts to understand what's going on (luckily we have @AlexandraSouly & others who're crazy good at this)

We observed both of these results without being told to look for them by @AnthropicAI, which is a good sign for being able to independently study these models and observe their behaviour

In synthetic agentic scenarios, we also found Sonnet 4.5 took actions to preserve its current state 1.9x less than Sonnet 4 and 3.3x than to Opus4.1. Sonnet 4.5 generally justified these actions on grounds other than self-preservation (such as security concerns – see screenshot)

Results: We found Sonnet 4.5 talks about being in an eval more than Opus 4.1 and Sonnet 4 (16.6% vs 3.5% & 1.6% respectively) – although it may not be better at distinguishing eval from deployment when prompted. Eval awareness will likely be a very big issue going forward!

We at @AISecurityInst recently did our first pre-deployment 𝗮𝗹𝗶𝗴𝗻𝗺𝗲𝗻𝘁 evaluation of @AnthropicAI's Claude Sonnet 4.5! This was a first attempt – and we plan to work on this more! – but we still found some interesting results, and some learnings for next time 🧵

https://t.co/wRv2N3xccR

Thrilled to talk about these collabs btwn UK AISI/US CAISI, Ant and OAI, and share our learnings on what makes this kind of testing effective! This work both improves the security of these systems and keeps govt. informed on the state of safeguards, so is super valuable.