We’re releasing an open-source library and public interactive interface for tracing the internal “thoughts” of a language model. Now anyone can explore the inner workings of LLMs — and it only takes seconds!

RT @chingfang17: Humans and animals can rapidly learn in new environments. What computations support this? We study the mechanisms of in-co….

You can also find a detailed tutorial on the code, including worked examples and analyses, at Huge thanks to @michaelwhanna and @mntssys for developing the library, @johnnylin for the Neuronpedia integration, and @mlpowered for orchestrating!.

This release implements the method - attribution graphs - from our recent paper, where we uncovered plans, goals, and complex circuits inside Claude. It supports  Gemma 2-2B + Llama 3.2-1B, and can be extended to other models. Try making a graph at .

RT @AnthropicAI: Our interpretability team recently released research that traced the thoughts of a large language model. Now we’re open-s….

RT @ch402: The Anthropic Interpretability Team is planning a virtual Q&A to answer Qs about how we plan to make models safer, the role of t….

RT @Butanium_: New paper w/@jkminder & @NeelNanda5! What do chat LLMs learn in finetuning?. Anthropic introduced a tool for this: crosscode….

RT @emollick: There’s at least a dozen dissertations to be written from this paper by Anthropic alone, which gives us some insight into how….

Great summary of a fun finding from our paper.

Come join us! Especially colleagues from my past life in neuroscience — working on this stuff feels like dissecting a weird alien brain, except we get to run experiments at 100X speed. It’s so exciting!.

It turns actually *using* these tools to navigate the weird, messy guts of the model is a whole new challenge, requiring different skills and perspectives. I expect people with experience or interest in biology to play an important role. .

5. Understanding how AI works is in many ways a biology problem. A lot of work focuses on building mathematical and computational tools for inspecting models. We need those tools, like biologists need microscopes. But that’s just the first step….

4. We can’t trust the models to tell us how they work – they may not know themselves! My favorite example: we figured out the rough algorithm Claude uses to add two numbers. Then we asked it how it adds two numbers. The answer it gave was… not what it actually does!.

3. Other times, they seem alien, and their mind oddly “fractured.” E.g. you can get Claude to start telling you how to make a bomb without “realizing” what it’s saying. Even when it does realize, its commitment to finishing sentences outweighs its reluctance to do harm. Weird!.

2. The steps models use often feel very… thoughtful. Planning responses ahead of time, representing goals, considering multiple possibilities at once. Sometimes they are alarmingly clever - in one case we see the model work backwards from a predetermined answer to justify it.

For me, there are five profound takeaways from this work. 1. Modern language models’ internal computation can be decomposed into steps that we can make some sense of. Our tools are imperfect and miss a lot – but there’s a lot we can see and understand already!.

Human thought is built out of billions of cellular computations each second. Language models also perform billions of computations for each word they write. But do these form a coherent “thought process?”. We’re starting to build tools to find out! Some reflections in thread.

I participated in this as an auditor, poking around in an LLM's brain to find its evil secrets. Most fun I've had at work! Very clever + thoughtful work by the lead authors in designing the model + the game, which set a precedent for how we can validate safety auditing techniques.

RT @leedsharkey: Big new review! . 🟦Open Problems in Mechanistic Interpretability🟦. We bring together perspectives from ~30 top researchers….

If you’re interested in interpretability of LLMs, or any other AI safety-related topics, consider applying to Anthropic’s new Fellows program! Deadline January 20, but applications are reviewed on a rolling basis so earlier is better if you can. I’ll be one of the mentors!.