🤖 !! Model-free agents can internally plan !! 🤖. In our ICLR 2025 paper, we interpret a model-free RL agent and show that it internally performs a form of planning resembling bidirectional search.

Attending #ICLR2025 to present this research - please reach out if you want to chat about interp, reasoning, or anything else!!.

RT @_fernando_rosas: Preprint time:.“AI in a vat: Fundamental limits of efficient world modelling for agent sandboxing and interpretability….

This research would not have been possible without the amazing support and mentorship I recieved from my brilliant co-authors @Stephen36910351 , @usmananwar391 , @AdriGarriga and @DavidSKrueger. Paper: Blog post:

This research would not have been possible without the amazing support and mentorship I recieved from my brilliant co-authors @Stephen36910351 , @usmananwar391 , @AdriGarriga and @DavidSKrueger . Paper: Blog post:

For instance, in the example below, the agent initially plans to push the upper-left box to the center-most by pushing it down and then right. However, the agent realises this is infeasible – once the box is been pushed down, it is stuck – and then forms an alternate plan.

Furthermore, the agent has learned something akin to the “a-ha!” moments that have been exhibited by RL-tuned LLMs like DeepSeek-R1. That is, the agent often (1) forms initial plans, (2) recognize flaws in its these plans, and then (3) update its initial plans accordingly.

For instance, in planning forward from boxes and backward from targets, the DRC agent has learned an algorithm that is especially efficient in Sokoban. We think this explains its ability to outperform model-based agents (whose planning depends on handcrafted elements) in Sokoban.

Why does all of this matter? Because, it shows that generic model-free training can give rise to the emergence of advanced, complex reasoning capabilities, even in tiny agents!.

We also find that these planning representations emerge concurrently with the agent's ability to perform better when given time to "think" at the start of episodes.

We find that we can intervene on the agent’s activations to force it to form and execute alternate plans, changing its behaviour over entire episodes. For instance, below, we intervene to cause the agent to form and execute a sub-optimally long plan.

In the final step of our analysis, we investigate whether the internal plan representations we uncover are causally linked to the agent's behaviour.

We even find that, when forced to pause and "think" at the start of episodes, the agent's internal plan iteratively improves. This ability to iteratively improve plans explains the phenomenon noted above where the DRC agent solves additional levels when given time to "think"!

How do these internal plans form? We find that the agent appears to form internal plans using a fully-learned, Sokoban-specific procedure that involves simultaneously constructing multiple plans iteratively forward from boxes and backwards from targets.

For instance, here, teal and purple arrows indicate that a linear probe decodes that the agent plans to step onto, or push a box off of, a square in the respective direction. Note this internal plan formed by the agent corresponds to a complete plan to solve the level!

We find that probes can decode, from the agent's activations, plans that the agent formulates in terms of the above two concepts!.

In the second step of our analysis, we use our linear probes to investigate whether the agent uses its representations of these concepts to form internal plans within its activations.

We find that linear probes can correctly predict the directional classes assigned to each square (i,j) of an observed Sokoban board using only the agent's cell state activations at position (i,j).

First, we investigate whether the agent internally represents two concepts that could be used for planning. These concepts capture (1) how the agent navigates the board, and (2) how the agent pushes boxes. These concepts assign direction classes to each square of Sokoban boards.

We take a concept-based interpretability approach to investigating whether this behaviour is the result of the DRC agent internally planning. In doing so, we perform three steps of analysis.

Specifically, we study a Sokoban-playing DRC agent as introduced by Guez et al. (2019). DRC agents are generic recurrent model-free agents that behave as though they perform planning. For example, DRC agents solve extra Sokoban levels when forced to "think" prior to acting.