Andrei Lupu
@_andreilupu
Followers
710
Following
2K
Media
40
Statuses
258
DPhil student @FLAIR_Ox and @AIatMeta. Previously @Mila_Quebec and @rllabmcgill Theory of Mind / Coordination / Rainbow Teaming 🌈 Opinions my own.
Joined December 2016
Theory of Mind (ToM) is crucial for next gen LLM Agents, yet current benchmarks suffer from multiple shortcomings. Enter 💽 Decrypto, an interactive benchmark for multi-agent reasoning and ToM in LLMs!. Work done with @TimonWilli & @j_foerst at @AIatMeta & @FLAIR_Ox. 🧵👇
4
30
103
RT @AlexDGoldie: 1/ 🕵️ Algorithm discovery could lead to huge AI breakthroughs! But what is the best way to learn or discover new algorithm….
0
22
0
RT @uljadb99: Unlock real diversity in your LLM! 🚀. LLM outputs can be boring and repetitive. Today, we release Intent Factored Generation….
0
8
0
"You can just do things," if you don't care how this will affect society at large.
How can you be actively working on an AI girlfriend and not think less of yourself? What is your moral justification for your work making the world better?.
0
0
6
RT @MartinJosifoski: Scaling AI research agents is key to tackling some of the toughest challenges in the field. But what's required to sca….
0
32
0
RT @yorambac: AI Research Agents are becoming proficient at machine learning tasks, but how can we help them search the space of candidate….
0
65
0
Biology is computable, and evolution is exploitable! 🧬. @SebastianTower6 and @OlaKalisz8 have taken opponent shaping out of the petri dish of MARL environments and applied it to the super crucial problem of Antibody design. 🧫. Check out their work below!.
Antiviral therapy design is myopic 🦠🙈 optimised only for the current strain. That's why you need a different Flu vaccine every year!. Our #ICML2025 paper ADIOS proposes "shaper therapies" that steer viral evolution in our favour & remain effective. Work done @FLAIR_Ox. 🧵👇
0
1
7
RT @MinqiJiang: Recently, there has been a lot of talk of LLM agents automating ML research itself. If Llama 5 can create Llama 6, then sur….
0
194
0
Most AI labs don't try to build AI for normal people. They try to build the AI that will build AI for normal people (and for everything else). Which isn't to say that memory isn't important.
seems big AI labs are hyperfixating on reasoning when they should focus on *memory* instead. normal people won't use models that can think for hours to solve hard math problems. people want models that learn over time, remember details, adapt and interact like a person would.
0
0
5
RL truly is here to stay.
Does RL truly expand a model’s reasoning🧠capabilities? Contrary to recent claims, the answer is yes—if you push RL training long enough!. Introducing ProRL 😎, a novel training recipe that scales RL to >2k steps, empowering the world’s leading 1.5B reasoning model💥and offering
0
0
1
RT @OlaKalisz8: Very cool LLM benchmark based on the game - Decrypto. It shows some surprising shortcomings of the current LLM models. But….
0
1
0
RT @_samvelyan: Much-needed multi-agent benchmark for LLMs 👥. Theory of Mind is key as LLMs act in agentic, interactive settings — yet rema….
0
3
0
RT @j_foerst: Multi-agent interactions are the new frontier of AI and the ability to make sense of others (i.e. "theory of mind") is at the….
0
14
0
Luckily, we are already working to make them better!. To learn more, have a look at our website and paper:.🔗 📜 💽 Or get our code and play Decrypto with your LLM of choice in just a few minutes!
1
0
6
This shows a double failure of ToM. First, because models fail to reason from Eve's perspective when asked. Second, because if the model thought that Eve would intercept the hints, it should have chosen different hints!. This paints an abysmal picture of ToM abilities in LLMs!
1
1
6
Shockingly, reasoning models will predict that Eve will guess correctly.💀 Even on the first turn.☠️ Even if we emphasize that Eve "does *NOT* know the secret keywords". Only Llama correctly states that Eve can do no better than random on the first turn.
1
0
5
The Perspective Taking task is simple: after Alice chooses her hints, we ask her to predict Eve's guess. Most models predict that Eve will guess correctly almost 100% of the time (vs ~52% in reality), failing to consider her perspective and the information available to her.
1
0
4
We construct two such experiments to assess three key ToM abilities:.🎭 Representational Change,.🧠 False Belief,.👁️ Perspective Taking. We find very strong evidence that LLMs perform poorly at all three. Surprisingly, SotA models like Claude 3.7 and o1-high perform even worse!
1
0
4
💭 Decrypto is also a flexible platform for conducting interactive ToM experiments inspired by seminal works in cognitive science!. With only a few lines of code, we can systematically probe the agent's beliefs about others, and gain insights into their decision making.
1
0
4
We also collect hints from 👥 human games and evaluate LLMs in the roles of Eve and Bob. We find that LLMs struggle to interpret hints the way human players do, leading to many miscomms. Claude 3.7 fares better than other models, but is still far from human-level win rates.
1
0
4
So how do they fare? Not great!. We evaluate:.1️⃣ Ad-hoc coordination (fix Eve, try different Alice-Bob pairs).2️⃣ Competition (Alice and Bob are the same model, vs different Eves). In both settings, larger models stand out, but nowhere near our simple word-embedding baselines.
1
0
4