Brand new paper with @johnjhorton that I'm very excited to share: "General Social Agents". Suppose we wanted to create AI agents for simulations to make predictions in never-before-seen settings. How might we do this? We explore an approach to answering that question!

RT @soumitrashukla9: This is a very thought-provoking and novel paper. Highly recommend reading it!. (link in replies) .

Turn old photos into videos and see friends and family come to life. Try Grok Imagine, free for a limited time.

RT @MichaelEddy: Impact-focused funders often ask: if it worked here, will it work there? . This paper is a small, but impt step toward a g….

RT @emollick: This is a fascinating paper that suggests that AI agents can indeed be used for social science experiments, but that just usi….

RT @johnjhorton: .@BenSManning preparing big twitter thread about our new paper; tags wrong John Horton. i'm dying.

RT @johnjhorton: 🚨New working paper! 🚨. In a nutshell, it's about how you might create "general" agents in the sense that their behavior wo….

RT @mdahardy: this is a really cool paper.

RT @matthewclifford: This is very, very interesting….

RT @tylercowen: This and related work will revolutionize several different fields of economics:.

cc: @tylercowen @emollick.

Feedback is still very much welcome! Agents and games are available in hyperlinks throughout the paper. Link:

Optimized agents predict the human responses far better than an off-the-shelf baseline LLM (3x) and relevant game-theoretic equilibria (2x). In 86% of the games, all human subjects chose a strategy in support of the LLM simulations; only 18% were in support of the equilibria.

The sampled games are highly diverse with a striking range of equilibrium distributions. Some equilibria spread probability across many options, while others concentrate only on extremes. A are almost uniform, while others exhibit sharp spikes at specific actions.

We then put the strategic and optimized agents to an extreme test. We created a population of 800K+ novel strategic games, sampled 1500, which the agents then played in 300,000 simulations. But we first have 3 humans (4500 total) play each game in a pre-registered experiment.

When we test the approach on agents motivated by atheoretical and scientifically meaningless prompts, they do no better at predicting the human responses in the novel games than the LLM off-the-shelf (even when they have good training performance).

Next, we design 4 new games and have humans play them in preregistered experiments. Optimized agents are very accurate predictors of humans - far better than the baseline LLM. In some games, AI sims predict human responses better than relevant human data from Arad & Rubinstein.

When we have these optimized agents play two new related, but distinct games, the optimized set performs well in matching these out-of-sample human distributions. The off-the-shelf LLM still performs poorly.

For the 11-20 money request game, the theory is level-k thinking, and the seed game is the human responses from the original paper. We construct a set of candidate agents based on a model of level-k thinking and then optimize them to match human responses with high accuracy.

However, we can do better—even in settings outside the underlying LLM’s training data. The idea, in a nutshell, is to use a "seed" game & some relevant theory to create AI agents that can then be used in conceptually related settings.

We asked GPT-4o to play the 11-20 money request game from Arad & Rubinstein (2012) 1000 times without additional instructions. Its responses are FAR from the human distribution.

Link to the paper: Off-the-shelf, LLMs are often poor predictors of human responses. To give an example, consider the 11-20 money request game from Arad & Rubinstein (2012).