Registration closes on July 4! Come join us in-person or virtually in Amsterdam!.

Dutch academics, please share - replication games are coming to Vrije Universiteit Amsterdam on July 19! If you live in the Netherlands, or want a good reason to visit, then come network, hone your skills, and potentially coauthor a paper. Registration:

RT @I4Replication: #GDRI_rep Update 7: Retraction! Our comment of "Parent–teacher meetings and student outcomes: Evidence from a developing….

RT @I4Replication: #GDRI_rep Update 5: We have a new report. We reproduced the paper entitled "Partisan Effects of Information Campaigns in….

RT @I4Replication: #GDRI_rep Update 4: We have a new report. We reproduced the paper entitled "Parent–teacher meetings and student outcomes….

Thank you @carlislerainey for boosting my job market paper! I’m a fan of your prior work on this topic. If you want to learn more, check out the paper and my BIBAP seminar on it tomorrow, 3/12 7 AM CET/5 PM Sydney time.

RT @JohnHolbein1: Here's another recent critique of close elections RDDs. It's unclear whether there is actually good evidence against pot….

I'll be waking up early (7 AM CET) on Tuesday, March 12 to present my job market paper at 5 PM Sydney time! If you're awake too, stop by to hear me talk about equivalence testing, replication-based methods research, and the robustness of null results in economics!

I also develop new methods for testing running variable (RV) manipulation in RDD (similar to Hartman 2021). Using the Stommes et al. (2023) data, I find >44% of RV density discontinuities at treatment cutoffs can't be bounded beneath 50% upward jumps. 🧵:

RT @I4Replication: #GDRI_rep Update 2: I4R sent a report to the original authors for the PLOS One article "“Food insecurity and mental heal….

RT @I4Replication: #GDRI_rep Update 1: I4R are now reproducing all published papers that use data from GDRI (, or a….

RT @paulnovosad: Big kudos to this team for doing the work of uncovering what looks like large-scale research malpractice. Some highlights….

As a coauthor on the AEJ:AE report, it’s a lot. Oh my god, It’s a lot. In that paper, main outcomes are inconsistently handled both in the code and in the field, the paper’s data is connected to a bunch of other experiments, and we find irregularities in the raw survey files.

I’m a fan of LLMs and use them to program all the time, but some general lessons here. 1: Don’t trust an LLM’s reproduction of a paper; do it yourself. 2: Humans are necessary to ensure reproducibility. 3: For the love of all that is holy, please double-check LLMs’ output. 7/7.

We weren’t alone. Reproducibility rates for AI-led teams were over 57 p.p. lower than those for AI-assisted teams, and over 59 p.p. lower than those for human teams. AI-led teams also detect fewer major errors and propose/implement fewer good robustness checks. 6/7.

This replication package had *annotated code*. We gave ChatGPT data + scripts that *label the code for Figure 4*, and then asked it to reproduce Figure 4. It didn’t even get close to reproducing Figure 4, even after a mind-melting amount of effort put into prompt engineering. 5/7.

Hours were spent trying to get ChatGPT to write code that would produce any plot at all. Eventually, we abandoned R for Python. We switched ChatGPT terminals, considering our current terminal to be a lost cause. We still couldn’t reproduce the figure by the end of the day. 4/7.

Our team could only upload the paper and its replication package into ChatGPT along with an image of the figure we wanted to reproduce, try to get ChatGPT (o1 for our Games) to write code that would reproduce the figure, and then run this code in R. This was ✨misery✨. 3/7.

In AI Replication Games, we were randomized into human, cyborg (AI-assisted), + machine (AI-led) teams, all trying to reproduce a published paper. Humans couldn't use ChatGPT, whereas cyborgs could use it as much/as little as desired. My machine team could *only* use ChatGPT. 2/7.

New working paper! @I4Replication does a good job detailing the experiment’s results, so for anyone who considers LLMs/AI as a (soon-to-be) solution to reproducibility in social science, let me walk you through the 🔥hell🔥 of trying to reproduce a paper on an "AI-led" team. 1/7.