Tenure letter write: I completely disagree with all of his work but it’s good work

AI presents a fundamental threat to our ability to use polls to assess public opinion. Bad actors who are able to infiltrate panels can flip close election polls for less than the cost of a Starbucks coffee. Models will also infer and confirm hypotheses in experiments. Current

We also developed a new R package, localLLM ( https://t.co/hijpvkPT3T), that enables reproducible annotation using LLM directly in R. More functionalities to follow!

Based on these findings (and more in the paper), we offer recommendations for best practices. We also summarized them in a checklist to facilitate a more principled procedure.

Finding 4: Bias-correction methods like DSL can reduce bias, but they introduce a trade-off: corrected estimates often have larger standard errors, requiring a large ground-truth sample (600-1000+) to be beneficial without losing too much precision.

Finding 3: In-context learning (providing a few annotated examples in the prompt) offers only marginal improvements in reliability, with benefits plateauing quickly. Changes to prompt format has a small effect (smaller and reasoning models more sensitive).

Finding 2: This disagreement has significant downstream consequences. Re-running the original analyses with LLM annotations produced highly variable coefficient estimates, often altering the conclusions of the original studies.

There is also an interesting linear relationship between LLM-human and LLM-LLM annotation agreement: when LLMs agree more with each other, they also tend to agree more with humans and supervised models! We gave some suggestions on what annotation tasks are good for LLMs.

Finding 1: LLM annotations show pretty low intercoder reliability with the original annotations (coded by humans or supervised models). Perhaps surprisingly, reliability among the different LLMs themselves is only moderate (larger models better).

The LLM annotations also allowed us to present results on: 1. effectiveness of in-context learning 2. model sensitivity to changes in prompt format 3. bias-correction methods

We re-annotated data from 14 published papers in political science with 15 different LLMs (300 million annotations!). We compared them with the original annotations. We then re-ran the original analyses to see how much variation in coefficient estimates these LLMs give us.

New paper: LLMs are increasingly used to label data in political science. But how reliable are these annotations, and what are the consequences for scientific findings? What are best practices? Some new findings from a large empirical evaluation. Paper: https://t.co/F8FlrsLbzM

Happy to say that I’ll be a postdoc @KelloggSchool w/ @dashunwang and join @purduepolsci as an assistant professor! Stoked and grateful to continue my study on the politics of science and AI

In between cat videos @cmcclean just popped up on my TikTok?! This gotta be the biggest flex

A paper entitled "The impact of US–China tensions on US science: Evidence from the NIH investigations" by Ruixue Jia @rxjia, Margaret Roberts @mollyeroberts @GPS_UCSD, Ye Wang @YeWang1576, Eddie Yang @EddieYang_ @UCSanDiego is published @PNASNews! https://t.co/zjWxEZJKkX

Supporting models on and off the stage

More expensive to watch Caitlin Clark than the Chicago Bulls😬 at what point in my career can I bring up season ticket with the chair?

😅

Every once in a while I come across one of these gems by Henry Farrell. This one published in 2018 and perfectly describing the moment we're living in now. https://t.co/D0IemyUZ8w

I’m in Washington DC this year. If you’re around and want to grab coffee, let me know!