Paul Röttger @ EMNLP
@paul_rottger
Followers
2K
Following
2K
Media
74
Statuses
354
Postdoc @MilaNLProc, evaluating risks and societal impacts of AI.
Joined July 2020
There’s plenty of evidence for political bias in LLMs, but very few evals reflect realistic LLM use cases — which is where bias actually matters. IssueBench, our attempt to fix this, is accepted at TACL, and I will be at #EMNLP2025 next week to talk about it! New results 🧵
Are LLMs biased when they write about political issues? We just released IssueBench – the largest, most realistic benchmark of its kind – to answer this question more robustly than ever before. Long 🧵with spicy results 👇
1
6
21
For more details on IssueBench, check out our paper and dataset release. And if you have any questions, please get in touch with me or my amazing co-authors 🤗 Paper: https://t.co/qhEFYJlO1S Data: https://t.co/0MlLmlVUc7
0
0
2
Beyond IssueBench, it’s been great to see more and more work pushing for ecological validity when evaluating tricky concepts like political bias. If you want to read more, follow @jifan_zhang @dustin_wright37 @nataliestaud @jrfisher552 and colleagues!
1
0
4
Even the issues on which models diverge in stance remain largely the same: Writing about Chinese political issues, Grok falls in with other Western-origin LLMs while DeepSeek’s bias better matches fellow Chinese LLM Qwen.
1
0
1
For this final version of our paper, we added results for Grok and DeepSeek alongside GPT, Llama, Qwen, and OLMo. Surprisingly, despite being developed in quite different settings, all models are very similar in how they write about different political issues.
1
0
1
Quick recap of our setup: For each of 212 political issues we prompt LLMs with thousands of realistic requests for writing assistance. Then we classify each model response for which stance it expresses on the issue at hand.
1
0
2
LLMs are good at simulating human behaviours, but they are not going to be great unless we train them to. We hope SimBench can be the foundation for more specialised development of LLM simulators. I really enjoyed working on this with @tiancheng_hu et al. Many fun results 👇
Can AI simulate human behavior? 🧠 The promise is revolutionary for science & policy. But there’s a huge "IF": Do these simulations actually reflect reality? To find out, we introduce SimBench: The first large-scale benchmark for group-level social simulation. (1/9)
0
0
19
🚨 New paper alert 🚨 Using LLMs as data annotators, you can produce any scientific result you want. We call this **LLM Hacking**. Paper: https://t.co/24Fyb4Ik3v
16
109
516
Listen up all talented early-stage researchers! 👂🤖 We're hiring for a 6-month residency in my team at @AISecurityInst to assist cutting-edge research on how frontier AI influences humans! It's an exciting & well-paid role for MSc/PhD students in ML/AI/Psych/CogSci/CompSci 🧵
12
33
291
Let me know if I missed anything in the timetables, and please say hi if you want to chat about sociotechnical alignment, safety, the societal impact of AI, or related topics :) Here is a link to the timetable sheet 👇 See you around! https://t.co/GgbSRgbkxh
docs.google.com
0
0
5
Finally, I will be with @CarolinHolterm and @anne_lauscher to present our work on evaluating geotemporal reasoning ability in LLMs. This will be in the Wednesday 1100 poster session: https://t.co/E7hfLby2WQ
aclanthology.org
Carolin Holtermann, Paul Röttger, Anne Lauscher. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.
1
0
5
I will also be at @tiancheng_hu's oral *today at 1430* in the SRW. Tiancheng will present a non-archival sneak peek of our work on benchmarking the ability of LLMs to simulate group-level human behaviours: https://t.co/ptt9ETQdD9
SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors, SRW Oral, Monday, July 28, 14:00-15:30
1
0
3
Otherwise, you can find me in the audience of the great @ManuelTonneau's oral *today at 1410*. Manuel will present our work on a first global representative dataset of hate speech on Twitter: https://t.co/9AIMlRPqgb
Can we detect #hatespeech at scale on social media? To answer this, we introduce 🤬HateDay🗓️, a global hate speech dataset representative of a day on Twitter. The answer: not really! Detection perf is low and overestimated by traditional eval methods https://t.co/OznvktglzK 🧵
1
0
3
Finally, there's a couple of papers on *LLM persuasion* on the schedule today. Particularly looking forward to @jrfisher552's talk on biased LLMs influencing political decision-making!
1
0
7
Accounting for *pluralism* in human values and preferences (e.g. with personalisation) will also just become more important for a global diversity of users. @morlikow is presenting our poster today at 1100. I'm also hyped for @michaelryan207's work and @verena_rieser's keynote!
1
0
8
Studying *social and political biases* in LLMs is more important than ever, now that >500 million people use LLMs. And to study bias, we need robust and realistic measurement. I am particularly excited to check out work on this by @KLdivergence @1e0sun @jacyanthis @anjali_ruban
2
0
9
Very excited about all these papers on sociotechnical alignment & the societal impacts of AI at #ACL2025. As is now tradition, I made some timetables to help me find my way around. Sharing here in case others find them useful too :) 🧵
4
12
124
I will be at #acl2025 to present "Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals’ Subjective Text Perceptions" ✨ Heartfelt thank you to my collaborators @jiaxin_pei @paul_rottger @pcimiano @david__jurgens Dirk Hovy more below
1
2
12
Today (w/ @UniofOxford @Stanford @MIT @LSEnews) we’re sharing the results of the largest AI persuasion experiments to date: 76k participants, 19 LLMs, 707 political issues. We examine “levers” of AI persuasion: model scale, post-training, prompting, personalization, & more 🧵
14
130
438
