Alan Ritter
@alan_ritter
Followers
5K
Following
3K
Media
27
Statuses
410
Computing professor at Georgia Tech - natural language processing, language models, machine learning
Atlanta, GA
Joined March 2009
What if LLMs could predict their own accuracy on a new task before running a single experiment? We introduce PRECOG, built from real papers, to study description→performance forecasting. On both static and streaming tasks, GPT-5 beats human NLP researchers and simple baselines.
What if LLMs can forecast their own scores on unseen benchmarks from just a task description? We are the first to study text description→performance prediction, giving practitioners an early read on outcomes so they can plan what to build—before paying full price 💸
2
7
40
🎉 Excited to see that our paper on cost-efficient data annotation for LLMs won an SAC Highlight Award! 🔗 Check out @mohit_rag18's work here:
aclanthology.org
Mohit Raghavendra, Junmo Kang, Alan Ritter. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.
Check out @mohit_rag18's recent work analyzing data annotation costs associated with SFT vs. Preference Fine-Tuning.
1
4
42
Had a great time visiting Sungkyunkwan University this week! Lots of interesting conversations, and very insightful questions from students. Thanks to @NoSyu for hosting!
We are honored to host @alan_ritter at SKKU for his talk: "Towards Cost-Efficient Use of Pre-trained Models" He explored cost-utility tradeoffs in LLM development, fine-tuning vs. preference optimization, which led to more efficient and scalable AI. Thanks a lot!
2
1
16
We are honored to host @alan_ritter at SKKU for his talk: "Towards Cost-Efficient Use of Pre-trained Models" He explored cost-utility tradeoffs in LLM development, fine-tuning vs. preference optimization, which led to more efficient and scalable AI. Thanks a lot!
1
2
21
For people attending @naaclmeeting, I created a quick script to generate ics files for all your presentations (or presentations of interest) that you can import into your Google Calendar or other calendar software:
gist.github.com
Create ics format calendar files for NAACL 2025. GitHub Gist: instantly share code, notes, and snippets.
0
3
43
0
0
3
Wondering what review scores you need to get accepted at ACL? Maybe this data from NAACL 2025 can help: https://t.co/iKClKUbASI
gist.github.com
NAACL 2025 Scores-to-Decisions. GitHub Gist: instantly share code, notes, and snippets.
3
16
81
Want to learn about Llama's pre-training? Mike Lewis will be giving a Keynote at NAACL 2025 in Albuquerque, NM on May 1. https://t.co/c9VdW0GYKM
@naaclmeeting
3
24
273
🚨o3-mini vastly outperforms DeepSeek-R1 on an unseen probabilistic reasoning task! Introducing k-anonymity estimation: a novel task to assess privacy risks in sensitive texts Unlike conventional math and logical reasoning, this is difficult for both humans and AI models. 1/7
1
17
22
Very excited about this new work by @EthanMendes3 on self-imprving state value estimation for more efficient search without labels or rewards.
🚨New Paper: Better search for reasoning (e.g., web tasks) usually requires costly💰demos/rewards What if we only self-improve LLMs on state transitions—capturing a classic RL method in natural language? Spoiler: It works (⬆️39% over base model) & enables efficient search!🚀 🧵
0
4
20
🚨New Paper: Better search for reasoning (e.g., web tasks) usually requires costly💰demos/rewards What if we only self-improve LLMs on state transitions—capturing a classic RL method in natural language? Spoiler: It works (⬆️39% over base model) & enables efficient search!🚀 🧵
1
11
26
🚨 Just Out Can LLMs extract experimental data about themselves from scientific literature to improve understanding of their behavior? We propose a semi-automated approach for large-scale, continuously updatable meta-analysis to uncover intriguing behaviors in frontier LLMs. 🧵
1
15
41
Check out @mohit_rag18's recent work analyzing data annotation costs associated with SFT vs. Preference Fine-Tuning.
🚨Just out Targeted data curation for SFT and RLHF is a significant cost factor 💰for improving LLM performance during post-training. How should you allocate your data annotation budgets between SFT and Preference Data? We ran 1000+ experiments to find out! 1/7
0
2
16
🚨Just out Targeted data curation for SFT and RLHF is a significant cost factor 💰for improving LLM performance during post-training. How should you allocate your data annotation budgets between SFT and Preference Data? We ran 1000+ experiments to find out! 1/7
2
33
142
Awesome talk yesterday by @jacobandreas! Interesting to hear how models can make better predictions by asking users the right questions, and update themselves based on implied information. @mlatgt
1
1
45
📢 Working on something exciting for NAACL? 🗓️ Remember the commitment deadline is December 16th! https://t.co/z9apuiBf8b
#NLProc
0
3
8
Come see @EthanMendes3 talk on the surprising image geolocation capabilities of VLMs today at 2:45 in Flagler (down the escalator on the first floor). #emnlp2024 @mlatgt
0
2
12
Please take a moment to fill out the form below to volunteer as a reviewer or AC for NAACL!
📢 NAACL needs Reviewers & Area Chairs! 📝 If you haven't received an invite for ARR Oct 2024 & want to contribute, sign up by Oct 22nd! ➡️AC form: https://t.co/4KSWkEfxoO ➡️Reviewer form: https://t.co/3DqVNOSGXF Please RT 🔁 and help spread the word! 🗣️ #NLProc @ReviewAcl
0
0
8
Great news! The @aclmeeting awarded IARPA HIATUS program performers two "Best Social Impact Paper Awards" and an "Outstanding Paper Award." Congratulations to the awardees! Access the winning papers below: https://t.co/M7DsMOndy5
https://t.co/RunhsjF5vp
https://t.co/iCSqv7NlNu
0
8
15