LLMs: great at judging… until it’s their own homework. 📚🔥So we built the math to call them out 🤷‍♀️ To learn more, check out our new paper: Play Favorites: A statistical method to quantify self-bias in LLM-as-a-judge 🎭 📄 Paper:

Thanks to our co-authors Riccardo Fogliato, H. Burnsky, T. Soliman, J. Ma, G. Horwood and @migballesteros! Also thanks to @awscloud Bedrock for supporting our work! 📄 Paper: https://t.co/fWxcEhMsbE 📷 Code & data:

Self-bias is not a "fixed" quantity, it varies based on the dimension and dataset (although some overall trends can be observed)

Family-bias accounts for a big part of self-bias of LLMs. Also negative self-bias is possible; some LLMs are more "critical" of themselves!

📄 Paper: https://t.co/fWxcEhMsbE 💻 Code & data:

In our paper, we also: ✅ Conduct an empirical study with 5k+ prompts & 9 LLM judges ✅ Release human annotations to support future research ✅ Find systematic self-bias (+ family-bias) in GPT-4o & Claude 3.5 Sonnet

Our framework: ✅ Explicitly models conditions under which self-bias can be detected ✅ Separates true quality differences from self-bias ✅ Accounts for consistent annotator differences

We introduce a statistical framework that isolates and quantifies self-bias in LLM-as-a-judge, while separating genuine quality differences (via independent human judges) from bias. Our study also reveals a strong family-bias problem — LLMs favoring models from their own family.

Thrilled to share that our Byte Latent Transformer won an Outstanding Paper Award at ACL 2025! 🏆

🚀 Introducing the Byte Latent Transformer (BLT) – An LLM architecture that scales better than Llama 3 using byte-patches instead of tokens 🤯 Paper 📄 https://t.co/5QGrlJdK0y Code 🛠️ https://t.co/jCdDI5BXwe

We will present QLoRA at NeurIPS! Come to our oral on Tuesday where @Tim_Dettmers will be giving a talk. If you have questions stop by our poster session!

Ground breaking research! We need competitive, open-sourced models that one can fine-tune with limited resources!

4-bit QLoRA is here to equalize the playing field for LLM exploration. You can now fine-tune a state-of-the-art 65B chatbot on one GPU in 24h. Paper: https://t.co/7gX1oIUHEx Code and Demo:

A NYT article on the debate around whether LLM base models should be closed or open. Meta argues for openness, starting with the release of LLaMA (for non-commercial use), while OpenAI and Google want to keep things closed and proprietary. They argue that openness can be

We are excited to announce QLoRA, a new method for LLM fine-tuning that uses only a fraction of the memory footprint. Please consider joining our private beta to gain early access to QLoRA! Stay tuned for the paper and code release, coming soon.

ICYMI since you have a social life and aren’t perpetually online. I wrote for @Forbes on how @RepGallagher could lead the new Select Committee focused on CCP to succeed by addressing critical issues that are sometimes missing from the conversation. https://t.co/uql4kNAALM

Join our oral presentation @emnlpmeeting in the Commonsense Reasoning track, at 10am Sunday 12/11, Hall B. Our paper is available here:

The legendary fights of Physical Interaction vs Language-only models, their epic journeys through the valleys of Artificial Environments and Naturally Occurring text, and their challenges on in-domain & out-of-domain attributes ONLY in EvEntS ReaLM! @ArtidoroPagnoni @ybisk @ehovy

Very excited to present our work “Threat Scenarios and Best Practices for Neural Fake News Detection” at COLING 2022! https://t.co/ObCpMcDdQw with Yulia Tsvetkov and Martin Graciarena

1/n 📣 New Preprint: "𝑪𝒐𝒖𝒏𝒕𝒆𝒓𝒇𝒂𝒄𝒕𝒖𝒂𝒍 𝑷𝒉𝒆𝒏𝒐𝒕𝒚𝒑𝒊𝒏𝒈 𝒘𝒊𝒕𝒉 𝑪𝒆𝒏𝒔𝒐𝒓𝒆𝒅 𝑻𝒊𝒎𝒆-𝒕𝒐-𝑬𝒗𝒆𝒏𝒕𝒔" w/ @MononitoGoswami @AutonLab and @DrDufendach @UPMC https://t.co/88eqEJcJA3 #MachineLearning #EpiTwitter #MedTwitter #causalinference #DataScience