We have been overlooking a key factor in LLM-as-a-judge evaluation: the feedback collection protocol. Our #COLM2025 paper presents a comprehensive study on how feedback protocols shape reliability and bias in LLM evaluations.

We did this work with amazing collaborators @ManyaWadhwa1 @gregd_nlp @scottniekum Read the full paper:

Generator models can exploit these distractors to game rankings and artificially climb leaderboards, making evaluation protocols a crucial design choice for fair comparison.

Distractor features can make an LLM judge change its preference even when one response isn’t actually better. We observed these preference flips in 35% of pairwise evaluations, compared to just 9% with absolute scoring.

Pairwise feedback is far easier to distract with spurious features like length, or assertiveness. Absolute scoring individual responses is substantially more robust to these superficial factors.

After a MEGA finals run in which we scored 18 points, Team MARBLE has placed 3rd in the @DARPA #SubTChallenge! We are extremely happy with our performance as a team. Congratulations to the 1st and 2nd place finishers @CerberusSubt and @CSIRORobotics!!

Dear @airindiain, it has been a terrible experience with the airlines. Firstly, my flight was canceled but I still haven't received any message/email regarding that. After finding out about the cancellation somehow, spent 6 hours trying to connect to your ticketing department 👇