RT @Elenal3ai: 🚨 New paper alert 🚨. Ever asked an LLM-as-Marilyn Monroe who the US president was in 2000? 🤔 Should the LLM answer at all? W….

HypoEval evaluators ( are now incorporated into judges from @QuotientAI — check it out at .

12/n Acknowledgments:.Great thanks to my wonderful collaborators @lihanc02 and my advisor @ChenhaoTan!.Check out full paper here at (.

11/n Closing thoughts:.This is a sample-efficient method for LLM-as-a-judge, grounded upon human judgments — paving the way for personalized evaluators and alignment!.

10/n Code:.We have released to repositories for HypoEval:.For replicating results/building upon: For off-the-shelf 0-shot evaluators for summaries and stories🚀:

9/n Why HypoEval matters:. We push forward LLM-as-a-judge research by showing you can get:.Sample efficiency.Interpretable automated evaluation.Strong human alignment. …without massive fine-tuning.

8/n 🔬 Ablation insights:.Dropping hypothesis generation → performance drops ~7%.Combining all hypotheses into one criterion → performance drops ~8% (Better to let LLMs rate one sub-dimension at a time!).

7/n 💪 What’s robust?.✅ Works across out-of-distribution (OOD) tasks.✅ Generated hypothesis can be transferred to different LLMs (e.g., GPT-4o-mini ↔ LLAMA-3.3-70B).✅ Reduces sensitivity to prompt variations compared to direct scoring

6/n 🏆 Where did we test it?.Across summarization (SummEval, NewsRoom) and story generation (HANNA, WritingPrompt).We show state-of-the-art correlations with human judgments, for both rankings (Spearman correlation) and scores (Pearson correlation)! 📈.

5/n Why is this better?.By combining small-scale human data + literature + non-binary checklists, HypoEval:.🔹 Outperforms G-Eval by ~12%.🔹 Beats fine-tuned models using 3x more human labels.🔹 Adds interpretable evaluation.

4/n These hypotheses break down complex evaluation rubric (ex. “Is this summary comprehensive?”) into sub-dimensions an LLM can score clearly. ✅✅✅

3/n 🌟 Our solution: HypoEval.Building upon SOTA hypothesis generation methods, we generate hypotheses — decomposed rubrics (similar to checklists, but more systematic and explainable) — from existing literature and just 30 human annotations (scores) of texts.

2/n What’s the problem?.Most LLM-as-a-judge studies either:.❌ Achieve lower alignment with humans.⚙️ Requires extensive fine-tuning -> expensive data and compute. ❓ Lack of interpretability.

1/n 🚀🚀🚀 Thrilled to share our latest work🔥: HypoEval - Hypothesis-Guided Evaluation for Natural Language Generation! 🧠💬📊.There’s a lot of excitement around using LLMs for automated evaluation, but many methods fall short on alignment or explainability — let’s dive in! 🌊.

RT @mouradheddaya: 🧑‍⚖️How well can LLMs summarize complex legal documents?  And can we use LLMs to evaluate?. Excited to be in Albuquerque….

RT @HaokunLiu5280: 🚀🚀🚀Excited to share our latest work: HypoBench, a systematic benchmark for evaluating LLM-based hypothesis generation me….

RT @divingwithorcas: 1/n You may know that large language models (LLMs) can be biased in their decision-making, but ever wondered how thos….

RT @HaokunLiu5280: 1/ 🚀 New Paper Alert!.Excited to share: Literature Meets Data: A Synergistic Approach to Hypothesis Generation 📚📊!.We pr….