Paper link:🔗 https://t.co/cvemrBS94r

Our paper with @vectara, “Benchmarking LLM Faithfulness in RAG with Evolving Leaderboards”, is now published in the EMNLP 2025 Industry Track! Check out our work on enabling more reliable LLM faithfulness benchmarking in RAG!

As LLMs evolve, our benchmarks must too. We’ll continue updating FaithJudge (currently using an o3-mini-high judge) and the leaderboard for more complete evaluation. 📄 Full paper:

FaithJudge uses hallucination annotation examples to guide the automated evaluation of LLM responses for the same tasks. ✅ Outperforms prior methods 🤝 Matches human judgment more closely 📊 Works across summarization, QA, and data-to-text

We introduced FaithBench to evaluate hallucination detectors on challenging summaries from 10 modern LLMs. FaithBench highlights just how hard hallucination detection still is. 🔗

Existing hallucination detection methods (fine-tuned models or zero-shot LLM judges) are promising, but often struggle. FaithJudge improves detection by prompting LLM judges with human-labelled hallucination examples from FaithBench and RagTruth.

LLM hallucinations remain stubbornly persistent, even with RAG. LLMs still ➕ add, 🔄 distort, or ❌ contradict contexts. The well-established Hallucination Leaderboard from @vectara evaluates hallucination rates for 130+ LLMs on summarization. 🔗

Introducing 🔍 FaithJudge: improving how we evaluate LLM faithfulness in RAG tasks, including summarization, QA, and data-to-text and powering a more accurate LLM Hallucination Leaderboard. 🔗

Super excited to share FaithJudge - a new hallucination benchmark and leaderboard. Shoutout to @lintool @ManveerTamber and many others who contributed. Paper: https://t.co/WTEa9bQMXd Github:

Our work raises questions about the commercial viability of embedding models and the security of black-box retrieval models. Work done with @jsprxian and @lintool https://t.co/WqWuKE0G8I

💥Turns out: commercial embedding models are very vulnerable to being distilled. We also show how to train even better models by simply concatenating embeddings from multiple teachers and distilling those, attaining strong results.

Commercial embedding models are locked 🔒 behind APIs, but how protected are they from theft🥷? In our #NAACL2025 Findings paper, we show they can be stolen at a modest cost, with thief models that rival their victims and generalize well to out-of-domain retrieval tasks.

LiT5 (led by @ManveerTamber) is heading to #ECIR2025! And the 🍒 on top? We've added LiT5-v2 which can rerank 100 segments at once 🤯 Paper soon, but it's already live on RankLLM ( https://t.co/5tpi09PtmY)!

Pablo Picasso said "good artists borrow, great artists steal". So much innovation happening so quickly in the embedding space... it's easier just to steal other people's models. @ManveerTamber and @jsprxian show you how: https://t.co/9WirycBhmb

🔗 Our models are now accessible on 🤗 Hugging Face: https://t.co/UjyHbq3IX6. You can replicate our findings with our code: https://t.co/ZMH226KL6k.

🌍 Tested on diverse BEIR collections, LiT5-Distill & LiT5-Score prove their mettle, showcasing exceptional adaptability and generalization capabilities.

📊 MS MARCO benchmarks reveal LiT5-Distill's competitive edge – at times outshining state-of-the-art models – all while being more compact and built on outdated T5 weights.

💡 Introducing LiT5-Distill & LiT5-Score: models ranging from 220M to 3B parameters for listwise reranking. 🌟 LiT5-Distill distills RankGPT3.5 & RankGPT4's power into smaller, yet effective models. 🔍 LiT5-Score uses cross-attention scores for reranking.

🚀Thrilled to unveil our work in efficient zero-shot listwise reranking! LiT5 harnesses T5 models to challenge state-of-the-art standards, with significantly smaller models. Discover more:

Here we revisit the wonders of T5, Fusion-in-Decoder (FiD), and building smaller and effective rerankers, focussing on the listwise paradigm of prompt decoders! You can find the code and models here - https://t.co/JAoBQIJHLa. Led by the amazing @ManveerTamber!