🚨 New Paper Alert! 🚨.When using LLMs for judgements, ever wondered about the consistency of those judgments? 🤔.Check out our latest work, where we quantify, evaluate, and enhance the logical/preference consistency of LLMs. 📚. 🔗 Read more:

RT @_yixu: 🚀Let’s Think Only with Images. No language and No verbal thought.🤔 . Let’s think through a sequence of images💭, like how humans….

Blazing-fast image creation – using just your voice. Try Grok Imagine.

RT @_yixu: 🔥Are we ranking LLMs correctly?🔥. Large Language Models (LLMs) are widely used as automatic judges, but what if their rankings a….

RT @Leon_L_S_C: 🌟 MMR1 Multimodal Reasoning Project Now Open-Source!. We’re thrilled to announce the release of MMR1, an open-source projec….

RT @river_dong121: 🚨New Paper Alert🚨.Many personalization methods optimize performance but ignore real-world impact. We examine its effects….

Long-text factuality is a challenging topic and here’s our cheap & effective approach! 🚀🚀🚀.

RT @abeirami: 𝐛𝐞𝐬𝐭-𝐨𝐟-𝐧 is a strong baseline for .- improving agents.- scaling inference-time compute.- preference alignment .- jailbreakin….

RT @li_chengzu: Forget just thinking in words. 🚀 New Era of Multimodal Reasoning🚨.🔍 Imagine While Reasoning in Space with MVoT. Multimodal….

RT @SuZhaochen0110: 🚀 Interested in building a reliable PRM? Check out our new paper on PRMBENCH – the first process-level reward benchmark….

Round of applaud to my amazing collaborators! @ZhijiangG @EhsanShareghi @licwu @nigelhcollier.

When LLMs are used as logical operators, maintaining a high level of consistency is critical to ensure predictable and efficient decision-making. We examine how logical consistency influences the performance of LLM-based algorithms in such ‘logically grounded’ tasks. 6/n.

We introduce a data refinement and augmentation framework that enhances the consistency without sacrificing human alignment. It augments noisy and sparse pairwise comparison annotations by estimating a partially ordered preference rankings using rank aggregation methods. 5/n

Through our evaluations, we show that:. Transitivity shows strong correlations with self-agreement (self-consistency). Commutativity shows a generally strong correlation with human preference agreement rates across various LLMs. 4/n

We quantify the logical consistency of preference judgements via three fundamental proxies: transitivity, commutativity and negation invariance. We then evaluate logical consistency, using the defined measures, of a wide range of LLMs. 3/n

LLMs exhibit inconsistent and biased behaviour when making decisions or judgements. We focus on studying logical consistency of LLMs as a prerequisite for more reliable and trustworthy systems, where decisions are based on a stable and coherent understanding of the problem. 2/n.

RT @Renee42581826: I'll be presenting CLUES🔍 at #NeurIPS2024 in person! .Catch us at the poster session on: .⏰ Wed, Dec 11, 4:30–7:30 PM P….

RT @ZhijiangG: Life update: 🎉 I'm excited to share that I will be joining @HKUSTGuangzhou as an Assistant Professor in Spring 2025! .I'm lo….

RT @caiqizh: 🔥Check our EMNLP paper with @vlachos_nlp and @ZhijiangG. 🤔Do We Need Language-Specific Fact-Checking Models? The Case of Chine….

RT @hanzhou032: Attending #EMNLP2024 Virtually📺! .If you've ever wondered how to PROMPT your LLM-as-a-Judge⚖️, stay tuned! We will present….

RT @Yingjia_Wan: 💥 Introducing "AutoPSV: Automated Process Supervised Verifier" - accepted at #NeurIPS2024!. AutoPSV automatically annotate….