LLM4Eval Workshop
@llm4eval
Followers
84
Following
40
Media
9
Statuses
44
The Third Workshop on Large Language Models for Evaluation in IR (LLM4IREval@SIGIR2025)
Padua, Italy
Joined February 2024
LLMs are reshaping how we evaluate ranked lists, conversations, and even simulate users. At #LLM4Eval @ #SIGIR2025, we’re spotlighting 3 special themes:.🧠 Beyond query-doc.🗣️ Simulating users.🧪 Synthesizing evaluation data.👇 Curious? Read on & submit by April 23!.#LLMs #IR.
📢 Call for Papers: LLM4EVAL @ SIGIR '25 - The 3rd Workshop on Large Language Models for Evaluation in IR! @SIGIRConf . 🗓️ Submission: April 23, 2025 (AoE).🔗 Full CFP: 🌐 Workshop:
1
9
15
Chirag Shah is this morning giving a keynote on: From Relevance to Reality: Scaling Human-Centered Evaluation in the LLM Era
0
2
5
The deadline for submitting your LLM evaluation paper is tomorrow. Looking forward to interesting contributions!.
📢 Deadline extended!.Submit to #LLM4Eval @ #SIGIR2025 by April 30 (AoE). We welcome work on LLM-based evaluation, including:.🧠 beyond query-doc.🗣️ simulating users.🧪 synthetic eval data. 🔗 #CFP #IR #LLMs #SIGIR2025.
0
0
3
📢 Deadline extended!.Submit to #LLM4Eval @ #SIGIR2025 by April 30 (AoE). We welcome work on LLM-based evaluation, including:.🧠 beyond query-doc.🗣️ simulating users.🧪 synthetic eval data. 🔗 #CFP #IR #LLMs #SIGIR2025.
llm4eval.github.io
Call For Paper
LLMs are reshaping how we evaluate ranked lists, conversations, and even simulate users. At #LLM4Eval @ #SIGIR2025, we’re spotlighting 3 special themes:.🧠 Beyond query-doc.🗣️ Simulating users.🧪 Synthesizing evaluation data.👇 Curious? Read on & submit by April 23!.#LLMs #IR.
1
7
9
RT @maliannejadi: We will host multiple breakout discussions to pick your brains 🧠 on the future challenges and research on this topic. Let….
0
4
0
🧵 Theme 3:. 🧪 Synthesizing evaluation data: When judgments are scarce, can we generate evaluation data with LLMs?. We want approaches for creating evaluation datasets in low-resource or privacy-sensitive scenarios, where collecting real data is hard, expensive, or impossible.
0
2
3
🧵 Theme 2:. 🗣️ Simulating users: What if LLMs could act as realistic user proxies?. We are looking for papers on simulating user behavior, preferences, context, or personas for evaluation.
1
2
3
🧵 Theme 1:. 🧠 Evaluation beyond query-doc: How can LLMs evaluate full sessions, ranked lists, or multi-turn conversations?. We welcome work that goes beyond traditional Qrels and explores richer evaluation units.
1
2
2
Organized by @cnsiro, @srahmanidashti , @maliannejadi, @nickcr , @claclarke, @guglielm0f , @UnderdogGeek , @pt_ir , and @emine_yilm . #SIGIR, #IR, #Evaluation, #LLM4Eval.
0
2
6
In this series, we move BEYOND search relevance to tackle: .✨ Personalized search evaluation .✨ Conversational interfaces .✨ Holistic assessment of integrated experiences .✨ Bias & fairness in LLM evaluations.
1
3
2
📢 Call for Papers: LLM4EVAL @ SIGIR '25 - The 3rd Workshop on Large Language Models for Evaluation in IR! @SIGIRConf . 🗓️ Submission: April 23, 2025 (AoE).🔗 Full CFP: 🌐 Workshop:
llm4eval.github.io
The Third Workshop on Large Language Models for Evaluation in IR (LLM4Eval @ SIGIR 2025)
1
8
13
RT @TechAtBloomberg: In today's 2nd Workshop on #LLMs for Evaluation in #InformationRetrieval (#LLM4Eval) at #WSDM2025 (09:15 CET), @edgarm….
0
4
0
RT @srahmanidashti: The workshop will focus on models, techniques, data collections, and methodologies for information retrieval evaluation….
0
2
0
RT @srahmanidashti: Organiser: @srahmanidashti, @cnsiro, @maliannejadi, @nickcr, @claclarke, @guglielm0f, @UnderdogGeek, @pt_ir , and @emin….
0
1
0
RT @srahmanidashti: The call for papers is now open for the second workshop of #LLM4Eval:. 💡 See CFP: 🗓 Submission….
0
3
0
RT @srahmanidashti: 🔊 The second workshop on Large Language Models for Evaluation in IR (#LLM4Eval), co-located with #WSDM2025, is now acce….
0
6
0
RT @srahmanidashti: 📢 Thrilled to announce that, "SynDL: A Large-Scale Synthetic Test Collection for Passage Retrieval" is publicly availab….
arxiv.org
Large-scale test collections play a crucial role in Information Retrieval (IR) research. However, according to the Cranfield paradigm and the research into publicly available datasets, the...
0
15
0
RT @guglielm0f: Great panel discussion with @ian_soboroff, @lauradietz99, @claclarke, Michael Ekstrand moderated by @UnderdogGeek at @llm4e….
0
6
0