Eval is often detached from practical applications. Today I extrinsically evaluated one WMT25 translation system on the task of getting hair done without knowing Chinese. Yes you got 67 BLEU points but is the result slaying? 💇 See the result on one datapoint (my head) at EMNLP

The inspiration for the subset2evaluate poster comes from Henri Matisse's The Horse, the Rider and the Clown. 🐎🚴‍♀️🤡

- How to Select Datapoints for Efficient Human Evaluation of NLG Models? https://t.co/WCpqDT0hAt - Estimating Machine Translation Difficulty https://t.co/nsHyi8Hmh9 - COMET-poly: Machine Translation Metric Grounded in Other Candidates

Let's talk about eval (automatic or human) and multilinguality at #EMNLP in Suzhou! 🇨🇳 - Efficient evaluation (Nov 5, 16:30, poster session 3) - MT difficulty (Nov 7, 12:30, findings 3) - COMET-poly (Nov 8, 11:00, WMT) (DM to meet 🌿 )

...research problems I was passionate about and planning my research future. You should apply to these fellowships, even if it's for the exercise of periodically refining your reserach statement.

Grateful to receive the Google PhD Fellowship!🙂 I am not secretive about having applied to 4 similar fellowships during my PhD before and didn't succeed. Still, refining my research statement (part of the application) helped me tremendously in finding out the real interesting..

📢 Announcing the First Workshop on Multilingual and Multicultural Evaluation (MME) — co-located with #EACL2026 🇲🇦 📅 Mar 24–29, 2026 | Rabat, Morocco MME focuses on resources, metrics & methodologies for evaluating multilingual systems! https://t.co/60yCZUjbzH 🗓️ Submit by

📢Shared task deadline extended: You now have a whole week to go (until August 6 AoE) to register and send us your submissions!!

📢 Test Set RELEASED! 🚀 The test set for the #WMT25 Shared Task on QE-informed Segment-level Error Correction is now LIVE! It's time to put your MT error correction / APE methods to the test. Let's see how well they can correct machine translation! #NLProc #MT #WMT2025

Organizers are happy to help with any questions. 🙂 Website with all details and contacts:

📐Task 3: Quality-informed segment-level error correction Automatically post-edit machine-translated text using quality annotations to generate minimal and accurate corrections. Description: https://t.co/844QeBTI9A Submission platform:

📐Task 2: Span-level error detection Identify and locate translation errors within each segment (start/end indices) and classify their severity. Description: https://t.co/baKvWUuPGq Submission platform:

📐Task 1: Segment-level quality score prediction Predict a quality score for each source–target segment pair, using document-level context and either ESA or MQM annotations. Description: https://t.co/M9oEULegNk Submission platform:

The 2025 MT Evaluation shared task brings together the strengths of the previous Metrics and Quality Estimation tasks under a single, unified evaluation framework. The following tasks are now open (deadline July 31st but participation has never been easier 🙂)

Thank you everyone who helped. 😊 Special thanks to @mrinmayasachan and Peng Cui from @CSatETH and all my friends I bugged with proofreading. 😁

"How to Select Datapoints for Efficient Human Evaluation of NLG Models?" has now been accepted to TACL (a)! 🌿 📃 Paper (with nuances and caveats): https://t.co/oBP8qb5Bs0 📦 Package:  https://t.co/OdPeoycIHa Feedback welcome!

Recommendation based on translation and summarization: 1️⃣ if you have a good automatic metric, use variance/consistency 2️⃣ if not, use model output diversity 3️⃣ if outputs not available, use artificial crowd/distilled predictors 4️⃣ if those are not available, use source diversity

We frame this as a 0/1 Knapsack problem: find a subset Y ⊆ X with maximum utility while staying under budget B. 🤓 maximize: ∑ zₓ · Utility(x) subject to: ∑ zₓ · Cost(x) ≤ B zₓ ∈ {0, 1} The Utility(x) can be metric average, variance, diversity, etc.

This works even if you don't have the model outputs yet. 1️⃣ "artificial crowd" simulate what model outputs would look like; apply the previous methods. 2️⃣ "utility predictors" estimate usefulness from the source text. 3️⃣ "source-based diversity" remove similar inputs.

So what works? Selecting inputs that expose model differences: 1️⃣ high variance in metric scores 2️⃣ diversity in model outputs 3️⃣ high metric consistency with the rest of the dataset We now need almost 30% fewer annotated examples to get the same model ranking.