Last week was my final one at @Unbabel. I'm incredibly proud of our work (e.g., Tower, MINT, M-Prometheus, ZSB). Now, alongside my PhD studies at @istecnico, I'm joining @swordhealth as Senior Research Scientist under @RicardoRei7. Super confident in the team we're assembling.

RT @ManosZaranis: 🚨Meet MF²: Movie Facts & Fibs: a new benchmark for long-movie understanding!.🤔Do you think your model understands movies?….

Check out the latest iteration of Tower models, Tower+. Ideal for translation tasks and beyond, and available at three different scales: 2B, 9B, 72B. All available on huggingface: Kudos to everyone involved!.

RT @dongkeun_yoon: 🙁 LLMs are overconfident even when they are dead wrong. 🧐 What about reasoning models? Can they actually tell us “My an….

RT @psanfernandes: MT metrics excel at evaluating sentence translations, but struggle with complex texts. We introduce *TREQA* a framework….

RT @dongkeun_yoon: Introducing M-Prometheus — the latest iteration of the open LLM judge, Prometheus!.Specially trained for multilingual ev….

RT @seungonekim: Here's our new paper on m-Prometheus, a series of multulingual judges!. 1/ Effective at safety & translation eval.2/ Also….

Models and training data: Paper:

Massive kudos to co-authors @dongkeun_yoon @psanfernandes @ianwu97 @seungonekim @RicardoRei7 @gneubig @andre_t_martins.

There were a lot of open questions on what strategies work for building multilingual LLM judges. We perform ablations on our training recipe that highlight the importance of backbone model choice and of using natively multilingual—instead of translated—training data.

We fine-tune Qwen2.5 models with a recipe inspired by Prometheus 2. We release two multilingual datasets: M-Feedback-Collection, and M-Preference-Collection. They contain DA and PWC data for 5 languages, and MT eval data for 8 LPs. Our models perform well on unseen languages.

For their size, M-Prometheus models achieve SotA performance on multilingual reward benchmarks and literary MT evaluation. They can also be used to significantly improve multilingual LLM outputs via best-of-n decoding (QAD)! Very useful for refining synthetic data, for example.

We just released M-Prometheus, a suite of strong open multilingual LLM judges at 3B, 7B, and 14B parameters!. Check out the models and training data on Huggingface: and our paper:

Massive kudos to collaborators @nunonmg @RicardoRei7 and @andre_t_martins . To use our benchmarks and create your own, check out our repository:

We also perform a series of ablations finding that dataset variety (not size!) and judge model size are crucial factors driving benchmark performance.

We meta-evaluate ZSB across LLM general capabilities in 4 languages (English, Chinese, French, and Korean), translation, and VLM general capabilities in English. ZSB consistently outperforms widely-used standard benchmarks, like MMLU, GPQA, MMBench, etc… at ranking models.

We simplify previous approaches: ZSB only requires the creation of a prompt for data generation and one for evaluation, and the rest is handled by a language model, which functions as a data generator and a judge. This approach is scalable to various tasks and modalities.

New paper out 🚀 Zero-shot Benchmarking: A Framework for Flexible and Scalable Automatic Evaluation of Language Models: We present a framework and release a repository for creating reliable benchmarks for (V)LM tasks quickly and fully automatically.

RT @slatornews: .@Unbabel exposes 🔎 how using the same metrics for both training and evaluation can create misleading ⚠️ #machinetranslatio….

RT @fbk_mt: Our pick of the week by @apierg: "Adding Chocolate to Mint: Mitigating Metric Interference in Machine Translation" by José Pomb….