Many thanks for this big honour! 🤩.

RT @IJCAIconf: Congratulations to the winners of the 2025 IJCAI–JAIR Prize for their paper “Compositionality Decomposed: How Do Neural Netw….

RT @vernadankers: Proud to accept a 5y outstanding paper award @IJCAIconf 🏆 from JAIR for the impact Compositionality Decomposed has had,….

RT @WiAIR_podcast: 🧠 What does it really mean for an LLM to generalize? And are we even measuring it right?.In the latest #WiAIR episode, w….

Could not be more thrilled about this partnership, allowing us to keep MultiLoKo's test set truly hidden and have experts at Kaggle independently run the leaderboard 😍🔥💪.

Thrilled about the launch of this platform 🤩, the feature to host secret test sets is a deal breaker in the game against contamination and a gift to both benchmark builders and modellers 🔥🔥 Excited to be one of the first to use it for @AIatMeta's MultiLoKo's test set 💪!.

RT @kaggle: 🚀 Kaggle Benchmarks is here! Get competition-grade rigor for AI model evaluation. Let Kaggle handle infrastructure while you f….

RT @WiAIR_podcast: How do we know if a language model really generalizes - or is just repeating patterns it’s memorized?.Let’s talk about c….

RT @ryan_nefdt: My new book on Linguistic Relativity (with Jeff Pelletier) just dropped with @OUPPhilosophy: We di….

Want to know more? Have a look at our paper or our github repository:.- - Don't forget to check the related work section for other awesome work on multilingual evaluation for LLMs. @metaai.

But if you want to appear on our github leaderboard, reach out via a github issue on our repo

Lastly, MultiLoKo has an OOD test split! We split the data based on frequency of the sourcing topic on Wikipedia and is more difficult than dev. We keep the test split secret until further notice to prevent overfitting.

As for the difference between human vs machine translations: also this changes language difficulty rankings, but not as drastically. It does create a substantial average drop for most languages.

We find that sourcing changes language prioritisation! Rank correlations between language difficulty are between 0.54 - 0.70 for the best models; locality effect ranges from -13 to +17 for Llama 3.1 405B, from -21 to +15 for Gemini 2.0 Flash, and from -22 to +14 for GPT4-o.

We also consider the impact of local sourcing vs translating English data (the norm for most benchmarks used by LLM releases), which we express in the Locality Effect metric (locally sourced EM - translated English data EM).

We also run some response-level consistency tests inspired by those previous works (adapted version though), and find that even for the best performing models, the overlap of questions correctly answered across languages is not even 50%.

This pretty result confirms earlier results from a.o. Qi et. al ( @Jirui_Qi @AriannaBisazza @raquel_dmg ) and Ohmer et al ( @xenia_ohmer @eliabruni) that knowledge transfer between languages in LLMs is suboptimal.

Next, we studied the effect of the question language and found that generally, performance is higher when asked in the 'native' language. In the plot, *mother tongue effect* = performance when question is asked in the language to which it is relevant - performance in English

For multilingual benchmarks, average can hide a lot. We therefore add a second metric to MultiLoKo: the gap between the best and worst language. Unsurprisingly, there is quite a lot of variance across languages, even when only the topN languages apart from English are considered.

Perhaps the most boring result: MultiLoKo is hard. Best average EM on the dev set is EM of around 34.