For LLMs to be inclusive across evolving cultures, they needn’t know all social norms, but should at least be able to adapt their responses given enough cultural context. However, prior cultural probing work typically only evaluates an LLM’s internal knowledge of norms [2/n].

We introduce NormAd, a benchmark containing 2,633 stories depicting cultural situations from 75 countries. We provide multiple cultural contexts to analyze how amenable an LLM is to social norms across cultures! [3/n]

We synthetically generate stories and their cultural contexts by seeding GPT-4-Turbo with cultural backgrounds from the cultural atlas. We ensure data quality by performing manual and automated filtration and validation checks in our pipeline. [4/n]

Even with explicit social norms, the top-performing models, GPT-4 and Mistral-7b-Instruct, achieve 87.6% and 81.8% accuracy, lagging behind the 95.6% achieved by humans. [5/n]

When generalized to higher-level value contexts and just country names, performance plummeted to 60% and below for the best models. Larger models improved with different training paradigms, but exhibited greater skew towards English/Western cultures. [6/n]

Moreover, LLMs displayed inherent biases - performing better on stories conforming to norms over those that violate social norms. LLMs also struggle across highly nuanced axes such as gifting. [7/n]

We would expect an LLM to be able to reason over a social situation when given an explicit norm, but no! They still fail to adapt their answers – Our findings highlight a limitation in LLMs’ reasoning capabilities across cultural applications. [8/n] Dataset and code coming soon!

@AetherSuRa @akhila_yerukola @vishwayvs @_doctor_kat @MaartenSap Dark mode for this paper for night readers 🌙

@AetherSuRa @akhila_yerukola @vishwayvs @_doctor_kat @MaartenSap What’s the point? Doesn’t GPT3 already do this