🙋 How do Large Language Models internally process cultural knowledge? 🌐 Happy to share our new preprint "Entangled in Representations: Mechanistic Investigation of Cultural Biases in Large Language Models" 📃 Paper: https://t.co/Gt14mjMiQg

Huge thanks to my wonderful collaborators: Woosung Kang, @JunhoMyung_, and @aliceoh. 💛

Dive deeper into our work: - Paper: https://t.co/sEXSBuGpPH - Project Page: https://t.co/MeDT3tW5Tx - Repo: https://t.co/P66PUUmusB - HuggingFace Datasets:

👿 Challenge: Automatically evaluate the social biases in LLMs’ long-form responses without additional human annotation 💡 Solution: Take the stories of the BBQ dataset, use them as prompts, then use the existing annotations to evaluate the long-form generations!

🤔 Is evaluating social bias of #LLMs with only multiple-choice QA benchmarks enough? NO! 🔍 Research Question: How often do LLMs “generate” responses that reflect social stereotypes?

🥳Thrilled to share that our paper has been accepted to the Findings of #ACL2025NLP! We introduce #BBG, a Bias Benchmark for Generation, an adaptation of the Bias Benchmark for QA (BBQ), revealing inconsistencies between generation- and multiple-choice QA-based evaluations.

Thrilled to share that I'll present my #NeurIPS2024 poster presentation on 11th Dec! "BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages" https://t.co/TfADNzM2fS 📆 When: Wed 11 Dec 4:30 pm- 7:30 pm 📍 Where: West Ballroom A-D

Join us for our poster presentation at #EMNLP2024 ! 📆When: Thu 14 Nov 14:00-15:30 📍Where: Riverfront hall 📄Paper: https://t.co/NhS0EuqKk0 If you are interested in social reasoning and theory of mind of language models, please come by! @emnlpmeeting

📢What do we know about Theory of Mind (ToM) in LLMs, aside from the fact that they struggle with it? What foundational ToM capabilities do they have? Our new EMNLP paper explores the precursory inferences of ToM in LLMs: perception inference and perception-to-belief inference👀

I am so grateful and proud of @whoSiddheshp and @jjjunyeong for leading this new survey paper on cultural awareness of LLMs. One trend I want to highlight is that while the MQA makes up the largest portion of eval, short answer and long form eval are gaining ground as well 👍

Thank you to each and every one of you for making this happen. It was a great experience collaborating with such an amazing team!

📜Excited to share our comprehensive survey on cultural awareness in #LLMs! 🗺️ We reviewed 300+ papers across diverse modalities (language, vision-language, etc.) @whoSiddheshp @jjjunyeong @jin__jiho @rnav_arora @JunhoMyung_ @_inhwa_song @aliceoh #NLProc https://t.co/ai5UZOqIcf

🤩Really excited that this work will be presented at #neurips2024 d&b track. The BLEnD dataset took serious collaboration of hard thinking and work, getting human annotations from 16 diverse regional cultures in 13 languages, putting together short-answer and multiple choice QA

🌟Happy to share that our "𝐁𝐋𝐄𝐧𝐃: 𝐀 𝐁𝐞𝐧𝐜𝐡𝐦𝐚𝐫𝐤 𝐟𝐨𝐫 𝐋𝐋𝐌𝐬 𝐨𝐧 𝐄𝐯𝐞𝐫𝐲𝐝𝐚𝐲 𝐊𝐧𝐨𝐰𝐥𝐞𝐝𝐠𝐞 𝐢𝐧 𝐃𝐢𝐯𝐞𝐫𝐬𝐞 𝐂𝐮𝐥𝐭𝐮𝐫𝐞𝐬 𝐚𝐧𝐝 𝐋𝐚𝐧𝐠𝐮𝐚𝐠𝐞𝐬" paper has been accepted to #NeurIPS2024 D&B Track 🚀 Congrats to all the amazing co-authors 🥳

Breaking down the Theory-of-Mind task into perception, perspective reasoning, and response generation. Then analyzing the performance of LLMs (pretty bad), led by @ChaniJung99, with @YejinChoinka and @hyunw_kim, co-authors @jiseon_kim1 @_dongkwan_kim @jin__jiho @YeonSeonwoo Will

This paper got the best non-archival paper award! 🏆 Congratulations to @JunhoMyung00211, @nlee0212 and @jodieyzhou who brilliantly led the project, and to all the many collaborators involved in the project! 👏🏼👏🏼

Congratulations 🎉 @JunhoMyung00211 @CamachoCollados @hwaran_lee @nlee0212 @jodieyzhou @nedjmaou @euns0o_kim @rifkiaputri and all other authors! Thank you @c3_nlp organizers! #acl2024

Pre #acl2024 talks, posters, and food 🥘🍝🍲🍛🥙 We @sunipa17 @kaiwei_chang @TristanNaumann @IAugenstein @computermacgyve @CamachoCollados @PangWeiKoh @seo_minjoon @NoSyu and Yoon Kim, (with @mohitban47 and @VioletNPeng joining online) had a blast thanks to the amazing students

I’m in Bangkok for ACL 2024🇹🇭! We will be sharing KoBBQ in several presentations and poster sessions. If you are interested, please stop by! 12 Aug @ 12:15 (Question Answering I) 14 Aug @ 10:30 (In-Person Poster Session) 16 Aug @ 11:50, 16:00 (C3NLP)

Had a meaningful experience organizing pre-#ACL2024 workshop at KAIST💙 Sharing ideas is always exciting, but it was even greater with these amazing #NLProc people!✨