Can we detect #hatespeech at scale on social media?. To answer this, we introduce 🤬HateDay🗓️, a global hate speech dataset representative of a day on Twitter. The answer: not really! Detection perf is low and overestimated by traditional eval methods. 🧵

RT @sehgal_neil: 🚨 New study!.We tested whether AI-generated messages – single static messages vs. conversations – can boost intent to scre….

RT @sehgal_neil: 🚨 New preprint on AI persuasion and public health 🚨 . A 3-min conversation with GPT-4o nudged HPV-vax-hesitant parents (wh….

RT @lalizaveta: 🚨New article🚨 We explore the potential of popular chatbots to detect misinformation & propose a new conceptual framework fo….

RT @oiioxford: New blog! @oiioxford doctoral researchers @deeliu97.@ManuelTonneau and Juliette Zaccour propose a series of recommendations….

Your feedback is much appreciated as we prepare the final version of the paper. We would like to thank @JurgenPfeffer and team who collected the TwitterDay dataset from which HateDay is sampled and without whom this work would not have been possible! 🙏🙏🙏.

What about moderation? Given low perf, automatic moderation is not desirable. We investigate the feasibility of human-in-the-loop moderation where models flag and humans verify. Moderating >80% of all hate would require humans to review >10% of all daily tweets which can get 💸💸

We also find other reasons for low perf, such as the misalignment between target focus in academic work and target prevalence in the wild, as well as the difficulty to distinguish use and mention of hate presented in past work from @krisgligoric.

Why is perf so low? An important reason is it is hard to distinguish between offensive and hateful content (as exposed by @thomasrdavidson in seminal work) and offensive content is much more prevalent than hate in the wild, crowding out hate in the predicted positives

We then evaluate popular hate speech detection LLMs on HateDay and compare with their performance on academic hate speech datasets and functional tests (HateCheck). We find that traditional eval methods systematically overestimate performance on representative data, which is low.

We first look at the prevalence and composition of hate in HateDay and find that most types of hate are represented across contexts, with some local specificities in the importance of each hate type (e.g., green-bashing in German tweets, islamophobia in India).

RT @agostina_cal: I'm creating a 🦋 starter pack for researchers working on (fighting) online harms, if you would like to be added send alon….

RT @WOAHWorkshop: 🥳We are excited to share that WOAH 2025, our 9th edition, will take place at #ACL2025 in Vienna! @aclmeeting . Our speci….

Our #hatespeech supersets, combining all 🤬 corpora in 8 lang, reached 2K downloads on 🤗🎉. To enable more cross-cultural 🤬 research, we release:.- author country for English, Arabic and Spanish posts from our @WOAHWorkshop 📜.- 3 country sets 🇮🇳🇳🇬🇰🇪.

RT @DG_Rand: 🚨Out in Nature!🚨.Many (eg Trump JimJordan @elonmusk) have accused social media of anti-conservative bias.Is this accurate?.We….

RT @c_schwemmer: We are looking for a student to assist our team in using AI to detect toxic content on social media. If you are familiar w….

RT @enfleisig: Does ChatGPT discriminate against speakers of different dialects? Our #EMNLP2024 paper finds that ChatGPT exhibits consisten….

RT @gvrkiran: Video data is widely available but has been difficult to analyze automatically. Particularly, TV data is a rich and important….

RT @davidschlangen: Time to accept we didn't manage to pull hybrid conferences off. But travel must still be reduced, no way around it. So….

Your feedback is super welcome! We are also expanding this analysis beyond 🇳🇬 with a paper out soon, so stay tuned! . This was fun team work and wouldn't have been possible without the awesome work from our annotators. 🙏🙏🙏.