There's a lot more in the paper (lookin' at you, prosody) -- we hope our work demonstrates the power of using LLMs as alternative models to understand language writ large! None of this would have been possible without the support of @esfinn and the FinnLab :).

How does sensory context facilitate language acquisition? We hypothesized that training LLMs with richer sensory context would partially ”rescue” performance relative to humans. LLMs trained with richer sensory context performed better with much less data (~20%)!

However, LLMs can still be useful models of behavior – LLM prediction distributions better reflected human predictions when reading written language.

Human predictions were more accurate than LLMs, even when just reading! This performance gap scaled with the amount of sensory context provided to humans.

We collected human predictions of natural language under varying sensory conditions (written, audio- only, or audiovisual language) and compared these with LLM predictions to the same stories.

Humans language is informed by audiovisual information. Yet, LLMs are often trained solely on written text – i.e., their view of language lacks the sensory richness of our human experience. How essential are these sensory contexts for cultivating "human-like" language?.

New preprint! Thrilled to share my latest work with @esfinn -- "Sensory context as a universal principle of language in humans and LLMs".

RT @esfinn: Our lab ( is excited to review PhD applications this cycle! Check us out if you wanna do interesting st….

RT @c_welk: So excited to introduce this new work from myself, @ThaliaWheatley , and @meghanlmeyer out now in @PNASNews! . .

RT @SfNJournals: New in #JNeurosci from @tommy_botch and @esfinn: Stable neural patterns that underly how we interpret words grounded in se….

RT @ericalbusch: Come visit me at poster 2051 to learn about neural manifold constraints on neurofeedback learning with real-time fMRI! Thi….

How much do we really understand about how the brain processes movies/stories? Come to poster #0948 today/tomorrow at #OHBM2024! With @esfinn and our all-star collaborators @HayoungSong5, @headspace_lab, and @monicarosenb

RT @HayoungSong5: Excited for #OHBM2024 in Seoul!🇰🇷 I'll be presenting "Neural mechanisms of insight during narrative comprehension" on Pos….

RT @Eshjolly: Some very exciting news. after many, many, winter beards, I'm overjoyed to be joining sunny @UCSDPsychology as an Asst Prof….

RT @InnovatorsTalk: Our upcoming presentation will be given by Dr. Ali Cohen (@aliocohen; @EmoryPsychology) on "Motivated learning and memo….

RT @GretaTuckute: 1/.Really excited to share:.Language in Brains, Minds, and Machines w @Nancy_Kanwisher @ev_fedorenko.@AnnualReviews. We s….

RT @Guo_Jiahui_: My lab (CoCo-Neuro) at UTD is recruiting a lab manager/tech to start in the summer of 2024 (2 years preferred). If you hav….

RT @Rekha_Va: Today at #SANS2024. I'll be presenting our work that looks at the shared and individual processes involved when people interp….

RT @choongwanwoo: Finally, out today in PNAS: "Brain decoding of spontaneous thought: Predictive modeling of self-relevance and valence usi….

RT @avmello_: Happy to share that my first first-author paper is out @TheLancet! . Here, we present a case of PMO, a striking neurological….