Deep Language Models are getting increasingly better by learning to predict the next word from its context. Is this really what the human brain does? Here, we hypothesize that our brain 🧠 rather makes distant and hierarchical predictions.

To test this hypothesis, we first confirm that language models like GPT-2 build language representations partly similar to those of the brain, with the fMRI brain recordings of 345 subjects listening to stories.

Then, we further trained GPT-2 with two different objectives: 1. The classic next-word prediction loss 2. A hierarchical loss to predict latent and distant representations of the future. Our results show that the hierarchical model is more similar to the brain than the other.

To clarify how these hierarchical predictions are organized in the brain, we manually enhanced language models with different types of predictions. For each brain region, we assess whether brain activity is best accounted for by shallow or deep predictions.

Similarly, we assess whether brain responses are best modeled by proximal or distant predictions. The results reveal a hierarchy of predictions in the 🧠: the fronto-parietal areas predict deeper & more distant representations than the superior temporal areas.

Overall, these results strengthen the importance of distant and hierarchical predictions in natural language processing, and thus pave the way towards better algorithms inspired by the human brain.

Thanks @MetaAI, @inria, @ENS_ULM, @NatureHumBehav, @samnastase @HassonUri, @nilearn, @huggingface and @scikit_learn for all the support 🙏

@c_caucheteux Nice knowledge share Charlotte! May language help us all grow in awesome new ways! Movement of sound creates life!

@c_caucheteux Very interesting subject!, Let's check that out! Love the fMRI brain recordings picture you show there!