We release stereo models for all MusicGen variants (+ a new large melody both mono and stereo): 6 new models available on HuggingFace (thanks @reach_vb). We show how a simple fine tuning procedure with codebook interleaving takes us from boring mono to immersive stereo🎧👇

Thanks @MetaAI, @inria, @ENS_ULM, @NatureHumBehav, @samnastase @HassonUri, @nilearn, @huggingface and @scikit_learn for all the support 🙏

Overall, these results strengthen the importance of distant and hierarchical predictions in natural language processing, and thus pave the way towards better algorithms inspired by the human brain.

Similarly, we assess whether brain responses are best modeled by proximal or distant predictions. The results reveal a hierarchy of predictions in the 🧠: the fronto-parietal areas predict deeper & more distant representations than the superior temporal areas.

To clarify how these hierarchical predictions are organized in the brain, we manually enhanced language models with different types of predictions. For each brain region, we assess whether brain activity is best accounted for by shallow or deep predictions.

Then, we further trained GPT-2 with two different objectives: 1. The classic next-word prediction loss 2. A hierarchical loss to predict latent and distant representations of the future. Our results show that the hierarchical model is more similar to the brain than the other.

To test this hypothesis, we first confirm that language models like GPT-2 build language representations partly similar to those of the brain, with the fMRI brain recordings of 345 subjects listening to stories.

Deep Language Models are getting increasingly better by learning to predict the next word from its context. Is this really what the human brain does? Here, we hypothesize that our brain 🧠 rather makes distant and hierarchical predictions.

Our paper is out in Nature Human Behaviour🔥🔥 ‘Evidence of a predictive coding hierarchy in the human brain listening to speech’ 📄 https://t.co/bkZ3AYMqDi 💡Unlike language models, our brain makes distant & hierarchical predictions with @agramfort and @JeanRemiKing Thread👇

Happy to release a collection of LLaMA 🦙, large language models ranging from 7B to 65B parameters and trained on publicly available datasets. LLaMA-65B is competitive with Chinchilla and PaLM. Paper:

Today we release LLaMA, 4 foundation models ranging from 7B to 65B parameters. LLaMA-13B outperforms OPT and GPT-3 175B on most benchmarks. LLaMA-65B is competitive with Chinchilla 70B and PaLM 540B. The weights for all models are open and available at https://t.co/q51f2oPZlE 1/n

🙏 Thanks to @samnastase, @HassonUri, John Hale, @nilearn, @pyvista and the open-source and open-science communities for making this possible! 7/7

This is a joint work with our great team 🤩🤩 Juliette Millet, @PierreOrhan, Y Boubenec, @agramfort, E Dunbar, @chrplr and @JeanRemiKing, at @MetaAI, @ENS_ULM, @Inria & @Neurospin 6/n

Conclusion: Modeling human-level intelligence is a far-off goal. Still, the emergence of brain-like functions in self-supervised algorithms suggests that we may be on the right path. 5/n

Result 3: With an additional 386 subjects, we show that wav2vec 2.0 learns both the speech-specific and the language-specific representations of the prefrontal and temporal cortices, respectively. 4/n

Result 2: The hierarchy learnt by the algorithm maps onto the brain's: The auditory cortex is best aligned with the first layer of the transformer (blue), whereas the prefrontal cortex is best aligned with its deepest layers (red). 3/n

Result 1: self-supervised learning suffices to make this algorithm learn brain-like representations (i.e. most brain areas significantly correlate with its activations in response to the same speech input). 2/n

Question: can a model trained on a *plausible* amount of *raw* speech explain both intelligent behavior and its brain bases? Here, we train wav2vec 2.0 w/ 600h of audio and map its activations onto the brains of 417 volunteers recorded with fMRI while listening to audio books.

🔥Our work has now been accepted to NeurIPS 2022 !! `Toward a realistic model of speech processing in the brain with self-supervised learning’: https://t.co/weiGlaiD65 Let’s meet in New Orleans on Tue 29 Nov 2:30pm PST (Hall J #524). A recap of the 3 main results below 👇

Thanks to @samnastase and @HassonUri's lab for having publicly released their dataset, as well as @nilearn, @huggingface, @Inria, @ENS_ULM and @MetaAI for making this possible 🙏! 9/9