I'm excited to share our latest work — "Data Heterogeneity Limits the Scaling Effect of Pretraining in Neural Data Transformers", where we examined the effect of scaling up pretraining data in neural foundation models carefully.🧐 (1/9). Preprint:

This is joint work with @ChinSengi, Iman Tanumihardja, @XiaochuangHan, @WeijiaShi2, Eric Shea-Brown, and @RajeshPNRao. Please check out the preprint for more details. Any feedback is appreciated! (9/9).

Together, our results show that pretraining with more sessions does not naturally lead to improved downstream performance. We advocate for rigorous scaling analyses in future work on neural foundation models to account for data heterogeneity effects. (8/9).

We note that similar results have been found in NDT3 by @_JoelYe, where several downstream datasets enjoyed little benefit from scale with 100 minutes of finetuning data. (7/9).

We found that models trained with as few as five top-ranked sessions outperformed those with randomly chosen sessions even when the full dataset was used, demonstrating the impact of session-to-session variability in performance scaling. (6/9)

For the forward-prediction task that did exhibit consistent scaling, we identified implicit data heterogeneity arising from cross-session variability. We proposed a session-selection procedure based on single-session finetuning performances. (5/9)

In this work, we systematically investigate how data heterogeneity impacts the scaling behavior of neural data transformer. We first found that brain region mismatches among sessions reduced scaling benefits of neuron-level and region-level activity prediction performances. (4/9)

Yet, previous studies typically lack fine-grained data scaling analyses. It remains unclear whether all sessions contribute equally to downstream performance gains. This is especially important to understand as pretraining scales to thousands of sessions and hours of data. (3/9).

Neural foundation models are gaining increasing attention these days, with the potential to learn cross-session/animal/species representations and benefit from multi-session pretraining. (2/9).

RT @WeijiaShi2: Our previous work showed that 𝐜𝐫𝐞𝐚𝐭𝐢𝐧𝐠 𝐯𝐢𝐬𝐮𝐚𝐥 𝐜𝐡𝐚𝐢𝐧‑𝐨𝐟‑𝐭𝐡𝐨𝐮𝐠𝐡𝐭𝐬 𝐯𝐢𝐚 𝐭𝐨𝐨𝐥 𝐮𝐬𝐞 significantly boosts GPT‑4o’s visual reasoning….

RT @rockpang6: I'm honored and humbled to receive the @IBM Ph.D. fellowship! Thank you to my advisor @_doctor_kat, all my friends, mentors….

RT @JieyuZhang20: Excited to share my intern project at Salesforce Research! Huge thanks to everyone on the team!!.

RT @WeijiaShi2: Introducing 𝐋𝐥𝐚𝐦𝐚𝐅𝐮𝐬𝐢𝐨𝐧: empowering Llama 🦙 with diffusion 🎨 to understand and generate text and images in arbitrary sequen….

RT @cloudwaysX: I will be at #NeurIPS2024! 🚨On academic/industry job market this year 🚨and excited to catch up in person! .My research foc….

Bring attention to the communications committee listed on the website: @neuro_kim @smfleming @somnirons.

RT @RajeshPNRao: Excited to announce that Nature Neuroscience @NatureNeuro has published my article "A sensory-motor theory of the neocorte….

RT @RajeshPNRao: Final version of our Dynamic Predictive Coding paper with @lpjiang97 is now out in @PLOSCompBiol - explains V1 space-time….

RT @ylecun: The 2024 Brain Prize goes to pioneers of computational and theoretical neuroscience: Larry Abbott, @HSompolinsky, and Terry Sej….

A huge thanks to @RajeshPNRao for supporting me in this process, and @sathish_vishwas Dimi Gklezakos Ares Fisher @DaogaoLiu for discussions!. Any feedback is welcome!.

A deeper three-level model shows the capability of DPC to capture progressively longer temporal regularities in the data. Coincidentally, it offers a neural solution to the exact question proposed by MacKay in his classic "The Epistemological Problem for Automata" (1954)!.