Visit our poster2, #68, at Hall 3 + Hall 2B on Saturday, April 26th, from 3.00 PM to 5:30 PM SST

8/n.Future trends? Expanding from humans to animal models for brain-machine alignment & exploring neural population control—the next frontier!.

7/n .Conclude with key challenges such as issues in processing neural recordings, selection of appropriate evaluation metric, & ethical considerations in neuro-AI.

6/n.Evaluation Metrics:.We highlight popular metrics for assessing 🧠 encoding and decoding models. Insights from Recent Studies:.We provide insights from recent 🧠 encoding and decoding studies, focusing on language, auditory, visual, and multimodal stimuli.

5/n.Ecological Neuroimaging Datasets:.We discuss widely used, publicly available datasets involving audio, visual (reading and viewing), and other multimodal stimuli that have been utilized in the field.

4/n.In this review, we cover several key aspects of brain encoding and decoding studies:. Stimulus Representations:.We explore various stimulus representations proposed in the literature across different modalities, including text, visual, audio, video, and multimodal stimuli.

3/n .3️. Incorporating brain recordings into DNN training pipelines. 4️. Advancing brain-computer interfaces (BCIs) for decoding brain patterns. 5️. Utilizing curated ecological stimuli datasets. 6️. Understanding the taxonomy of CCN models and methodologies.

2/n DNN researchers can benefit from an interdisciplinary approach to CCN in several ways:. 1️. Interpreting DNN models and evaluating their capabilities using naturalistic brain datasets. 2️. Leveraging open-source brain datasets as benchmarks to enhance DNN performance.

1/n Our review paper introduces the challenges in Computational Cognitive Neuroscience (CCN) to AI researchers familiar with recent advances in deep neural networks (DNNs). We highlight how DNNs are being applied to CCN problems, providing an entry point for DNN researchers to.

🚀 Check out our recently accepted work at #TMLR Transactions on Machine Learning Research 🧠🤖. Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey).📄 Read it here: Work done jointly with @ZijiaoC, @blitzprecision,

RT @gab709_1: 🚨 Thrilled to announce that our paper "Language models and brains align due to more than next-word prediction and word-level….

I will be at #ACL2024NLP from Aug 11-16 @Bangkok!. Excited to present our work: Speech language models lack important brain-relevant semantics. @emincelik, Fatma Deniz, @mtoneva1. Visit our poster during in-person poster session 4 on Tuesday, Aug 13th, from 10:30 AM to 12:00 PM

8/n .3 main conclusions:.- Speech models useful for modeling early listening: investigate speech models to learn more about AC.- Text models useful for modeling late language in both listening and reading.- But, more work to do for a complete end-to-end model of reading and.

7/n .For speech language models, we find that:.- Some of the very high alignment with early AC (~75% of estimated noise ceiling) due to low-level speech features, but much residual alignment remains.- Alignment in late language regions entirely due to low-level stimulus features.

6/n.For text LMs, we find that:.- Alignment with early auditory cortex (AC) during listening is due to low-level textual features, which correlate with low-level speech features processed in the AC.- High alignment in late language regions (~60% of estimated noise ceiling) is not.

5/n.We investigate these questions via a direct approach, by removing the information related to low-level textual, speech, & visual features and analyzing how this removal impacts the alignment with brain recordings of participants listening to and reading the same stories.

More generally, w𝐡𝐚𝐭 𝐭𝐲𝐩𝐞𝐬 𝐨𝐟 𝐢𝐧𝐟𝐨𝐫𝐦𝐚𝐭𝐢𝐨𝐧 𝐝𝐨 𝐥𝐚𝐧𝐠𝐮𝐚𝐠𝐞 𝐦𝐨𝐝𝐞𝐥𝐬 𝐭𝐫𝐮𝐥𝐲 𝐩𝐫𝐞𝐝𝐢𝐜𝐭 𝐢𝐧 𝐭𝐡𝐞 🧠?.

𝐃𝐨 𝐭𝐞𝐱𝐭-𝐛𝐚𝐬𝐞𝐝 (📖) 𝐚𝐧𝐝 𝐬𝐩𝐞𝐞𝐜𝐡-𝐛𝐚𝐬𝐞𝐝 (🔊) 𝐥𝐚𝐧𝐠𝐮𝐚𝐠𝐞 𝐦𝐨𝐝𝐞𝐥𝐬 𝐡𝐚𝐯𝐞 𝐭𝐡𝐞 𝐬𝐚𝐦𝐞 𝐝𝐞𝐠𝐫𝐞𝐞 𝐨𝐟 𝐩𝐫𝐞𝐝𝐢𝐜𝐭𝐢𝐯𝐢𝐭𝐲 𝐢𝐧 𝐭𝐡𝐞 𝐥𝐚𝐧𝐠𝐮𝐚𝐠𝐞 𝐫𝐞𝐠𝐢𝐨𝐧𝐬 𝐨𝐟 𝐭𝐡𝐞 🧠?.

𝐃𝐨 𝐭𝐞𝐱𝐭-𝐛𝐚𝐬𝐞𝐝 (📖) 𝐚𝐧𝐝 𝐬𝐩𝐞𝐞𝐜𝐡-𝐛𝐚𝐬𝐞𝐝 (🔊) 𝐥𝐚𝐧𝐠𝐮𝐚𝐠𝐞 𝐦𝐨𝐝𝐞𝐥𝐬 𝐡𝐚𝐯𝐞 𝐭𝐡𝐞 𝐬𝐚𝐦𝐞 𝐝𝐞𝐠𝐫𝐞𝐞 𝐨𝐟 𝐩𝐫𝐞𝐝𝐢𝐜𝐭𝐢𝐯𝐢𝐭𝐲 𝐢𝐧 𝐭𝐡𝐞 𝐥𝐚𝐧𝐠𝐮𝐚𝐠𝐞 𝐫𝐞𝐠𝐢𝐨𝐧𝐬 𝐨𝐟 𝐭𝐡𝐞 🧠?.

These observations raise several research questions:. 𝐖𝐡𝐲 𝐝𝐨 𝐭𝐞𝐱𝐭-𝐛𝐚𝐬𝐞𝐝 (📖) 𝐥𝐚𝐧𝐠𝐮𝐚𝐠𝐞 𝐦𝐨𝐝𝐞𝐥𝐬 𝐡𝐚𝐯𝐞 𝐢𝐦𝐩𝐫𝐞𝐬𝐬𝐢𝐯𝐞 𝐩𝐞𝐫𝐟𝐨𝐫𝐦𝐚𝐧𝐜𝐞 𝐢𝐧 𝐭𝐡𝐞 𝐞𝐚𝐫𝐥𝐲 𝐚𝐮𝐝𝐢𝐭𝐨𝐫𝐲 𝐜𝐨𝐫𝐭𝐞𝐱?.