Pumped to share this work from my PhD, “A single computational objective drives specialization of streams in visual cortex”, just in time for the holidays! With the all star team @eshedmargalit , @cvnlab , @dyamins & @kalatwt .🔗:🧵: 1/n….

RT @dyamins: 1/ Our work on unified principles for Topographic Deep Artificial Neural Networks is finally out in Neuron! 7 years in the mak….

RT @AndrewLampinen: What is representational alignment? How can we use it to study or improve intelligent systems? What challenges might we….

22/ Whew, that's it! Checkout our preprint for more (including effective dimensionality, task transfer performance benefits from adding the spatial loss & model unit RF properties).

21/ As parallel processing streams exist in other species, cortical systems, and scales, we’re curious to what extent similar principles may explain the emergence of parallel processing streams across the brain.

20/ I was particularly surprised that a local spatial constraint, which encourages clustering of similar tuning within small neighborhoods of units, can percolate up to create such broad-scale stream structure.

19/ better explains both the functional and spatial organization of the human visual system into processing streams than a system trained to perform multiple visual behaviors in parallel.

18/ In other words, the surprising result from this project is that a single, biologically plausible computational principle – self-supervised learning of the statistics of visual inputs under a local spatial constraint that encourages nearby units to have correlated responses –.

17/ In comparison, the task performant DNNs implementing the multiple behavioral demands hypothesis not only poorly capture the spatial segregation into streams, but also relatively poorly predict neural responses.

16/ That is, despite the single unified training, TDANN model units within the different streams develop different functional properties that are better suited for the visual behavior of their corresponding stream.

15/ And despite being trained without any explicit task differentiation built in, functional differentiation emerges naturally in exactly the pattern neuroscientists have observed empirically.

14/ Okay back to the main point: we find that the TDANN successfully predicts the spatial segregation into streams and the neural responses in each stream, even approaching the brain-to-brain noise ceiling in Dorsal and Ventral!

13/ Lots of excellent work is being done on this topic; (@meenakshik93, @itsneuronal) for a recent fav.

12/ but also if our models get good enough, we could use these mappings to run all sort of simulations and spatially-localized lesion studies in silico that we can’t do in the actual human brain!.

11/ Brief aside: I’m particularly excited about stricter mapping generally. Not only will 1-to-1 mappings allow us to more clearly arbitrate between different models of the brain (for ex, we see a massive improvement for the bio plausible self-sup over supervised training),.

10/ This not only provides a stricter test of models than previous mapping methods, but also enables a direct comparison of both function and spatial arrangement (topography) between model and brain.

9/ Given brain activations to the images & then activations from our candidate DNNs to these same images, we directly map from each model unit to a brain voxel using a new 1-to-1 mapping algorithm.

8/ We then test these hypotheses to see if they can replicate human visual streams both spatially & functionally using the massive Natural Scenes Dataset (@Emsquem) - a high-resolution human fMRI dataset with cortical responses to ∼10,000 MS-COCO images in each of 8 participants.

7/ Unlike standard DNNs, the TDANN: (1) embeds model units in a 2-dim simulated cortical sheet & (2) during training learns to balance a self-supervised goal (simCLR) w/ a spatial constraint which minimizes wiring length by encouraging nearby units to have correlated responses.

6/ and implementing the spatial constraints hypothesis using the recently developed Topographical Deep Artificial Neural Network (TDANN; check out @eshedmargalit 's )

5/ We use DNNs to computationally formalize these hypotheses, implementing the multiple behavioral demands hypothesis using models trained for the specific visual behaviors empirically associated with each stream.