It's a wrap! Across 46 new benchmarks from 8 submitters, we have now evaluated 19 reference models for a total of 874 alignment scores. These new behavioral and neural benchmarks are showing the short-comings of our models (although some occasionally prevail). Winners TBA at CCN!

All the benchmarks from the Brain-Score Benchmarking Competition @CogCompNeuro are now live on https://t.co/pQcbTHkFHt. Huge thanks again to all who submitted! We're looking forward to new models improving on those benchmarks

Please join us tomorrow at 2 pm at Sala de Puerto Rico at MIT for @CogCompNeuro 2024 Community event. We will have talks from the winners of the Brain-Score benchmarking competition ( https://t.co/k2OxnSmSwJ), open discussions about the scope and future of such platforms and

Thanks so much to all the contributors! We honestly didn't expect this many submissions, and even better there are now new benchmarks as goal posts for computational models. See you at CCN in August :)

By popular demand we have extended the competition deadline until *July 8*. Please make sure to submit your final benchmarks by then, and we will let you know asap if you broke the SOTA models :)

The 2024 Brain-Score competition now has a confirmed $10k prize pool. Prizes are split up across behavioral and neural benchmark submissions. Only a little over 20 days left!

The 2024 Brain-Score Benchmarking Competition is now officially part of the @CogCompNeuro schedule. We will announce winners as part of the CCN conference -- only 1.5 months left to submit your benchmark!

This year, we want to convert the existing data, hunches, and anecdotes into quantifiable frustration and pain for the models and modelers. Let's disrupt the rankings and iteratively progress towards more robust models of vision. Please join @martin_schrimpf and me (@KohitijKar)

Well that should be easy!

This is an absolutely brilliant idea! Bravo!

well now that's something new! Very curious to see the entries :)

Super excited for the 2024 Brain-Score Benchmarking Competition. Submissions will be rewarded on how much the current best models fail on them! We have been working hard to make model testing easy and I hope this competition uncovers some explanatory gaps in our models of vision

There are two tracks to participate in (behavioral + neural), both with multiple prizes. Submissions are due by the end of June, and the winners will be invited to present at MIT in August at a workshop co-occurring with CCN. Happy benchmarking! /fin

We provide the entire compute infrastructure along with model implementations https://t.co/7iNucgEpLi (also see the tutorials). In addition, we will incentivize benchmark submissions by providing visibility to participants and monetary prizes (TBA) to the winning benchmarks. 4/

This edition of the Brain-Score Competition aims to find benchmarks on which the predictions of our current top models break down. We will reward those benchmarks that show where models are the least aligned to the visual ventral stream (human + non-human primate). 3/

This competition is partly motivated by critiques of the Brain-Score platform we have been hearing over the years: that models should be tested on different data, in a different way, and all models will fail on benchmark X. We built the tools to execute on these claims. 2/

🧠 Announcing the 2024 Brain-Score Benchmarking Competition 📣 This year — we have turned the table! We invite experimentalists and the community at large to expose the explanatory gaps between current models of primate vision and the biological brain. https://t.co/adSr5H2pmJ 1/

15/ These are only some of the directions we are currently exploring that stem from having brain-aligned models of machine vision (e.g. see @brain_score), so if you are a small/medium/large business or research lab and would like to work with us, also let us know!

2/4 But can't neuro up its contribution via large-scale datasets of neural responses on tasks that AI isn't yet good at? These can be either benchmarks (as in @brain_score) or as optimization targets (cool progress from Leila Wehbe @mldcmu & co using @cvnlab's NSD dataset).