Shaokai Ye
@shaokaiyeah
Followers
209
Following
333
Media
14
Statuses
124
PhD student at MathisLab Bridging AI and neuroscience!
Geneva
Joined May 2018
Insightful! I have been thinking about "thinking in video" too.
Some further thoughts on the idea of "thinking with images":. 1) zero-shot tool use is limited -- you can’t just call an object detector to do visual search. That’s why approaches like VisProg/ViperGPT/Visual-sketchpad will not generalize or scale well. 2) visual search needs to
0
0
2
RT @sainingxie: About two years ago, we started building V* to bring visual search into a multimodal LLM and show t….
0
3
0
I am on the job market! Contact me if you are interested in building multi-modal agentic systems that understand behaviors!.
✨ Introducing a new #SOTA action recognition large multimodal language model: #LLaVAction!. Understanding human behavior requires recognizing actions—a challenging task given the complexity of behavior. Large multimodal language models (#MLLMs) offer a promising path forward,
0
0
2
So sweet to see them standing together.
0
0
1
RT @arankomatsuzaki: Automating the Search for Artificial Life with Foundation Models. Presents ASAL, the first use of vision-language mode….
0
40
0
RT @DeepLabCut: 🚀 DLC AI Residents call for 2025! This year, we are doing something a little different to support even more underrepresente….
0
12
0
RT @TrackingActions: 🚨adversarial robustness is becoming even more critical as AI systems are deployed in the real-world, but how can we de….
0
13
0
Really cool!.
✨🥰 check out our article - and cover 🤩- about Decoding the Brain in @CellCellPress . We review the mathematics, current approaches, and muse about the future…. #BCI #neuraldecoding #neuroAI. Thanks to my awesome co-authors Adriana Perez Rotondo,
0
0
4
Come check this really cool paper if you are at ECCV!.
I'm thrilled to present our paper "Elucidating the Hierarchical Nature of Behavior with Masked Autoencoders" at #ECCV24! 🎉 We focus on self-supervised learning for action segmentation to uncover the hierarchical structure of behavior. Paper: 👇🧵.
0
0
0
RT @TrackingActions: How does sensorimotor (S1/M1) cortex support adaptive motor control? . Come find out in our latest preprint, which sp….
0
101
0
RT @DeepLabCut: DLC Residency 2024 Recap! 🎓💜 . 🎉Check out the awesome video from DLC Community Manager, @TheVetFuturist, and the 2024 Resid….
0
6
0
Discover more at ! Send us your feature requests and post issues!.
0
0
1
Step by step guide from @StuckertAnna for using #SuperBird to analyze kinematics of two birds!
1
0
0
Step by step guide from @dikra_masrour for using #SuperBird, a new member of #SuperAnimal to detect picking!
1
1
2
We are releasing AmadeusGPT-0.1.3!. Many use cases to demonstrate its capability. We also have invited DLC residents (.@StuckertAnna , @mrc_canela, @dikra_masrour to share their use cases. Seamless integration with #SuperAnimal and GPT-4o
🔮Ever wished you could just say what behavioral analysis you wanted? . "Plot the animal's trajectory" "Count the head hips in the open arm". 👀🪄we present AmadeusGPT🎻 to do this . 🗞@shaokaiyeah @jessy_lauer @zhoumu53 @trackingplumes and me 🧵⬇️
1
6
19
RT @DeepLabCut: 🎉 introducing the 2024 AI Residents!!! . We welcome @StuckertAnna @mrc_canela and @dikra_masrour 👏. Read about them on the….
0
8
0
RT @TrackingActions: 🎉 Happy to have contributed a bit to this with my PhD student @shaokaiyeah, where we use @DeepLabCut #SuperAnimal foun….
0
11
0
the pace of progress blows my mind.
0
0
2