Why do pre-o3 LLMs struggle with generalization tasks like @arcprize? It's not what you might think. OpenAI o3 shattered the ARC-AGI benchmark. But the hardest puzzles didn’t stump it because of reasoning, and this has implications for the benchmark as a whole. Analysis below🧵

Really good to be back in SF for GDC (yes, our game is still cooking 👀). If you're around and want to meet up next week, let me know!

RT @GregKamradt: Seeing this chart go around a bunch, I think the main point is being missed. - “LLMs can’t solve large grids because of pe….

RT @simone_m_romeo: I recommend reading @mikb0b 's article on o3's performance on the ARC challenge. He proves that LLMs' struggle with ARC….

For a deeper analysis of why o3 did so much better than previous models, and the caveats there might be in that evaluation, check out this thread!.

RT @olcan: more evidence (including experiments varying sizes of problems) that grid size alone plays a significant role in arc. this is ob….

When models can't understand the task format, the benchmark can mislead, introducing a hidden threshold effect. And if there's always a larger version that humans can solve but an LLM can't, what does this say about scaling to AGI?. Read the article here:

So even if a model is capable of the reasoning and generalization required, it can still fail just because it can't handle this many tokens. When testing o1-mini on an enlarged version of ARC, we observe an 80% drop in solved tasks - even if the solutions are the same.

LLMs are dramatically worse at ARC tasks the bigger they get. However, humans have no such issues - ARC task difficulty is independent of size. Most ARC tasks contain around 512-2048 pixels, and o3 is the first model capable of operating on these text grids reliably.

The full post goes through all these examples, and I'd love to hear your thoughts and theories:

And on a couple occasions - the model appears to give up on the 2nd attempt, in this case outputting a single black pixel? . It's unclear if OpenAI feeds in the previous attempt and tells the model it was wrong, or if something else led to this behaviour.

In several cases, we see o3 struggle to output an aligned grid at all. It seems that problems which require outputting repeated rows can make o3 struggle to keep track.

Here's the post with all the examples and analysis: You might have seen this task today on x dot com as a failure! I actually think o3's answer is as valid as the ground truth here

You've seen some of the puzzles o3 failed, but have you seen the attempts?. Yesterday, @OpenAI's o3 dramatically beat the SOTA at @arcprize. But there were 34 tasks that even it couldn't solve with 16 hours of thinking. I've compiled and analyzed all of o3's mistakes below 🧵

RT @MeganRisdal: Really great to meet and catch up with @mikb0b in person after many years! 😄

I'm heading back to San Francisco for @Official_GDC 🎮 - if anyone's around the bay area late March and wants to meet up let me know!.

I'll be speaking at @NVIDIA's AI & DS Virtual Summit about the journey to becoming the youngest Kaggle Grandmaster, along with @Rob_Mulla and @kagglingdieter. 🔥. Come and join us for a live Q&A on Wednesday 9th at 12pm PT (for free!) @NVIDIAAI

I'm going to be in San Francisco in early November! ✈️. If anyone's in the bay area and wants to meet up, or if anyone knows any events I should check out, let me know! 😊.

I've recently been playing with @fchollet's Abstraction and Reasoning Corpus, a really interesting benchmark for building systems that can reason. As part of that, I've just released a small 🐍 library for easily interacting with and visualising ARC:

Really proud to be published in a Nature Portfolio journal for the first time!. We set a new SOTA for single-cell protein localisation on the @ProteinAtlas, building on our work in the 2nd HPA Kaggle comp. @ForecomAI @cvssp_research @d_minskiy.