Quick weekend project: how good are LLM's at "Who's That Pokémon?" . answer: not great!. I tested some of the best models on a simple game segment from the show with a small benchmark I call PokeShadowBench. some results below

didn't expect this to get as much attention as it did! . we're working on more exciting things so if you're looking to understand problems with your agents or want to make them better, message me or @JuliaANeagu and let us know 🙂 also try Quotient.

blog post later this week more details! if you're having problems with tool use in your agents, what are your biggest issues right now? where do you see things go wrong?. let me know how limbic can help or what's missing.

this is part of our new system we're building called limbic, which captures and processes agent behavior, helps you understand it, and automatically improves your agents for you. we'll be adding a more accurate model to Detections (our async processing component) soon.

if you want to try the model out, check out the huggingface link. alternatively, you can send requests to a deployed version up on @modal_labs. gist here: trained with @unslothai btw

of course, this wouldn't be complete without some eval dataset, so we're also releasing the test set on huggingface with info on our training data curation pipeline.

it's open weights and available on @huggingface .

today we're releasing a new small model (0.5B) for detecting problems with tool usage in agents, trained on 50M tokens from publicly available MCP server tools. it's great at picking up on tool accuracy issues and outperforms larger models

RT @jxnlco: how you can catch hallucniations in production with @QuotientAI . sign up for study notes and recordings afterwards even if you….

RT @JuliaANeagu: Just dropped: three new cookbooks for building AI research agents with @ExaAILabs, @LangChainAI, @OpenAI, and @AnthropicA….

this book is actually a very good read about context engineering. chapter 5 and 6 is all about building context and chapter 8 is all about building sources for context.

RT @JnBrymn: In light of all the attention that context engineering is getting, today I proudly introduce the second book that Albert Ziegl….

RT @JuliaANeagu: If you're shipping LLMs to production and still finding out about critical from your users, this course is for you. Real-….

RT @code_star: Should I change my bio?.

still meeting plenty of people who haven’t even heard of ChatGPT.

John is a great person to chat with and learn from — check this out 👀!.

if you’re at @vercel Ship ask Hai about . have used it a bunch for understanding new and trending github repos it is nice 👌🏼.

RT @JuliaANeagu: “You want your model hitting milestones, not minefields.”. Most AI eval talk is hand-wavy. This isn’t. @freddie_v4 (@Quot….

it’s good everyone here can share the misery of the internet being broken. hopefully this doesn’t jinx it.

so who’s the bad node in the dependency graph right now? . supabase, aws, gcp, cursor, cloudflare, azure… surely something upstream?.