RT @aipulserx: How can protein representations be improved by leveraging text annotations?@NatMachIntell @UofT . "Boosting the predictive p….

RT @aidangomez: Big news today: we’ve raised $500M to grow @cohere, and have added some incredible new leaders to our team!. We’re fortunat….

What do you want to know?.

RT @cgeorgiaw: Lots of hype around “AI scientists” but they have all been focused on the fun parts, not the hard parts. The focus has bee….

RT @barrald: I love single sign on because you only have to sign on once, 8 times a day.

RT @jiarlu: Check out our recent work of benchmarking modern LLMs on the end-to-end scientific discovery tasks!! 😎👇.

📄 Paper: 💻 Code: 🌍 Website: 🗂️ Dataset: 📊 Eval:

@HaonanDuan will present this work at the GenBio workshop at #ICML25 on Friday July 18th. Welcome to chat with him and @cottascience about this work in Vancouver!.

See a demo on our website! SciGym was an incredible collaboration at UofT with amazing teammates:. @HaonanDuan*, @stephenzlu*, Caitlin F. Harrigan, @nshdesai, @jiarlu, Michał Koziarski, @cottascience.

We hope that our framework brings the rich ecosystem of systems biology models to #AI community. People can now easily build on our framework: test different perturbation strategies, vary observability, focus on specific biological domains, or add new evaluation metrics.

Critically, we find that agents struggle with complex systems and overfit to the data they've seen. When we add noise to initial conditions, their predicted models break down dramatically— showing that they only work under conditions the agent has already observed.

We tested six models across three families on 137 tasks with 2-9 missing reactions. Gemini-2.5-Pro performed best overall, and all pro versions consistently outperformed their mini variants across metrics.

From this insight, we designed SciGym: start with a complete SBML model, remove key reactions, then task agents with recovering what's missing through experimental design and analysis. This mirrors real biology problems such as reconstructing metabolic networks.

But unlike real experiments, SBML simulations are instant and cheap! ⚡ This creates perfect "dry labs" where agents can: perturb models → observe responses → form hypotheses → test with new experiments

These systems often use SBML (an XML format) to encode biological reactions as ODE systems. We can then manipulate these models programmatically—change concentrations, modify reaction rates, etc.—then simulate to see how the system responds.

Our insight: Systems biologists have spent decades encoding biochemical networks (metabolic pathways, gene regulation, etc.) into machine-runnable systems. We can use these as "dry labs" to test AI agents!.

Designing experiments involves designing systematic perturbations of a system. Think Galileo rolling balls down inclined planes at different angles to study gravity. But measuring AI's experimental capabilities is hard—real-world interactions are messy and expensive.

What makes a great scientist? Most AI scientist benchmarks miss the key skill: designing and analyzing experiments. 🧪 We're introducing SciGym: the first simulated lab environment to benchmark #LLM on experimental design and analysis capabilities. #AI4SCIENCE #ICML25

RT @micahgoldblum: 🚨 Did you know that small-batch vanilla SGD without momentum (i.e. the first optimizer you learn about in intro ML) is v….

RT @BiologyAIDaily: Measuring Scientific Capabilities of Language Models with a Systems Biology Dry Lab. １．SCIGYM introduces a new benchmar….

RT @aidangomez: Happy Canada Day 🇨🇦 May The True North remain strong and free.