RT @ori_press: Do language models have algorithmic creativity?. To find out, we built AlgoTune, a benchmark challenging agents to optimize….

RT @SWEbench: We just updated the SWE-bench Multimodal leaderboard with new systems from @refact_ai, @allhands_ai and @TU_Muenchen. Congrat….

RT @a1zhang: Can GPT, Claude, and Gemini play video games like Zelda, Civ, and Doom II?. 𝗩𝗶𝗱𝗲𝗼𝗚𝗮𝗺𝗲𝗕𝗲𝗻𝗰𝗵 evaluates VLMs on Game Boy & MS-DOS….

Sonnet 3.7 was plateauing at $1.5 (the bump at the end is an artifact from correct solutions from terminated runs), so increasing cost limit to $3 would barely have made a difference for this comparison. Sonnet 4 at $3 still had 10 instances terminated for cost.

"Correctness" corresponds to the resolution rate among all runs that terminated without being killed due to cost etc. "Incorr. localization" means that some files from the gold patch weren't edited. "Incorr. edit" are all failed submissions that aren't in "incorr. localization".

Both compared agent runs correspond to near identical configurations here

Massive gains with Sonnet 4 on SWE-agent: Single-attempt pass@1 rises to 69% on SWE-bench Verified! Sonnet 4 iterates longer (making it slightly more expensive) but almost never gets stuck. Localization ability appears unchanged, but quality of edits improves.

RT @jyangballin: 40% with just 1 try per task: SWE-agent-LM-32B is the new #1 open source model on SWE-bench Verified. We built it by synt….

RT @plodq: Introducing SWE-bench Multilingual: a new eval in the SWE-bench family to test LLM coding abilities in *9* programming languages….

Maps. Diagrams. UI glitches. SWE-bench Multimodal benchmarks AI agents on real-world frontend issues and they struggle. Poster at #ICLR25 today. Multiple submissions to the leaderboard already.

Had a great time talking about building agents, SWE-agent, SWE-bench, and more.

Find it at The main code boils down to a few hundred lines, making it super easy to adapt. If you write an agent that requires code execution in (possibly multiple) shell sessions, this takes all the dirty work with managing environments off your hands.

Evaluating SWE-agent on SWE-bench lite was once an overnight job. With SWE-ReX parallelizing our execution it now takes half an hour! SWE-ReX spins up docker containers with a @FastAPI server that uses pexpect to interface with shell sessions. MIT licensed, lightweight & hackable

RT @OfirPress: The creators of LiveCodeBench just released a new, private, SWE-bench like benchmark in Java, C++, Python, JavaScript, TypeS….

Join @_carlosejimenez and me today at GenAI Collective NYC as we break down SWE-Bench, SWE-agent, and the future of AI-driven software engineering. What works? What’s next? What does this mean for developers? Let's discuss!. 📍 Today 1–4pm, Brooklyn Navy Yard.

RT @OfirPress: We just updated the SWE-bench Multimodal leaderboard. Congrats to Globant, Zencoder, and the Agentless team from UIUC for th….

RT @daytonaio: Watch Princeton's SWE-agent @KLieret reveal research on autonomous coding agents at Daytona AI Builders @github HQ! From ben….

SWE-agent 1.0 is so much more flexible than before. It has never been easier to set it up with various tool bundles or multiple LMs. And you can combine them all in a multi-attempt scheme!.

Join at