How to construct repo-level coding environments in a scalable way?. Checkout RepoST: an automated framework to construct repo-level environments using Sandbox Testing (. Models trained with RepoST data can generalize well to other datasets (e.g., RepoEval)

RT @nlpxuhui: Very excited to share that HAICosystem has been accepted to #COLM2025 ! 🎉. Multi-turn, interactive evaluation is THE future,….

RepoST was accepted to @COLM_conf !!! .See you in Montreal 🚀. #COLM2025.

RT @lmathur_: Future AI systems interacting with humans will need to perform social reasoning that is grounded in behavioral cues and exter….

RT @shubhamrgandhi: 🚨New preprint🚨 .I’m super excited to share our work: An Empirical Study on Strong-Weak Model Collaboration for Repo-le….

RT @GashonHussein: Excited to share our new paper, "One-Minute Video Generation with Test-Time Training (TTT)" in collaboration with NVIDIA….

RT @jacspringer: Training with more data = better LLMs, right? 🚨. False! Scaling language models by adding more pre-training data can decre….

If you’re interested in RepoST, checkout the:.- Paper: - Code & Data: Many thanks to my awesome collaborators: Alex Xie, @Divyanshu_Sheth, @stefan_fee, @dan_fried, @carolynprose!!.

Future work may include:.(1) Training and evaluating coding agents on RepoST-Train / RepoST-Eval. (2) Extending RepoST to multiple repo-level tasks. (3) Leveraging our datasets to further study the effect of data scale and context for code generation. (4) . .

We benchmark 12 Code LLMs on RepoST-Eval to evaluate their abilities to generate code in real GitHub repositories. The best model only achieves 39.53 Pass@1. We further conducted a human study on a sampled set, where the human participants solved 81.5% of the examples.

The execution feedback of RepoST-Train enables us to apply rejection sampling to obtain training targets. The finetuned model can generalize well to other public benchmarks (e.g., we obtained performance gains of 5.49% Pass@1 on HumanEval and 3.49% Pass@1 on RepoEval)

With the RepoST framework, we build a large-scale train set, RepoST-Train, with 7,415 functions sampled from 824 repositories. We also build RepoST-Eval. Note that RepoST is fully automated and can be potentially used to construct live benchmarks to avoid contamination issues.

For quality control purposes, we iteratively resolve environment or runtime errors and improve test coverage. We also conduct execution-based, AST-based, and LLM-based quality checks. Human studies demonstrate high agreement between humans and LLM quality checkers.

Compared to integration testing used by previous datasets, our sandbox testing method is highly scalable: We only need to install packages for the target function and its necessary local dependencies, which is typically much simpler than building the entire repo.

Given a GitHub function, we sandbox it and its local dependencies to a separate script and generate tests with an LLM. When generating the target function, the model can access the entire GitHub repo. We then use the evaluation script to obtain execution feedback.

RT @PranjalAggarw16: What if you could control how long a reasoning model “thinks”?. Presenting L1-1.5B, an RL-trained reasoning model with….

RT @FariaHuqOaishi: [1/6] 🤔 Ever wondered if you could collaborate with an agent on web tasks?. ​​We present CowPilot 🐮, a framework for hu….

RT @AutoScienceAI: Introducing Carl, the first AI system to create a research paper that passes peer review. Carl's work was just accepted….

RT @jiayi_pirate: We reproduced DeepSeek R1-Zero in the CountDown game, and it just works . Through RL, the 3B base LM develops self-verifi….

RT @gaotianyu1350: Introducing MeCo (metadata conditioning then cooldown), a remarkably simple method that accelerates LM pre-training by s….