I’m at Stanford and I research software engineering productivity. We have data on the performance of >50k engineers from 100s of companies. Inspired by @deedydas, our research shows: ~9.5% of software engineers do virtually nothing: Ghost Engineers (0.1x-ers)

Had a great time at NeurIPS talking about our position paper at the poster session -- and excited that the work also got an oral slot!

@YuningShen1 Interesting position paper: Machine learning conferences should establish a refutation and critiques track. By @yegordb. Have you ever struggled to reproduce prior work, only for the authors to never reply to your email? This paper proposes that conferences create a refutation

Excited to finally release the report @OpenRouterAI and I have been working on. The past year marked a decisive shift in how we build and use AI. Reasoning moved from the edges of research into the center of real-world production, driven by breakthroughs in model capability,

some cool insights from @yegordb ‘s talk on AI’s impact on developer productivity at @aiDotEngineer 👇 1. The most productive teams are pulling away from the median. The rich get richer

An LLM-generated paper is in the top 17% of ICLR submissions in terms of average reviewer score, having received two 8's. The paper has tons of BS jargon and hallucinated references. Fortunately, one reviewer actually looked at the paper and gave it a zero. 1/3

Our position paper "Machine Learning Conferences Should Establish a 'Refutations and Critiques' Track" was accepted at #NeurIPS2025 and will be appearing as part of a panel discussion! 1/3

We tested how autonomous AI agents perform on real software tasks from our recent developer productivity RCT. We found a gap between algorithmic scoring and real-world usability that may help explain why AI benchmarks feel disconnected from reality.

This phenomenon highlights that there are individuals at every company who work multiple jobs just like Soham did. I want to highlight that the above estimate uses certain assumptions that can sway the numbers 30%+ in either direction. I will be writing a short paper with a

We found that people working multiple jobs performed significantly worse than their peers: 0.62x the global median, on average.

Why might our initial findings underestimate the true rate? In our initial sample of 128.5K developers, we identified only 63 holding multiple jobs (0.05%). However, this significantly underestimates reality, as our dataset includes only 1.20% (957 out of 80,000+) of global

How did we determine if someone was overemployed? We matched individuals across companies using employer-provided data, including names, job titles, locations, GitHub IDs, emails, employment status (full-time vs. contractor), and additional encrypted identifiers.

How did we calculate this? We tracked engineers actively committing code to multiple companies' repositories across multiple months. We excluded contractors, part-timers, and people switching jobs. We then extrapolated these findings to estimate global numbers.

🧵How many software engineers secretly work 2+ jobs? My research group at Stanford has access to private code repos covering 100K+ engineers at ~1,000 companies: ~0.5% of the world’s developers. Our data shows that 4.1% of software engineers are working 2+ jobs.

New position paper! Machine Learning Conferences Should Establish a “Refutations and Critiques” Track Joint w/ @sanmikoyejo @JoshuaK92829 @yegordb @bremen79 @koustuvsinha @in4dmatics @JesseDodge @suchenzang @BrandoHablando @MGerstgrasser @is_h_a @ObbadElyas 1/6

Here is a related post, where we found that almost 10% of all software engineers are "Ghost Engineers" - doing no work. https://t.co/xELmJPZkfb

My research group at Stanford has access to private code repos from 100K+ engineers at almost 1,000 companies, ~0.5% of the world’s developers. Within this "small" sample, we routinely find engineers working 2+ jobs. I estimate that easily >5% of all engineers are working 2+

Third #ICML2025 paper! What effect will web-scale synthetic data have on future deep generative models? Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating World 🔄 @JoshuaK92829 @ApratimDey2 @MGerstgrasser @rm_rafailov @sanmikoyejo 1/7

How can we close the generation-verification gap when LLMs produce correct answers but fail to select them? 🧵 Introducing Weaver: a framework that combines multiple weak verifiers (reward models + LM judges) to achieve o3-mini-level accuracy with much cheaper non-reasoning

🚨New preprint 🚨 Turning Down the Heat: A Critical Analysis of Min-p Sampling in Language Models We examine min-p sampling (ICLR 2025 oral) & find significant problems in all 4 lines of evidence: human eval, NLP evals, LLM-as-judge evals, community adoption claims 1/8

Excited to be speaking in the AI Architects track of @aiDotEngineer in June!