@alexshander03 tabling his AI evals doctrine in DC this week, flanked by @JudgmentLabs' varsity cheer team - ie @carloagostinel2 & myself. despite best efforts we never made it past the fence

Readers responded with both surprise and agreement last week when I wrote that the single biggest predictor of how rapidly a team makes progress building an AI agent lay in their ability to drive a disciplined process for evals (measuring the system’s performance) and error

@alexshander03 and the entire @JudgmentLabs team have been quietly pushing the limits these past few months. Thrilled for what’s ahead for this exceptional group—big things are coming. Read more below: https://t.co/7jsqtjBfUh

@JudgmentLabs Read here:

At @JudgmentLabs we've had the opportunity to work with countless AI agent teams building fantastic products. Measuring and understanding agent behavior has become a bottleneck to agent improvement and everyone knows it. However, few get this process right and most teams fall

chat is this real?

https://t.co/hAG8NGUNKA

Brendan is right that methods for evaluations will push the bounds of what agents can learn. I think there's something to be said here about how we know what to evaluate and how we go about doing that. Sometimes we can rely on human experts to craft criteria, but this can break

There is insane demand for people who can understand and explain technology in a compelling way.

6 months, 25 million revenue agents & 3 trillion tokens later... Rox is now globally available 🌎 Just as coding agents 10x’d engineering, revenue agents 10x customer work. With Rox, humans are evolving to orchestrators while agents manage the end-to-end customer lifecycle.

This is insane - and foreshadows a future that will come fast. Cursor just handed us the first production-ready demonstration of how strong online RL can be!! The secret here to generalizing is figuring out how different apps, each with their own interface, can collect

we’re approaching the end of 2025 and there’s still no plug-n-play RL lib in the interrim: - i built a shitty version of this (llamagym) - RL started working (o1) - oss found out how it worked (r1) - “RL env” became the new buzzword - oss RL envs unified around `verifiers`

“Evals” are becoming an ever-growing umbrella of terminology that describes any measure of quality across an AI app. As a result, conversations and discourse are getting lost in definitions and semantics... here's an example. Frontier labs use evals (reward models, human

New blog post about asymmetry of verification and "verifier's law": https://t.co/bvS8HrX1jP Asymmetry of verification–the idea that some tasks are much easier to verify than to solve–is becoming an important idea as we have RL that finally works generally. Great examples of

reward is eval, eval is reward, reward is enough