With two extra lines of code, you can review conversations between your OpenAI Agents, users and tools. Looks like any other thread on Teams, Slack or WhatsApp. Oh, and it's free!.

I had a great time in the AI Engineer summit in NYC last week. One highlight was this very small, scary and entertaining bit of code. Basically, these ~20 lines let an AI agent execute any local code simply by thinking it.

I was just interviewed by Lawrence Cohen, CEO of a community of AI startups. We talked about the dark perils and sublime promise of AI, and US politics. Check it out:.

Super excited to announce the new RagMetrics API! 🚀 . A/B test your entire LLM pipeilne, retrieval through generation. No other product on the market supports this out-of-the-box AFAWCT.

We just shipped two highly-requested features: Criteria library, and user-defined criteria. You can use them to improve your AI bot. Here's how:

The antidote to snake oil was the clinical trial. The antidote to GenAI snake oil is a robust eval.

You can reach us at

How would you like to A/B test your prompts?. You can also A/B test foundation models, vector databases and anything else in your pipeline. If you'd like to try this out on your LLM application, drop me a line.

Three free trials slots taken, two left. If interested, please DM me ASAP. Also, here's another report type:

We are offering a free 1-month trial, valued at $500, to five of you. This trial will include white-glove support from our team. If interested, please reach out at

After seven months of late nights, we are thrilled to announce that RagMetrics is now ready to try!. RagMetrics helps LLM application builders like you prove ROI by setting up scalable evaluations. You can use it to win new customers, raise capital and make data-driven decisions.

How can you improve your AI application over time?. Managing feedback for an AI-powered application is tricky, just like it is for us humans. You need to settle an epistemological question, “where does knowledge come from?”

I recommend starting with the bottom left: Low visibility, low autonomy. These tend to be easier to execute and carry lower risk. Over time, you will find your use cases move up and to the right, and create more value.

Autonomy refers to how much your AI can do on its own. It ranges from “human in the loop” all the way to “proactive.”.

Visibility refers to how many people can see your AI’s output (horizontal axis). Internal projects are the least visible. New products and eventually new businesses are the most visible.

How should you pick your first AI use case?. Start simple. Imagine each of your candidate AI use cases as a point on a scatter plot with two axes: Visibility and Autonomy.

Hugging Face quietly unveiled a new LLM leaderboard a few days ago. Here are the highlights:.

Excited to announce has been admitted into the (YC W23) community of founders. Looking forward to meeting other dreamers and change makers.

You’ve developed an LLM app. It’s great on your demo script. But you get the sinking feeling it’s going to get rolled as soon as users hit it, kind of like Google’s AI Overviews. What do you do?.