LLMs have made exciting progress on hard tasks! But they still struggle to analyze complex, unstructured documents (including today's Gemini 1.5 Pro 002). We (UC Berkeley) built 📜DocETL, an open-source, low-code system for LLM-powered data processing:

RT @ihower: I’m currently taking AI Evals for Engineers & PMs (Cohort 2) by @HamelHusain and @sh_reya. The course focuses on building an a….

RT @clstrfq: Two projects are already being formed to apply the lessons from AI Evaluations for Engineers and PMs. This is the most compreh….

RT @amrit_za: One of my favourite aspects of this course is that it suits many learning styles. If you're like me and prefer reading, the….

RT @sh_reya: Our first DocETL paper has been accepted to VLDB 2025! DocETL is a system we’ve been building at Berkeley for reliable LLM-pow….

RT @najibninaba: 7/ 🔑 MOST VALUABLE LESSON:. Stop obsessing over perfect prompts. Start by systematically analyzing WHERE and WHY they fai….

RT @_Trang: What makes evals different is that you are taking a data science approach of deeply understanding your user's conversion histor….

lesson 4 is a fun one. Hamel and I do a role playing bit. We do error analysis while applying a poorly defined rubric to grade traces (to demonstrate poor cohen’s kappa). I play the role of someone only spot checking the traces. Hamel plays the role of a better dev who.

RT @Allmarsh713: Been in build mode the last few months, working on some new projects, gearing up to ship to real users in weeks. AI agents….

RT @gojira: Loved the "AI Evals for Engineers & PMs" course by @HamelHusain and @sh_reya. It takes “look at your d….

RT @skylar_b_payne: "My biggest problem with evals? I have no idea where to start". Stop feeling overwhelmed with all the tools, models, et….

RT @jcardonnet: Last week I finished @HamelHusain and @sh_reya 's "AI Evals For Engineers & PMs" fantastic course. I expected up to date,….

RT @glorioustango: Just finished "AI Evals For Engineers & PMs" with @HamelHusain and @sh_reya 🔥 and it was a total game-changer. Highly re….

I just saw an AI-generated comment on an AI-generated summary of an AI-generated substack article on evals. The result? I want to escape this timeline.

RT @joshpitzalis: Just finished my second round of the AI Evals for Engineers & PMs course by @HamelHusain and @sh_reya — and it was worth….

No? It’s because they don’t invest in evals processes lol. AI can be extremely useful as a component in software without data retention or memory features.

RT @vishal_learner: i have been looking for an explanation/analogy like this one for task-decomposition for awhile. this totally makes sens….

I’ll be presenting the paper on Tuesday, Sep 2 at VLDB. You can read it at Follow along with the open source project at and This first paper was a super fun collaboration with Tristan, Tarak, Eugene, and my.

For me, DocETL has been more than a lesson in how to build reliable AI-powered data systems. Bigger models aren’t enough; reliability comes from structuring tasks into pipelines, and from optimizers that can invent, rewrite, and validate those pipelines for us. But the DocETL.

On the Berkeley police misconduct dataset, DocETL’s optimizer automatically discovered pipelines that doubled recall compared to a strong human-engineered baseline. Pipelines that would have taken experts weeks of tinkering emerged automatically from the search. We are pretty.

So over the year we built DocETL: a system that lets users author pipelines of LLM-powered operators, with an optimizer that rewrites those pipelines top-down. Given a library of 13 open-ended rewrite directives — guidelines for how to logically decompose a pipeline — the.