Here's an animation of a @PyTorch DataLoader. It turns your dataset into a shuffled, batched tensors iterator. (This is my first animation using @manim_community, the community fork of @3blue1brown's manim). Here's a little summary of the different parts for those curious:.1/5

I included little inspirational snippets of wisdom for each question in the personalized eval roadmap builder here.

Who does evals (eng vs product vs domain experts), how often they do it, and how they do it varies wildly based on team size, personas, task complexity, and risk tolerance. There's no way simple off-the-shelf evals would work for @bytefuse_ai

RT @corbtt: At OpenPipe we built an entire SFT platform before pivoting to RL. It's theoretically possible to get similar results with eit….

This was my favourite talk I went to at the AI worlds fair. It makes a good case that teams sophisticated enough to have good evals can leverage open models to make custom, fine-tuned agents that are more reliable at their tasks using RL.

RT @willccbb: my full talk from AIE world’s fair is out now :).

RT @weights_biases: Unsure where to get started with AI Evals for your business? . Scott the PM for our W&B Weave product, he's talked to m….

RT @sh_reya: Big fan of Scott’s eval guide. I like that it’s highly interactive (“choose your own adventure”), and that it distills a lot o….

Most AI teams optimize eval metrics without knowing the business impact mapping. From @chipro's AI Engineering: if 80% factual consistency → 30% ticket automation, 90% → 50%, you can calculate ROI on improvements and set deployment thresholds. This is how you know when you're.

RT @sh_reya: @_ScottCondron This is so cool!!.

RT @AtharvaIngle7: this is really cool - like how it builds a personalized evaluation roadmap based on your specific situation.

RT @chipro: @_ScottCondron @eugeneyan This is cool!.

How I built this:.- @sh_reya's DocETL to help find relevant quotes / tips from my favourite eval guides / chapters / case studies across different key dimensions (defining eval requirements, dataset building, scoring, etc.) and prompt iteration.- Claude Code to synthesize the.

I made a choose-your-own-adventure for AI evaluation strategy. Your answers build a personalized roadmap based on task complexity, cost of failure, and your current evaluation. It also includes my favourite selection of tips from industry experts like @eugeneyan, @chipro and

RT @corbtt: Our customers that are using RL to train agents on their specific domain to build reliable agents are *extremely* happy fyi.

RT @capetorch: My multi-turn GRPO runs keep crashing as the vLLM server can't keep up (long traces and a lot of them when doing multiturn):….

Right-click highlighted images > Extract text to one file using Shortcuts. I tried to (vibe) code a tool to extract text from a bunch of images but it got annoyingly complicated with python OCR libs. When opening them in Preview, you can just highlight all of the text so I

The world needs more “playgrounds” for specific AI workflows/apps/personas. What makes it “specific” compared to ChatGPT etc. ?.That’s where the magic is, they’re built for a target persona, have examples of complicated workflows, and help that target persona build with AI in a.

RT @zmkzmkz: just finished the pretraining of our 7B baseline. this is the first time I've pretrained a model of this scale, just a measly,….