Agents are significantly more powerful than standalone LLM calls. But, debugging them is a nightmare. You can trace their reasoning and tool use, but traces get huge and are impossible to parse. To solve this, we spent the lasts several months building Gentrace for Agents,

LLM-as-a-judge evals often fail because they ask the same question twice. Instead, give the model an unfair advantage: extra context, constraints, or comparisons that make grading easier than generation. Here's a guide breaking down our approach: https://t.co/0Hf3VWFcbM

Your dataset will never be perfect! But you need one to get started with evals. Instead, the best AI teams don’t chase a "golden" dataset. They start small with 5-10 examples, capture real production use cases, and iterate continuously. Here’s a practical guide for building

Multiverse is using AI to improve how students learn on the job. Their AI team uses LLMs for delivering realtime feedback to students, requiring high quality and reliability. Before Gentrace, evals were managed in spreadsheets, creating bottlenecks. Now they: - Use LLM

What a night! Packed house and some of the sharpest minds in AI from Webflow, Asana, and 11x sharing how they’re building with agents today. Big thanks to Ampersand for co-hosting and our speakers our speakers @bryantchou @rodrigodavies @prabhavjain @ezelby @patrickt010 for

Agents are changing everything about how we build software. Will agents replace purpose-built tools or create entirely new opportunities? Learn where agents are actually headed this Thurs 2/6 at Gentrace SF with our speakers @bryantchou @rodrigodavies @prabhavjain @ezelby

Last week, we hosted some incredible AI builders from companies like Asana, Cribl, Block, Vanta, and Pinterest to share their stories on shipping AI apps to production. What we learned is that the road from POC to production isn't a straight path: 1. Turns out the biggest

Self-hosted just got an upgrade. Now you can deploy Gentrace in your @kubernetesio cluster with: - Helm charts for quick setup - @istiomesh for secure service-to-service communication - Support for your existing data infra (Postgres, Kafka, S3) Get started with our guide:

We're excited to join the @foundersysk showcase on Feb 5th in San Francisco! Meet our cofounders, @dougsafreno and @virtuallyvivek, and learn how we're helping teams at Webflow and Quizlet test their AI apps. Apply here: https://t.co/dxlrJoK98w

How do you process 45,000 tasks/day without adding infrastructure complexity? At @gentraceai, we built a task queue with @PostgreSQL using FOR UPDATE SKIP LOCKED. It’s simple and reliable while handling retries and large payloads. Clone our implementation for your own project:

It was a big year for Gentrace in 2024! 📈 21M+ evals and traces ran, growing 2x in the last 6 months (up from 40k when we first started Oct 2023!) 🤖 19 features and improvements launched 🪲 320 bugs squashed Thank you to our customers and team for making this possible. 🫶

It’s been an exciting year for us with lots of new releases and bugs fixed. Here’s a recap of our 5 favorite things we shipped: 1. Datasets support—organize test data into separate groups within a pipeline 2. Compare—updated compare mode for easily viewing outputs and test

Been a chaotic but fun year at @GentraceAI. We hosted 10+ community events, partnering with @modal_labs, @webflow, @memdotai, @aiDotEngineer, @matrixvc and @HeadlineVC, to host dinners, tech talks, conferences, and office parties. Intensity is high, but we also have fun along

I'll die on this hill: people are ripping on LLM as a judge because they aren't doing it right. 80% are asking the LLM the same question as the prompt. Avoid being the 80% by giving your LLM as a judge an "unfair advantage" — aka some additional context or capability that makes

With the rise of LLMs, the @webflow team set an ambitious goal to use natural language to make modifications to websites. To set up evals, they chose Gentrace. With Gentrace, the Webflow team: - Evaluates multimodal outputs (like website screenshots) using human and

@webflow Learn more about our partnership with Webflow in our case study: https://t.co/dDTQUK7cf0

With the rise of LLMs, the @webflow team set an ambitious goal to use natural language to make modifications to websites. To set up evals, they chose Gentrace. With Gentrace, the Webflow team: - Evaluates multimodal outputs (like website screenshots) using human and

What if your LLM testing system could automatically optimize your prompts? That's where we're headed with Experiments, our new feature helping developers speed up last-mile tuning. Here's how it works: Unlike prompt playgrounds, Experiments provides a testing environment

Big news today: @GentraceAI raised our $8M Series A led by @MatrixVC. We’re celebrating by launching Experiments, the first collaborative testing environment for LLM product development.

Most engineers approach LLM-as-a-judge all wrong. The usual high-level metrics like hallucination or safety rarely tell you if your app actually works as intended. Let’s talk about why that’s a problem and how to fix it: