Bigger models ≠ better AI. Better data = better AI. The future belongs to enterprises that can rapidly create, customize & evaluate their data. 🎥 Snorkel co-founder Braden Hancock explains: https://t.co/RyFKKCF3Pq #AI #LLMs #DataCentricAI

Snorkel’s Daniel Xu → “Optimized Infrastructure” panel, Sept 11, 1:50–2:30 PM. TY @awscloud + Amazon SageMaker for hosting; joined by Walgreens Boots Alliance, @datadoghq, Lablup Inc. #AIInfraSummit

We partner with our network of vetted experts, using detailed evaluation rubrics to generate high-quality datasets that improve AI applications. Follow us @SnorkelAI for updates, & if you need expert-verified datasets for your evaluation needs, talk to us!

The techniques and categories combine! A complete rubric typically includes some of all of them. Choose your rubric strategy based on domain requirements and evaluation goals🎯

METHOD: Automated (LLMs for scale / code for strict correctness), human experts (nuanced, intuitive evaluation), and reward models (emerging hybrid approaches)

LOCATION: Process-based (how models reason, catches lucky guesses) vs outcome-based (final answer quality, objective verification)

GRANULARITY: Coarse-grained (single "goodness" score / fast screening) vs fine-grained: multiple dimensions (diagnostic precision) SPECIFICITY: Dataset-level (universal criteria) vs instance-specific (tailored per prompt)

Well-crafted rubrics provide a systematic approach to improving AI applications, useful for both automated and HITL expert evaluation. Our post breaks them down in four ways:

Part 2 of our AI evaluation rubrics series is live! 🧵 The right tool for the job: an A-Z of rubrics https://t.co/iknUrWKGEB

SnorkelGraph is part of our expert-verified benchmark suite, designed to systematically test LLM reasoning across math, spatial, and graph domains. Follow us for updates, and if you need expert-verified datasets for your evaluation needs, talk to us!

As the leaderboard shows, this benchmark is far from saturation ATM: https://t.co/Qe0UiN94cf We’re looking forward to seeing continued advances in this skill, particularly in open models.

We control difficulty with: 🔹 Graph size: more nodes/edges = more elements to reason over 🔹 Operator complexity: different ops stress different reasoning skills (counting, paths, optimization)

Each question asks the LLM to compute the outcome of a natural language question (operator) over graphs encoded via node + edge lists. Example: “Find the minimum-density subgraph…” Because these questions all come with verifiable answers, LLM responses are easy to evaluate.

Operations in graph theory are fundamental to analyzing complex systems, and provide a multi-hop, long-context reasoning challenge for LLMs. That’s why we built SnorkelGraph: a benchmark for evaluating graph reasoning with verifiable ground truth. https://t.co/Qe0UiN94cf

The best RL envs look like product specs: tools, data, rubrics, simulators. That’s where Snorkel lives. Interested? Let’s chat.

Want to improve a smaller model’s reasoning skills? Our DaaS team is building expert-curated datasets for reasoning, function calling, and more. Let’s talk!

Key insights: 1️⃣Reasoning > non-reasoning models. Better living through inference-time compute! 2️⃣Smaller open-weights models struggle without fine-tuning or other post-training optimization 3️⃣SnorkelWordle is a strong signal for evaluating reasoning, especially in smaller models

Word games yield valuable insights when evaluating LLMs. We built the SnorkleWordle benchmark to test models on 100 rare English words—and the results are 🔥

Great AI isn’t about more data, it’s about the right data. Snorkel crafts expert datasets that help AI reason, and solve problems. From coding to video and multi-turn conversations, our data powers AI built for real-world challenges. See how: https://t.co/VLw1LGzlkg

Check out Snorkel researcher @chris_m_glaze’s teaser on experiments he’s been running on AI agents ⬇️⬇️