We're officially SOC2 Type 2 compliant at Libretto! 🎉 But forget the usual corporate speak—here's an honest look at the weird, messy reality of SOC2 compliance at a startup. Check out what we learned the hard way: #StartupLife #SOC2 #RealTalk https://t.co/XeSIrqOzSp

GPT-4.5 looks really interesting, but this pricing is... whoa.

7/7 I wrote up the whole detective story, including: * How we caught it (with receipts!) * Why this is terrifying for LLM-dependent products * What we can do about it Read more:

6/ It's like your reliable coworker suddenly deciding they're only going to do 50% of their job, but only for certain tasks, and not telling anyone about it 🫠

5/ The scary part? This only affected ONE of our test prompts. The other nine we were monitoring were perfectly fine. Which means if you aren't specifically monitoring your exact LLM prompts, you'd have no idea the model silently changed underneath you.

4/ We had only seen this behavior a handful of times before. Now it was happening constantly. 📊

3/ On February 17th, one of our test prompts suddenly started getting completely different responses from GPT-4o. Not the usual small variations – the model went from giving lists of answers to just one answer.

2/ At Libretto, we've been worried about model drift - where hosted LLMs silently change without warning. That's why we built LLM drift detection into our product.

1/ Well, this was kind of wild: we caught GPT-4o changing underneath us.

8/ Ready to level up your LLM-based applications? Dive into the full blog post here 👉

7/ What's next? Stay tuned for upcoming posts where we'll cover: * Integrating Libretto directly into your app's codebase for real-time monitoring. * Deep dives into customizing and calibrating Evals for nuanced assessments.

6/ In this post, we show how we set up our project in Libretto, generate test cases for our prompts, & improve consistency with function calling. This gave us: * Enhanced control over LLM outputs. * Efficient testing and debugging workflows. * Confidence in deploying our app 🎉

5/ Sure, tweaking prompts can help. But more prompts = more unpredictability. How do we ensure our LLM outputs are reliable and tailored? Enter Libretto 🎟️— our tool for better understanding and controlling LLM behavior.

4/ We faced several hiccups: * Profiles about songs instead of people 🎶🤦‍♂️ * Generating profiles for abstract concepts like "Prompt Engineering" 💻😂 * Inadvertently creating profiles for controversial figures 🚫 * Generic and repetitive language lacking personality 😴

3/ Introducing WikiDate 🗂️❤️— our fun app that generates dating profiles for any person or fictional character on Wikipedia. Sounds simple, right? Read Wikipedia content, prompt an LLM, and voila! But reality had other plans...

2/ Ever built a cool LLM demo only to struggle when scaling it for real-world use? We've been there! Transforming prototype apps into production-ready services can be challenging. That's where Libretto comes in.

1/ Excited to announce a new blog series: "Building an LLM-based App with Libretto"! In Part 1, we dive into turning simple LLM demos into robust, production-ready applications. Here's a quick rundown 🧵👇 https://t.co/caGsubjcvG

9/ We're eager to see how prompt engineers leverage this tool to elevate their work. If you're interested in more precise, scalable LLM evaluations, join our Libretto Beta and experience the difference firsthand! https://t.co/n3JRd1YUxt

8/ The result? A more efficient process that not only speeds up evaluation but also enhances the relevance & precision of feedback, allowing you to focus on refining the most promising prompt variations. Go from manually checking 5 or 6 prompt inputs to spot-checking 100 inputs.

7/ Continuous improvement is key. Our interface allows you to review, adjust, or add new criteria based on real-world testing results. Grade some outputs, and our system calibrates to align closely with human judgment.