Ian Webster
@iwebst
Followers
2K
Following
1K
Media
74
Statuses
389
building @Promptfoo (LLM security) + "curator of the world's largest digital dinosaur database"
CA
Joined December 2012
RT @simonw: Exploring Promptfoo via Dave Guarino’s SNAP evals.
simonwillison.net
I used part three (here’s parts one and two) of Dave Guarino’s series on evaluating how well LLMs can answer questions about SNAP (aka food stamps) as an excuse to …
0
5
0
RT @DynamicWebPaige: ❤️ Love seeing @promptfoo + @googleaistudio Gemini 2.5 experiments in the wild!. Check out this example benchmark that….
0
2
0
Anthropic has been quietly publishing top notch content on LLM fundamentals. Lots of great examples of using Promptfoo for evals in this new course!.
Our latest course on LLM prompt evaluations is out. Evals ensure your prompts are production-ready as you're able to quickly catch edge cases and zero in on exactly where your prompts need work. Let's walk through what the course covers:
3
1
7
RT @underyx: I wrote a deep dive blogpost about how @promptfoo helps us eval our AI features!
semgrep.dev
client.chat.completions.create
0
1
0
Doing online support can be a drag, but sometimes you get gems like this
1
0
3
RT @derrickharris: Democratizing Generative AI Red Teams < Really good discussion with @iwebst (of @promptfoo ) and….
a16z.com
PromptFoo creator Ian Webster discusses the importance of red-teaming for AI safety and security, and of bringing those capabilities to more organizations.
0
1
0
Had a great chat with @AnjneyMidha on the finer points of AI safety and security.
🎧 Listen to the whole discussion with @iwebst and a16z's @AnjneyMidha on the AI + a16z podcast here, or wherever you get your podcasts:
0
2
9
@promptfoo We're honored to have the support of @a16z and many industry leaders who share our vision for open-source, application-focused AI security. Thanks to @AnjneyMidha @zanelackey @tobi @fkerrest @adamely @svishnevskiy and many other excellent people. Read more about our vision for.
1
2
13
@promptfoo The beauty of open source is that it’s for everyone. The big AI labs have dedicated “red teams” - adversarial testers that find holes in your app. Now you do too!. LLM security is too important and too ubiquitous of a problem for this to not be in the hands of every developer.
1
1
8
@Promptfoo is the first open-source tool that generates synthetic adversarial attacks customized to your LLM app. It focuses on failures that developers and businesses actually care about, like data leaks and tool misuse. Over 25,000 developers at companies like Shopify,
1
0
10
AI is insecure by default. The open secret is that most AI apps launch with serious flaws. It takes MORE time to secure these apps than it does to build them. I learned this firsthand from shipping LLM agents to 200M users at Discord and supporting LLM evals for hundreds of
6
11
60
We all knew image generators can be jailbroken to generate violent and graphic content. It turns out the jailbreak process can be fully automated :o. Write-up here (open-source) with examples from OpenAI's Dall-E
0
0
3
Just added automated LLM red teaming/vulnerability scanning to promptfoo!. Uncovers app-specific jailbreaks and failure modes. Completely open source
0
2
9
promptfoo's 0.62.0 changelog is very crowded for just 1 week of work. Shoutout to @chrismaltais (@Shopify), @jumbld (@PortkeyAI), @pelikhan (@Microsoft), @dlssrt, @dangelosaurus and many others not on Twitter. Open source makes the world go round 💪
0
2
10
RT @tobi: For all our LLM (and many ML) projects at Shopify we standardized on for writing evals. That has caused a….
promptfoo.dev
Eliminate risk with AI red-teaming and evals used by 100,000+ developers. Find and fix vulnerabilities, maximize output quality, catch regressions.
0
25
0
The Eta Aquariids peak this weekend, when the Earth passes through a dust cloud left by Halley's Comet.
0
1
5
promptfoo has passed 250,000 evals + thousands of users from companies like Microsoft, Salesforce, Intel. Open source, developer-first, organic grass-fed LLM evals (choose the best prompt and model). If you are serious about deploying LLMs, check it out!
promptfoo.dev
Eliminate risk with AI red-teaming and evals used by 100,000+ developers. Find and fix vulnerabilities, maximize output quality, catch regressions.
1
1
10
Finally got around to publishing the largest per-parcel California property tax dataset - on @kaggle. These are the numbers behind the CA property tax map, which still gets a surprising amount of traffic and data requests
kaggle.com
Scraped data for 22 counties, 8.5M parcels, 86% of CA's population
0
0
3