Sasha Aickin: "Speedy, confident" - New York Times

We are now SOC2 Type 2 compliant. Yay! But also: if you're a startup going for compliance, it's a messy process that's hard to figure out, and I wrote up all the good, bad, and ugly stuff I wish I'd known before I started.

We're officially SOC2 Type 2 compliant at Libretto! 🎉 But forget the usual corporate speak—here's an honest look at the weird, messy reality of SOC2 compliance at a startup. Check out what we learned the hard way: #StartupLife #SOC2 #RealTalk https://t.co/XeSIrqOzSp

GPT-4.5 looks really interesting, but this pricing is... whoa.

This was fun. We found that GPT-4o just started giving different answers on Monday, without any change in our code. Learn more about model drift and why it's an issue with LLM development here:

Remember, absolutely none of the news today matters until results start coming in. None of it. Don’t read tea leaves from turnout. Absolutely ignore leaked exit polls. We are in the eye of the storm for political news, so go touch grass. There’s plenty of news to come tonight.

Reminder that early exit polls are nearly useless in terms of predictive value. They are neither consistently right nor wrong. Don't freak out if they seem bad; don't celebrate if they seem good.

Like, Terraform files to solve AWS config feels like a reallllllly deterministic task. I like AI, but why are we injecting AI here?

In a certain popular SaaS tool that helps you with security configuration, and learning it has a new AI feature that will generate Terraform files to help remediate problems. But the files seem to randomly conflict with each other; I'm not sure why AI is being used here at all.

Oh, and also: it’s just one poll. A good poll, but not magic and should be incorporated with other data and signals.

Pre-registering my extremely cold Selzer take: T+5 or less is good news T+10 or more is bad news Anything in between is kinda mushy and shouldn't move many priors.

Just learned that (at least in NYC) if you buy more natural gas in a month the price per unit goes down, but if you buy more electricity the price per unit is basically constant. This feels like a climate policy failure, no?

My grand unifying theory about AI use cases these days ends up being that on many problems AI makes the bottom third of performers better and the top third of performers worse. Turns everything into mushy, eh-I-guess-it's-good-enough quality level.

Shorter Anthropic: "We built something that can automate your computer, but you probably shouldn't allow it to do so. Oh also, if someone puts a jailbreak in a website, good luck lol"

This is a helluva disclaimer. I really like Anthropic and Claude, but I feel like we should start asking whether they are still a safety-first AI lab.

The Assistants API is in such an odd space at OpenAI. It's the easiest way to get a RAG prototype up and running, but folks I talk to find it very frustrating, and it doesn't have access to models like o1. It feels a little like OpenAI isn't committed to it.

The discussion with the sales person was absolutely bizarre. He kept trying to ask what price I would pay, and I was like "I expected it to maybe be a 1K add-on". Me: "Do you charge big companies 167% of their base price for this feature?". Him: "No, we charge them $10K".

Enterprise SaaS pricing is a wild ride. Signed up for an expensive service and qualified for their ultra-small company pricing, which was $6k/year (big companies pay 10s of K). Then wanted to add a relatively small bolt-on feature, which I was quoted at $10k/year.

It's really unfortunate that you can't see any representation of o1's reasoning in the API. It means that if you ask a question in the API and get the wrong answer, you don't have much of an idea of how to change your prompt to fix the issue.

Credit for getting the right answer, though!

LLMs remain deeply confusing things.