Brandur
@brandur
Followers
6,627
Following
230
Media
598
Statuses
6,059
Engineering @ Crunchy Data. Ex-Heroku, Ex-Stripe. Occasional writer. Themes: web APIs, Postgres, software resilience, and efficient human interfaces.
Joined March 2009
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
Madrid
• 1672243 Tweets
Champions
• 1311293 Tweets
Dortmund
• 745648 Tweets
WNBA
• 191706 Tweets
Clark
• 166954 Tweets
#ufc302
• 123826 Tweets
Alex Jones
• 102773 Tweets
Wilder
• 98895 Tweets
安田記念
• 96983 Tweets
Corinthians
• 59924 Tweets
Stonebwoy
• 56887 Tweets
#25thTGMA
• 49316 Tweets
Morono
• 44684 Tweets
Rangers
• 42196 Tweets
Botafogo
• 30234 Tweets
Panthers
• 25916 Tweets
ジンくん
• 22999 Tweets
Romero
• 22252 Tweets
Musetti
• 20467 Tweets
Igor
• 19905 Tweets
シャーシロ
• 17501 Tweets
Pachuca
• 16828 Tweets
Aaron Judge
• 16377 Tweets
ニジゴン
• 14844 Tweets
Seiya
• 12426 Tweets
Stanley Cup
• 11938 Tweets
#Bettel3D
• 11811 Tweets
Sandy Hook
• 11253 Tweets
PayPal evacuating SF June 3rd. That's everyone in payments gone — Block (Square), Stripe, Brex, Credit Karma, etc. I'm sure just a coincidence — no relation to Prop C.
Once again, San Franciscans get what they ostensibly want. We'll see how long this train can stay on the rail.
113
352
4K
A neat public art installation in Glen Park in SF: A lens focuses the sun to burn a line in a redwood log. It's advanced every day, creating a new line.
After a year, it's moved to a new log. When complete, the trio will be an archive of three years of sun and weather.
20
921
2K
Git tip I wish I'd discovered ten years ago: if you `git config --global diff.noprefix true` it removes the silly `a/` and `b/` prefixes so that when you double-click select one to copy, you get a usable filename instead of a mangled path.
33
706
2K
There's nothing quite as nefarious as a technical decision made via Slack: stakeholders miss it if they're not around, points are poorly stated as they're strung across 50 half-baked thoughts, context exists across pages of jumbled noise, *and* it's time consuming.
I miss email.
56
267
942
At Stripe we switched to a GUI-driven deploy process for services.
I'm a die-hard terminal person for life, but it's so much better. Especially during the duress of an incident, there's nothing like being able to click boxes to resolution instead of looking up obscure CLI flags.
24
78
827
Stripe's prime competency isn't payments, engineering, or product — it's PR, and always was. I look on in reluctant admiration at how even layoffs are executed about as masterfully as is possible.
Sorry to anyone who got hit. This sucks.
12
37
737
I am sorry for everyone who was hoodwinked into providing free content to Medium. Look what they've done to it.
This should've been predictable, and though the logos in play have changed, is still a relevant lesson to learn today ...
48
42
500
The next time someone says that a test suite is slow because it's a big project, point out that you can build Postgres from scratch on a fast machine in ~30s and run its test suite in ~15s.
Test suites are slow because their progenitors were sloppy, not because they have to be.
11
108
376
Car crashes over railing and down pedestrian staircase. Turns out, it's one of SF's infamous smash and grab teams, who promptly run off.
Meanwhile, progressives concentrate fire on the city's REAL scourge: AVs.
As often the case, SF is beyond parody.
Unreal footage: SF resident captures epic car crash at 19th St & Sanchez from multiple angles.
Source:
150
293
1K
18
24
261
Today, saw a person sit at a table in a cafe and read a book for three hours. Didn't pull out their phone or laptop or distract themselves in any other way.
Practically a superhuman feat in the contemporary age. I bet < 1% of humanity still has the capacity to do this.
10
17
223
ACID transactions are the most important primitive for building large, robust systems there is. Bake them into your system from day one. Never, ever use MongoDB.
I like when I'm not the only saying this stuff. Here's FoundationDB's transaction manifesto:
11
59
212
Myself and
@blakegentry
did what you should never do, and implemented a new open source job queue.
Written in our favorite language (Go) on our favorite database (Postgres), it's been a lot of fun hacking on this on and off over the last few months.
14
35
213
Continuing my streak of totally-popular-with-everyone database opinions, here's one on why soft deletion (`deleted_at` columns) is a net negative, and what to do instead.
24
45
204
Getting a patch reviewed on the Postgres hackers mailing list floors me every time — just at the next level in terms of attention to detail, effort invested, and thoughtfulness. I'm 10+ years into working in software professionally and I've never seen anything else like it.
6
22
186
Advice for the 99%: don’t use the tech du jour. Do use an RDMS with ACID guarantees. Probably just use Postgres.
10
117
182
Recently I had the hard realization that I've spent more years as a Mongo user than as a Postgres one, so I'm changing that this month. Excited to be joining the engineering team at
@crunchydata
.
A few more words on that:
20
7
177
Honestly, even as bad as all this JavaScript stuff has gotten, I can barely believe this. Bigger than my first computer's entire hard drive.
10
3
167
Meanwhile, execs involved will bathe in glory, receive equity awards, become Twitter-famous engineering management influencers, and eventually make a stepwise move elsewhere to join the C-suite, all because a product was created in their general vicinity.
My total comp for creating GitHub Copilot, from inception to GA:
+20k bonus and a title bump.
455
819
10K
2
17
153
Shopify's new HTTP server is Unicorn-esque, but reforks aged Ruby worker processes generationally to yield better copy-on-write memory sharing.
Very smart, and kind of obvious in retrospect. Wish I'd thought of it.
3
17
150
A few years ago, every day was bashing through process, meetings, and endless, sprawling Slack threads trying in vain to open an hour at the end of the day to code for a bit.
Now, <1h Slack + meetings/day, shipping product the rest of it.
Night and day.
Culture is a choice.
4
13
138
I don't always compliment Go, but we just looked into enabling HTTP/2 across our seven supported languages. In Go you get it automatically for both clients and servers using just `net/http`.
Excellent end user experience — 10x better than anything else and 100x better than most.
3
28
127
Even while REST is perfectly adequate as a modest baseline, I put together a few words on why I think GraphQL has the promise to lead to a generation of APIs that are more powerful, more discoverable, and most importantly, more adaptable.
4
37
128
My money's on Go for the most productive language — a winning combo of speed, brevity, correctness, and strict-but-not-too-strict types.
This morning I wondered how difficult it would be to implement graceful restart with exec on usr2. Went from idea to refined impl in < 30 min.
4
13
117
One of my favourite natural effects in San Francisco: morning sunlight filtering through the mist and eucalyptus of Mount Sutro.
2
3
119
I spent some time building a toy web service in Rust and wrote about some of the best parts. This piece explores a type-safe actor framework, Postgres-friendly concurrency model, and middleware with compiler-enforced modularity.
5
26
114
Mongo satirizes Mongo better than its critics ever could!
There are two paths in data architecture: use a relational database, or build a pale shadow of one in your app layer as you desperately try to shore up data integrity on your non-relational store.
2
20
105
A favorite operational trick from Stripe —
Metrics and dashboards are great, but despite its inefficiencies, logging will always have a place in operations — it gets you insight in tight spots that you'd never have otherwise.
3
16
101
I ran my first deploy to Amazon's Fargate — it worked, but I'd been hoping for something ... simpler.
Despite Docker, containers, and Kubernetes, the world is still looking for a deployment and operational experience on par with Heroku in mid-2011. There's lots of room to grow.
10
16
98
Creating columns with DEFAULT values will now be fast in the upcoming Postgres 11 release. It seems like a small feature, but it's a *huge* improvement operationally.
I put together some background on why it's important here —
2
44
97
Gods, now that I'm off JIRA, GitHub Enterprise, managed endpoint with < 2 hours battery life, mandatory multi-layer code reviews, 15 minute CI turnarounds, and 30 second test start up, I'm running out of things to complain about.
May need to look into picking up Java as a hobby.
4
2
95
Another month in, I'm finally starting to feel semi-literate in Rust. It's the hardest programming language I've ever learnt, but it pays off.
7
19
93
Someday, I would love to see the source code for S3.
That project must be the greatest pile of nightmare code ever written, and simultaneously one of the most impressive engineering feats in history.
3
4
93
Go on Lambda is *really* compelling. It's fast, highly concurrent, is batteries included, but its most interesting features for serverless might be API stability and easy, robust deployments. A few more words on the subject:
1
33
92
One day we realized that both our major production systems only had one stateful dependency, and decided to see how long we could keep it that way.
Not strongly prescribed, but food for thought anyway — on single dependency (Postgres only) stacks:
6
12
90
The unified log is an inspiring idea in distributed architecture. Redis streams are the perfect foundation for one in most cases — user-friendly, ubiquitous, and cheap.
A little commentary and a unified log demo built on the prerelease branch:
4
40
87
The best Bash advice.
(Bash (or sh, zsh, etc.) is objective garbage as a programming language, but it’s garbage that’s ubiquitous, and that’s valuable. Just don’t let a Bash program grow beyond trivial size or you’ll pay for that low bar to entry later — in blood and tears.)
Google has a styleguide for bash.
- If you find you need to use arrays for anything more than assignment of ${PIPESTATUS}, you should use Python.
- If you are writing a script that is more than 100 lines long, you should probably be writing it in Python.
16
156
455
6
24
89
@nonmayorpete
Prop C itself will always show a positive number (although a much lower one than expected now). The real figure you'd want to try and calculate is _total_ tax revenue loss — companies that leave not only don't pay Prop C, but pay no SF tax at all. Unfortunately hard to track.
2
1
82
Time to upgrade production to Go 1.17 — 15 minutes.
For comparison, time to upgrade Ruby at old major payments processing job — closer to 15 weeks.
There is something to this static binary / shallow stack thing.
8
10
84
Great to see Google writing in support of strongly consistent databases. Building on transaction-less databases (Mongo, etc.) is a great way to spend eternity desperately shoring up brittle code against a trillion edge cases caused by concurrent access.
3
19
85
A year into professional Go dev, my main day-to-day complaint: too. much. code.
Too many `if err != nil`, too many stanzas that should've been one-liner map/reduce, too much boilerplate. It's a problem for dev speed, but mainly refactoring because there's just so much to change.
3
6
83
My old man programmer take of the day: any metaprogramming DSL can be replaced with an API involving normal function calls and closures which might be maybe 10% less fluid to read, but is 10x more obvious and 10x as LSP friendly.
6
6
82
Been running our API project for 2+ years, and this will sound like hyperbole, but comfortable saying we have close to zero technical debt.
The key has been when noticing a bad pattern/practice, fix it _immediately_. Don't let anything snowball to where it'd be a project to fix.
10
5
80
Have to relearn this lesson the hard way every year: never write sh/bash code, NEVER compromise.
Even if you think it's just going to be two lines. Just stop, and start over in Ruby. The shell code will balloon and contain non-obvious bugs every time.
15
6
82
Postgres' `text` type, which lets you interleave unbounded data in with any database row, is a marvelous feat of technical accomplishment.
Published a piece on why you might want to consider using `varchar` anyway, inspired by lessons learned at Stripe.
8
20
80
Happy to report now supports HTTP/2 after dropping support for old crypto unlocked an upgrade path for us.
Go integrations on 1.6+ will start using HTTP/2 automatically. Upgrade stripe-php to get it. Support in other languages is more complex, but coming.
4
15
76
I published “How Postgres makes transactions atomic” which closely examines snapshots, xlog, clog, and the heap.
2
37
76
Go package of the day, which verifies no goroutines are still running after a test case:
Put it in sooner rather than later because it just gets harder. It's _really_ easy for tests to accidentally leave goroutines running that no one would ever notice.
1
11
75
On the incompleteness of C:
Postgres is a database, but also a runtime (fully custom memory management infrastructure), a standard library (custom string builder, linked list, hash map, sorting, ...), and even language (custom longjmp-based try/catch/raise mechanisms).
4
9
71
Wrote a piece on ditching Google Analytics to run analytics over logs, like it’s 1999 (except now with hosted Presto, etc. bootable in seconds from the cloud ;).
I also estimate how inaccurate hosted analytics are due to adblocked scripts. Answer: very.
7
15
71
This account from an ex-Oracle engineer on what it's like to work on Oracle is tooth-gnashingly painful.
Fast unit tests and a suite that runs in minutes locally (or less!) is productivity manifest. Making CI the dev feedback loop embraces the opposite.
2
18
68
We put our first partitioned table into prod, so did a small write up on what it's like to use partitioning in Postgres nowadays.
Progress on this feature over 5 years is downright incredible. Managing a partitioned table is 90% as easy as a normal one.
2
19
66
It's crazy that Okta is considered a top tier identity provider, yet can't quite get redirect-back-after-login working, and logs you out every four hours — basic facilities that we had figured out in our PHP apps in the halcyon days of the early 2000s.
5
2
65
I wrote a contrarian piece arguing for user interfaces that are more like terminals, and less like the web.
14
37
64
The wisest thing I found on Twitter yesterday.
If you're on a computer scrolling, nothing useful is happening. Stop using computer and do something else.
Made this post-it note to remind myself.
typing is the secret to using the computer. if you're not typing, you should be clicking on stuff. if you're scrolling, then it's already over... you're not doing shit
30
541
3K
8
3
61
After a lengthy battle, my SEO’s been usurped by an IKEA hangar rail for the garage. (Looks pretty nice actually, might get one.)
Do I have any chance of taking it back, or is this the end.
7
1
62
Discovered that it's possible to hike from the hills of Richmond to the redwoods of Oakland, and thanks to a buried highway, only cross ~3 minor roads.
This is now the best long walk that I know of in the Bay Area.
3
2
63
We encode IDs as an in-house format called EID, which is 128 bits like UUID, but easy to select. We use ULID's time-based algo so they're ordered.
Few downsides except that the SQL enc/dec funcs are the most brutal computer code I've ever written lol.
1
4
62
Maybe best small-ish Postgres improvement in years, 13 has a “force” option for dropping a database even where clients are connected:
$ dropdb --force
# DROP DATABASE my_db WITH ( FORCE );
That was probably PG's
#1
development annoyance. I can feel all that saved time already.
1
9
62
Dropbox on scaling transactions between shards using a two-phase commit. Like Google, they've concluded correctly that the answer to scaling consistency isn't to throw it out, but rather to build a substrate to make it possible, even if it's difficult.
0
15
60
A few words on GraphQL and how to build the next frontier of compelling web APIs.
5
21
62
Space Black looks cool as hell and M3 perf impressive, but literally not once since my first M1 three years have I thought to myself, "damn, I wish this computer was faster".
Every M computer is still perfect. New purchases are hard to justify.
12
2
60
In 2018, Make's error if you accidentally prefixed a command with four spaces instead of a tab is `*** missing separator. Stop.`, presumably because user hostility is a core design tenet.
Stop romanticizing Make/old Unix commands/shell scripts, and help make them die.
10
9
61
Inspired move by Patagonia — no more customized products with corporate logos because products with logos are more likely to end up in landfills.
Companies should embrace this and keep going Patagonia for swag. Brand T-shirts if you have to.
1
9
60
Wrote a piece on “canonical log lines”, a cool idea for operational visibility at Stripe.
4
21
59
@goodinvesting1
IMO: Benioff's a PR guy, saw an opportunity to be the "good billionaire" who supported it which Twitter would say nice things about, and went for it.
Ironically though, SFDC's also dropped 325k+ sqft in SF since then. Ostensibly because of WFH, but he'll also be paying less tax.
2
0
53
After trying a lot of database options in the Go world, we ended up moving everything over to sqlc — lets us use the excellent pgx driver everywhere, while also getting us high-level constructs with minimal boilerplate.
Longer writeup:
5
7
58
Jonathan Blow makes the case that software is in decline, and been free riding on hardware improvements for a long time. “We don't expect it to work anymore.”
So many great points — I'm 100% convinced. Most important talk of the year. Maybe the decade.
4
3
55
As a user, I can't overstate how much of a game changer Sorbet by
@darkdimius
and co. is — you get totally, wonderfully, can't-ever-go-back captivated by it after just a few hours of use. The extra type annotations cost a little, and return *a lot*.
4
8
57
It looks like there's a good chance that a native form of async/await will be coming to Rust's core (as opposed to existing in packaged layers of macro spaghetti above it). This is really, *really* good news.
4
9
52
Having lived in San Francisco’s SOMA district for five years before finally getting out, I’ll never again underestimate the value of silence.
It’s the most minute luxury in life. When you have access to it, you think nothing of it. When you don’t, you miss it dearly.
2
3
53
One of my proudest automations — docs/API structure read from Go code → OpenAPI → Hugo-friendly Markdown → HTML, all pushed automatically via GitHub Actions.
API ref is complete, but docs are human-written so they don't read like cold machine output.
Because we have
@brandur
you'd expect a well designed API and of course that all API reference doc be generated automatically from OpenAPI, and well you'd be right -
1
0
12
8
2
52
Renaming a table in a hot database without breaking anyone and with no downtime is harder than it sounds, but possible using a combination of Postgres' updatable views and transaction DDL. Good for schema hygiene fanatics like me.
Full instructions:
2
11
51
A ChatGPT-generated article hit the front page of HN this morning. Some notes.
AI is scary as hell, just not in the way that we thought it would be. Very soon we're going to be drowning in the empty calories of AI-recycled content.
4
4
51
Surprising how split popular sentiment is between siding with Apple vs Epic.
Imagine if the original PCs disallowed user-installable software, and IBM/Apple demanded 30%. Modern computing wouldn’t exist.
Mobile platforms are the future. Android/iOS are a duopoly. 30% is nuts.
4
5
50
Implementing idempotency keys and cultivating passive safety in APIs built on Postgres.
6
18
49
Assumed the 11’s 0.5x lens to be a gimmick, but 24 hours in, finding it’s the wide angle I always wanted, but didn’t know I did.
Apple knocked it out of the park. In including it, but also the new Camera app design, which subtly shows what 0.5x might look like from the 1x view.
2
2
49
I have never been so afraid for GitHub in my life.
More integrations is generally good, but Jira is the exception. When software is this irredeemably bad, don't integrate with it, replace it. Remember GitHub Projects? Good idea. Now finish the other 50%.
4
9
47
One of the least good parts about Postgres continues to be syntax validation and precision of error messages.
"Somewhere in your 50,000 character query, there is an error. Maybe it is by an AND keyword?"
5
0
46
Anecdote from Amazon.
My last BigCo, ~40% of time wrestling with internal tooling is about right. (Add 10% arguing with true believers making contra-reality claims that this stack from hell was good, actually.)
Dev at scale seems to get rough reliably.
6
4
46
A common mistake from industry laymen is that running software is "shelf stable". Like a bridge, once you build it, you can leave it in place for years.
Especially for large/complex projects, the opposite is true. Without human care, most would last days, or with luck, weeks.
7
10
47
Big news for PGBouncer!
Release notes say 15 to 250% perf improvement. I'd guess closer to 15% for most real world apps, but still, if you're using PGBouncer + a reasonably modern ORM, that's a free 15% with little work involved.
PgBouncer 1.21.0 - "The one with prepared statements" - this is absolutely huge
1
36
124
1
5
45
TIL: The “Re” we use in subject lines for email replies is a latin phrase abbreviated from “in re” (“in the matter of”). Hah, I always thought it was short for “reply”.
There’s an RFC of course:
2
14
45
Even having read about them a hundred times before, I still need to refresh my memory on the different SQL JOINs once a year or so (doing application development, `INNER JOIN` isn’t just the common case — it rules supreme).
Here’s a novel take on them:
2
10
46
We're experimenting with a technique in which we keep high-throughput data out of our main DB to stave off operational issues, minimize backup size/WAL, and speed up recovery in case of failure.
We call it "ephemeral DB". Working reasonably well so far.
4
7
46
Did what I thought I'd never do again, and started draining structured logs to Postgres.
Precautions: (1) canonical lines only, (2) ephemeral database, (3) partitioned tables, and (4) efficient, batch upserts. With those in place, reasonable operational insight without Splunk.
This piece on "Fast as a Service" is pure gold from
@brandur
Talks a lot about design of Crunchy Bridge, with some numbers to back it up on our p50, p95, p99
0
5
37
2
8
45
Haven't played with generics much yet, but hopeful.
Rust's `?` operator and better chainability in general would be so huge though.
Die-hard Gophers would say that it's bad for clarity, but they were wrong about dep management, wrong about generics, and would be wrong here too.
3
0
44
Remeasured today — Go code is ~2150 test cases and runs in < 10s uncached on my Air. Most are DB-backed, so no cheating.
At $lastJob, that was the time to run a single test case, AFTER 1000s of hours sunk into optimizations.
Start lean. Keep it lean.
.
@brandur
's thoughts on some of the codebases of Crunchy Bridge. If this intrigues you we're hiring
@crunchydata
...
0
3
20
2
2
44
Tailwind: first day on it is horrible as you're inextricably married to the ref docs page to look up Every. Single. Thing.
But having written 10s of thousands of lines of CSS over the years, I think I'm a believer already. CSS is unmaintainable on a fundamental level.
6
2
42
Application-level validations are optimistic — they work as long as you're 100% bug-free, and at low concurrency.
Database contraints are the only way to move from hoping data is valid to *knowing* it is. Use them. Don't use a database without them.
1
18
42
I assumed these Sentry spans were self-satisfied technical wankery on my part, but they really helped fix a perf bug that would've been hard to find.
See the ~80 ms gap in this pic — I wasn't sure what it was, but knew it was happening between auth check and remote service call.
3
5
43
A short story about moving a very hot rate limiting stack from a single standalone Redis to a 10-node Redis Cluster. Maybe the smoothest production rollout I've ever seen.
3
10
43
Hitting connection limits in Postgres is a common problem to have (and a surprising one when you first run into it).
This piece covers a few ways to make efficient use of available connections through techniques like pools and minimum viable checkouts.
1
18
43
My sympathies for the immortal souls of those who eschew the ORM (or more importantly, anyone who has to maintain their stuff).
One of the most airtight models in software design: ORMs for basic fetches and persistence. SQL for the complicated stuff.
2
4
43
Published a short piece on how to use transactions to build an idempotent API hardened against race conditions.
3
13
43
Today in writing on iPad: had to google the gesture for redo after accidentally undoing a whole paragraph by — I kid you not — "shaking" the iPad by placing it on a table. This certified-100%-undiscoverable touch/gyro input mystery meat buffet has to stop. More keyboards please.
6
4
42
The more I play with GraphQL, the more it seems like the obvious way forward for web APIs.
Strong conventions, introspection (GraphiQL is amazing), and improved change resilience (explicit field requests). You can still design great APIs minus the mystique/dogmatism of REST.
3
6
42
Every time.
The number of human-hours wasted every day by bad error messages are beyond measure. Spend a little extra time on the edges to make them good. It might seem like thankless work, but a great many will owe you a debt, even if unknowingly.
6
5
41
Great to see JetBrains doing a new IDE. Their current IDEs were great for their time, but now comically wasteful.
But they're the only IDE maker that understands the importance of refactoring tools, and those need a comeback. VSCode needs competition.
1
5
42
How to get Postgres logs in a GitHub Action with no added configuration:
This trick saved me recently as I was debugging an intermittent deadlock on upsert, for which almost no useful information is returned in-band, but the logs contain plenty.
1
9
41
