Ben Johnson
@benbjohnson
Followers
12,908
Following
32
Media
271
Statuses
11,518
Working on LiteFS & Litestream at @flydotio to help developers run SQLite in production and at scale.
Colorado
Joined March 2008
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
キスの日
• 276192 Tweets
Bronx
• 164284 Tweets
Taiwan
• 124886 Tweets
INCOMPARABLE WINNER ELVISH
• 77603 Tweets
HBD YTSTAR ABHISHEK MALHAN
• 76004 Tweets
Beşiktaş
• 72216 Tweets
FELIX LOUIS VUITTON CRUISE
• 64018 Tweets
CHEERS TO YOUTH MV IS OUT
• 60510 Tweets
LAYOVER DOMINATION
• 51173 Tweets
CONGRATULATIONS TAEHYUNG
• 48503 Tweets
#الهلال_الطايي
• 39548 Tweets
DILAW OUT NOW
• 38067 Tweets
سالم
• 26316 Tweets
الرائد
• 19131 Tweets
#BJKvTS
• 17860 Tweets
$BEER
• 15974 Tweets
SummerSlam
• 15632 Tweets
Montero
• 15550 Tweets
Claudio Iturra
• 13867 Tweets
Necip
• 10001 Tweets
Every time I hear someone say they need five 9's of uptime, I think of GitHub. Everybody relies on them, they have maybe 99.9% uptime, and they got bought for $7.5B by Microsoft.
Unless you're making pacemakers, ease up on your crazy uptime requirements.
99
676
4K
Replace a semicolon (;) with a greek question mark (;) in your friend's JavaScript and watch them pull their hair out over the syntax error.
131
3K
3K
My rule of thumb is that you’ll probably spend at least $1M to move to Kubernetes. That’s new ops hires, training, initial productivity loss, etc.
If you don’t have a $1M problem that Kubernetes is solving then I don’t think you should even consider it.
42
323
2K
Does anyone code anymore or are we all just reading AWS & Kubernetes documentation all day?
45
165
1K
I'm starting to write about SQLite internals which are both fascinating and surprisingly accessible. This first post is on the file format, pages, & b-trees.
I've got plans for future posts but I'd love to hear what folks would like to know more about!
15
144
699
So apparently a poor design decision I made on BoltDB in 2013 took down
@Roblox
for 3 full days last year. Damn. I'm sorry to all the folks at Roblox and
@HashiCorp
that probably pulled all-nighters to debug that issue.
15
68
658
Big news! I've joined on with and I'm working on building out SQLite replication & distribution tooling full-time now. We have some pretty big plans ahead. 🎉
36
51
566
I open sourced a new SQLite replication tool I've been working on called LiteFS. It allows you to run multiple replicated nodes and hand-off leadership on the fly. It's still in the alpha stage but I'm looking for feedback. Thanks!
2
92
523
I assume this is a joke but it really makes me want to try it out too. An AWS resource can have 50 tags and each tag value can be up to 256 Unicode characters.
SQLite supports a 512b page so you could pack that into the Unicode space available.
13
27
402
Love distributed systems and want to put your skills to the test? We teamed up with
@aphyr
to build a set of distributed systems challenges called the Gossip Glomers.
It's got eventual consistency, distributed logs, g-counters, & more!
12
85
399
2016: Serverless Architecture
2017: Codeless Architecture
2018: Architectureless Architecture
11
370
373
My SQLite replication tool is coming together. I hit over 1K write tx/sec on a $15/mo
@digitalocean
box while doing real-time replication to a separate block storage device & S3. The S3 replication only costs 1¢/hr! 🎉
I hate this phrase but seriously, it's kinda game-changing.
11
26
338
I'm continuing my SQLite internals series with a post on the rollback journal. It's a surprisingly simple approach to keeping your data safe & transactional.
7
50
319
Have you ever wondered how SQLite takes a SQL string and transforms it into a result set? I wrote about the parser, optimizer, & compiler, as well as the SQLite virtual machine that runs it all.
It's amazing what SQLite can fit in such a small codebase.
4
61
317
I've had some feedback that I should write a post on when NOT to use SQLite (aka "when does SQLite suck?").
I have my own list of situations where SQLite isn't great but I want to make sure I cover my bases. Anybody got good SQLite horror stories, or maybe just rules of thumb?
31
25
269
It's still rough but I got live read replication working for Litestream. This demo app on
@flydotio
has a SQLite DB in Chicago that's replicated out to 15 other regions around the world in real-time.
Click the button and watch the replication propagate!
8
35
255
I've been iterating a streaming SQLite tool for over a year. I'm finally releasing Litestream as beta!
You can now safely run fast, single-node applications that continuously backup to S3. I'd love feedback and leave a star if you find it interesting!
8
44
232
This frame accurately captures the feeling of developing with microservices.
8
31
228
"SQLite doesn't scale" is a common refrain but that's changed significantly in the last decade with the introduction of the write-ahead log, or WAL.
I wrote on the internals of the WAL and how it's a huge leap over the rollback journal:
4
35
224
Anybody know who drew this gopher? It has to be one of my favorites. 😂
9
12
225
We just released LiteFS v0.2.0—the distributed file system for your SQLite databases! It transparently handles replication so you can cheaply run your databases on the edge.
I wrote up some thoughts on where LiteFS is at and where it's going:
4
46
212
#protip
: If you're executing a long running script in on Mac OS X, add "; open -a Terminal" at the end and your Terminal window will refocus when it's done running.
Then you can read your Twitter timeline or watch cat videos without forgetting to check back in on your script.
3
43
207
P.S. I don't mean that as any slight to GitHub. I think they do a great job. Services go down sometimes—life goes on.
1
7
203
I wrote about why I built Litestream while ranting about the direction of our industry. Software is so much more complicated than it needs to be but there is potential to change that.
12
40
199
LiteFS started as a way to distribute SQLite geographically but why not distribute databases between services? Building APIs is a PITA so we skipped it and just replicated the database instead.
And that opened up some more interesting use cases!
5
26
167
My biggest surprise in reading the Go critiques this week is in how minor the complaints were. Like, I can think of way worse things to say about Go and it’s still one of my favorite languages.
13
4
148
I would like to clarify some things with regards to my BoltDB README message.
13
46
149
The biggest issue Go developers have with application layout is thinking of packages as groups instead of layers.
7
34
142
My Raft visualization made it up on HN today. I wrote that ten years ago and it's still got a bit of life left! 🎉
I keep meaning to get back into doing visualizations. Any distributed systems or database topics folks want to see visualized?
8
8
136
I gave a talk at GopherCon on why you should consider using SQLite for your next production database. If you missed it live, it's up on YouTube!
5
19
133
Gentle reminder that you don't need to tear down other people's work to showcase your own. We're all doing the best we can.
1
72
132
I've been hiding away for the last three months like a code gnome working on LiteFS. And today I get to announce we have WAL support! And also about a hundred PRs for stability and hardening. 🚀
Thank you to everyone who has tried it and given feedback!
1
16
131
I wrote up some thoughts on the Go object lifecycle after a Twitter discussion with
@rakyll
last month. You wouldn't think I could write 2,000 words on creating objects but, well...here we are.
2
34
128
I just realized I use the GoSH stack (Go, SQLite, HTML) and I gotta say I like the acronym.
3
10
117
"Go Walkthrough" is now a Medium publication! Thanks to
@egonelbre
for the awesome hiking gopher logo.
4
61
115
Have microservices & kubernetes made it more difficult for companies to bring on junior developers? It seems like there is such an escalating level of complexity where even mid-level and senior developers have trouble.
15
13
105
Writing CRUD application is boring, sure, but we do it A LOT. Since Go is notoriously anti-framework, you need to bring your own design. I walk through my approach and different design trade-offs in this post.
5
17
97
Go tip: If you're nesting maps (e.g. map[string]map[int]...) then you can usually simplify your code a ton by using struct keys. .
#golang
1
24
96
This week I wrote about organizing your SQL layer in your Go applications. It's a complex topic so I'm breaking it out into a few parts over the coming weeks.
3
17
94
Nice! LiteFS made it to the orange site front page.
It's always hard to explain HN to non-engineers considering my repo on distributed SQLite is competing with an instructable about duck tape book binding. 😂
3
5
91
We've released managed backups for SQLite! You can now run distributed SQLite on your application instances with LiteFS but also get the safe, warm, fuzzy feeling of point-in-time restores.
2
11
88
Am I doing this right?
6
9
85
You know what would be a cool hackathon? Where people get together and implement academic research papers. That would be badass.
11
89
82
Distributed systems UX is a bit of an oxymoron but it can exist!
I wrote about how we track consistency in LiteFS from an app's perspective and how it's evolved over time. We took a few mulligans early on but it feels like it's in a good spot now.
1
13
80
Apparently the rank 1 & 2 results for "systemd book" are an erotic fan fiction novel about the init system. And it only gets weirder when you read the reviews.
5
15
78
After four years, I'm revisiting the ideas of Standard Package Layout with a post looking at the WTF Dial package structure. I even add fancy graphs!
5
8
80
Public beta of the official SQLite WASM library is slated for next week. I can't wait!
2
11
78
This is basically how CI/CD pipelines work too.
0
12
78
I believe the open source model is untenable.
@pauldix
gave a good overview in his talk so I won't repeat it here:
2
20
76
@PetrHurtak
@SebastianSzturo
@dchest
SQLite added STRICT mode late last year to do type enforcement.
I haven't found the limited type system too bad with the notable exception of TIMESTAMP. I wish they had that.
1
5
72
Printf-debugging is the best worst practice. That’s how I debug about 95% of the time.
"Worst Practices" is a new YouTube channel about how people *actually* develop software. Plan is to publish one very short video each week. First episode is here — enjoy!
4
4
46
5
3
73
I google how to write the shrug emoji once a week so I finally just added it to my bash profile. 😂
alias shrug="echo '¯\_(ツ)_/¯'"
7
4
72
We just released LiteFS v0.4.0! The main focus was on making it easier to run and making it more transparent to the application.
We want it to feel like you're just writing a plain old SQLite app, except, ya know... it's globally distributed.
0
12
71
Go Walkthrough: io package
4
36
65
I'd like to start doing visualizations again like I did with this Raft visualization a decade ago. Any algorithms or CS concepts or tools I should do an explainer for?
6
2
64
LiteFS Cloud is moving out of "preview" and it's now generally available! I posted some details including pricing on our Community forum.
1
9
62
Is there anything like MS Access but for SQLite? The low-code platforms seem like they connect to client/server databases but not to SQLite. Seems like you could run a SQLite-based platform really cheap.
15
1
61
Great write-up on
@Tailscale
's transition from etcd to SQLite + Litestream. I appreciate the "this is not an April Fool's joke" disclaimer at the bottom. 😂
We recently celebrated
@Tailscale
's 3rd birthday.
Here's a post from
@davidcrawshaw
on Tailscale's 3rd database:
One day we'll stop migrating databases, but not today!
10
45
332
0
4
59
How is it that nearly every programming community struggles with sufficient documentation yet every LaTeX module seems to have a 600+ page PDF manual?
5
7
58
I had to give some ideas for visuals for the LiteFS blog post to our in-house illustrator and I said, “well, it ships logs.” The illustration she came up with was awesome. 😂
2
3
58
For anyone curious, this runs on 19 shared-cpu-1x VMs that have 256MB of RAM. Each one has a 1GB volume although only the primary actually needs one. The cost for all that is $38/mo.
Bandwidth is $0.02/GB and pages are LZ4 compressed so it costs around $1 per 1M writes.
It's still rough but I got live read replication working for Litestream. This demo app on
@flydotio
has a SQLite DB in Chicago that's replicated out to 15 other regions around the world in real-time.
Click the button and watch the replication propagate!
8
35
255
1
6
58
TIL that SQLite will save C-style comments to your schema table.
SQLite's catalog is pretty simplistic so it makes sense but it does make for a nice way to document your schema.
1
7
56
I'm going to do some blog series on writing applications in Go—kinda like WTF Dial, but I'll actually finish it this time. 😂
Does anyone have ideas for applications to build? I'm looking for something real-world & useful (unlike WTF Dial).
17
2
56
Regarding office hours, I'd like to be able to chat with folks about Go application architecture and provide advice/feedback.
I feel like there's a lot of folks who shoot themselves in the foot early on and can't get out from under it later on. If you're interested, hit me up!
6
5
55
Great detailed post from
@benhoytnz
on getting Go + SQLite running on . Even includes load testing on our tiny 256MB VMs! 500+ req/sec is pretty good! 👌
0
10
54
New release of Litestream is out! It now has Docker & ARM support, 3-4x faster recovery, and we've switched from GPL to Apache 2. 🎉
Next, we'll be working on refactoring the replicas to add native GCP & Azure support and adding better K8S support.
3
7
55
I completely agree!
Reviews should take into account the severity of the change as well.
Making some minor layout changes? Maybe don’t need a review. Changing your billing code? You better get eyes on that shit.
3
6
55
@bketelsen
@jeffsand
@listonb
@ashleymcnamara
"It looks like you're trying to use channels. Would you like some help with that?"
3
6
51
My ritual with Go releases is to thoroughly read the release notes top-to-bottom, get excited about a bunch of features, and then forget to actually use them for six months.
I’m stoked to use TryLock() some time in Sept 2022!
2
0
53
Do you ever have an idea that's so nerdy that you're taken aback? I had an idea to do a research paper Twitch stream. Like, instead of a programming stream we'd read over and breakdown CS research papers.
4
0
53
Yesterday, I was diagnosed with Aspergers (ASD L1). I'm not sure how I feel about it but it definitely feels like it fits.
I hope I haven't made anyone feel uncomfortable in the past at conferences or meetups—I'm just not great at social interactions. I'm truly sorry if I have.
5
0
53
Great hands-on post about running Go & SQLite. It's crazy how fast the two are together.
2
3
50
It's a weird feeling to spend so long working on a complex database replication tool and it ends up
#1
on HN for a paragraph I wrote at the bottom of the README. 😂
7
3
49
I would love to see alternatives to talks at conferences. Talks communicate what is understood. Discussions figure out what we don't know.
12
9
49
Anybody building applications with SQLite? I love writing apps with embedded databases so I wrote a SQLite streaming replication tool that's badass (if I do say so myself). I'd love to get some usability feedback from folks.
12
8
49
TIL Go has environment variable expansion function in the standard library called os.ExpandEnv()
4
7
48
ICYMI: Have a midsized Go application? I'm starting up online office hours for anyone who wants to get feedback on Go application architecture & design.
Sign up and let's chat. 🚀
2
13
46
It was always implicitly the goal to finish Bolt. There was never going to be a Bolt 2.0. Sometimes software should just be done.
2
3
47
I wrote a library for immutable collections in Go (List, Map, & Sorted Map). The library is in alpha but I'm looking for feedback on the API & documentation primarily.
3
11
46
I started writing again—this time a little rambling on how I manage errors in Go.
4
11
43
What kind of monster chooses a tab width of 7? How is this sacrilege allowed!?
5
3
43
The free storage part of this is seriously awesome. It's a great way to get a SQLite-backed application up and running for free.
I ran load testing last year on their 256MB instances and they're fast. I was able to hit thousands of req/sec with a basic Go/SQLite/HTTP benchmark.
Postgres on is now free for small projects. The hard part about free Postgres is storage, so this post is also about free storage.
27
106
693
3
3
44
If you're debugging raw bytes in Go, hex.Dump() is super easy and helpful.
3
7
43
A Discussion on Enabling Independent Open Source Businesses, in two parts:
5
14
42
Tech interviewee pro tip: hand the interviewer a “manager take home test” when they ask you to do a coding test.
2
6
42