v
@iavins
Followers
13K
Following
18K
Media
752
Statuses
7K
breaking databases @tursodatabase. W1 '21 @recursecenter excited about databases, storage engines and message queues
b tree
Joined April 2010
For the past few months, I have been learning about internals of databases. I found many excellent articles on writing compilers, but I could not find many practical resources for databases. So I wrote one. CaskDB is the project I wish I had started with. https://t.co/95EcPAjP4w
github.com
(educational) build your own disk based KV store. Contribute to avinassh/py-caskdb development by creating an account on GitHub.
10
95
678
TIL about Preston Thorpe. He learned Rust in prison and it eventually gave him an opportunity to be able to work for Turso remotely where he's working on the Rust implementation of SQLite. He sounds pretty diligent and knowledgeable. What an incredible person.
15
25
657
don't miss this 👇
Here we go TigerStyle 🚀 Episode with @jorandirkgreef Founder & CEO, @TigerBeetleDB on TigerStyle is now available to watch.. https://t.co/J2n3gRVDVe Please like, share and subscribe!
0
3
18
Over 85% said that Orange's transaction should commit here. I agree! In snapshot (Postgres Repeatable Read) and lower, Orange will commit, allowing 'write skew'. In the serializable alternative, Orange has to abort. No serial ordering of these transactions exists.
Should orange's commit succeed?
4
9
60
This is getting dunked on a lot, but I for one loved it!
Programming Isn't Math, It's Linguistics. Compilers and Humans have the same problem. We're all terrible at understanding each other. Join me for some formal language theory, a lot of C++, and some "recreational" insults.
1
0
22
underrated tweet
0
0
14
It’s crazy how many grown-ass adults think there’s some kind of mutant motivational spider that’s gonna bite them and turn them into intellectual Spiderman the next day. Wishful thinking. Iron Man is a much better mental model. Incrementally build up your skillset.
Ability is built, not unlocked. You do not have a latent superpower that can be unlocked in a day. You have to build up a skillset. You build it up high enough and then other people think it’s a superpower you always had. It’s crazy how many grown-ass adults think there’s some
5
16
158
how the lock gets released if the process dies abruptly🤔
29
4
261
super inspiring story ♥️
Rewriting SQLite... from prison! In this interview I talked with @PThorpe92, who is a full time software engineer and currently incarcerated for non-violent offenses. He's helping @tursodatabase rewrite SQLite from scratch. It's an inspiring story. https://t.co/ezVRAOIsTi
0
0
18
Redis list provides "at-most-once" delivery semantics (kinda). Very few applications are okay with at-most-once delivery. Whereas in systems like RabbitMQ, messages are preserved until you ack that you have processed them.
1
0
5
This tweet which advocates for Redis for everything has about 500K views, quite insane. Sadly, it's pure AI slop. e.g.: Below is not reliable at all. Once you BRPOP, it's removed from the list. If your app reboots due to new deployment or crashes, all messages would be lost
2
0
8
hacker news doesn't think that SQLite can be run directly on S3. they're wrong. @archildata makes it possible to run any program on S3, without extra persistent storage, including complex read/write database use cases
16
12
197
wow, this is nice 👏 If I use this with SQLite for read heavy workloads, I get instant free replication
wouldn't it be great if you could just go to the s3 console and run unix tools on what's in your bucket? now you can. today, @archildata is launching Shells, a one-click, ephemeral Linux machine attached to your S3 bucket so you can run vim, less, and grep right on your data.
1
3
40
I explained about SQLite's neat little trick here about reserved space management per page (6/6) https://t.co/pZ3Ik79UQn
The B Tree data structure fascinates me. Databases use B Trees to store data on disk, organizing everything into pages that typically range from 4kb to 8kb. All I/O operations happen in units of these pages. The page looks like this... (1/9)
0
0
3
For example, if we use AEGIS-256 with a nonce size of 32 bytes and a 16-byte tag, you'd need extra space for 48 bytes per page. During decryption, you read the tag and nonce from the reserved space and provide them to the decryption algorithm. (5/6)
1
0
1
The size of this tag and nonce varies by algorithm. To make space, SQLite uses reserved space per page. Once the page is encrypted, this portion can carry the metadata. (4/6)
1
0
1
Since we encrypt each page separately, we should use a different nonce for different pages. Even the same page, when encrypted again, should use a different nonce. So during encryption, we generate a secure random nonce every time for better security™️ (3/6)
1
0
1
Both the nonce and the tag become metadata for an encrypted page. We could store them separately, but it's much better and neater to store them in the page itself (2/6)
1
0
2
So the tricky part is: how and where do you make space in the B Tree page to store some additional data? AEADs have a verification tag, and then there's a nonce per page too. We need to store this data somewhere safely so it doesn't interfere with B Tree operations (1/6)
You never want to write your own encryption algorithm implementation. That task is best left for experts. However, you do want to spend time understanding the features and APIs of the algorithms. Allow me to explain Authenticated Encryption. Encryption algorithms take a bunch
1
0
9