I've written 18 posts (and counting) on table format internals. I've created a page that contains the list of my writings on the subject, including my formal verification work. Any suggestions on further table format analysis?.

New deep dive: Understanding Apache Fluss. I spent August reverse-engineering Fluss, Alibaba’s new table storage engine for Flink (partially forked from Kafka). This post covers its architecture, tiering, and how it tackles changelogs & low-latency state.

There's been some talk around storage unification and zero-copy lakehouse integrations recently, and I wanted to better define these terms as well as look at ways we should evaluate different design decisions in this space. So I’ve published a new post: A Conceptual Model for.

In a future of autonomous AI agents, we can't limit ourselves to error prevention and error detection, we must also include remediation. But when AI loses touch with reality due to hallucinations, confabulation and misinterpretation, who does the remediation? In cases of.

Science moves slowly because wrong theories waste decades. Engineering is careful because failures kill people. Software moves fast because mistakes are cheap, the expensive error isn't making the wrong choice, it's taking too long to make any choice.

A new case study is born.

In distributed systems, reliability isn’t just about retries and durability, it’s about knowing who owns recovery. My latest post, based on the Coordinated Progress model I posted previously, explores how reliable triggers create responsibility boundaries and how those boundaries.

Over the past few months, I’ve been thinking deeply about how systems make progress reliably in the face of partial failures, service boundaries, retries, and complex dependencies. Building reliable workflows across microservices, functions, and stream processors is one of the.

How to reliably distribute work across microservices, stream processors, durable execution, event-driven, orchestration and now AI agents?. Coordinated Progress is a 4-part series that explores the common structure behind reliable distributed systems.

Another Humans of the Data Sphere is out, with issue 10! In this issue people are talking fsyncs, tips for running ClickHouse at scale, the problems with MCP and more. Plus I dig up a classic paper from 1962.

The specs are in this repo for anyone interested.

Proud to have contributed formal verification (TLA+) for three key improvements in Kafka 4.0:. ✅ KIP-966: Strengthens the replication protocol. ✅ KIP-996: Introduces PreVote for more stable KRaft leadership. ✅ KIP-848: Delivers more efficient, predictable rebalancing.

Seems like I’m not alone. For what it’s worth, I’ve got a great fit at Confluent — but the more senior I get, the more I wonder how sustainable that is across future PE roles. Thinking of writing a blog post, maybe with interviews or perspectives from PEs who aren’t natural cat.

RT @ijuma: And the old group coordinator implementation is gone from Apache Kafka - love it when open-source projects can delete large chun….

Any Principal Engineers out there with ADHD or creative wiring — who don’t thrive in the tasks of project coordination, alignment meetings, and people management, but thrive on strategy, system design, writing, and shaping direction through ideas? Curious how you navigate the.

A new disaggregated log replication survey post is out. How does the combination of Apache Pulsar with Apache BookKeeper divide and conquer the responsibilities of log replication?

Another Humans of the Data Sphere is out, with issue #9! In this issue, we also look at whether software engineers can learn from mechanical engineering, and looking at table formats as a form of virtualization.

RT @ankushpd: If you are looking for formal models of a real-world distributed system, DeepSeek @deepseek_ai released P specifications for….

A new log replication disaggregation survey post is out! .The Kafka Replication Protocol:.🔹Separation of control plane from data plane. 🔹Role separation with minimal coupling. 🔹Kafka’s alignment with Paxos roles.