Prateek Gaur and co at @Snowflake reproduced the (great) results for the ALP encoding algorithm from @cwi_da / @afroozeh3 / @peterabcz. ALP achieves ZSTD levels of compression and much faster decode. We are discussing adding it to @ApacheParquet: https://t.co/gxwF5QqtNO

@SpiralDB @CMUDB @ApacheParquet The idea of using WASM as a forward compatibility mechanism I thought was especially neat

The talk on @SpiralDB at @CMUDB :  https://t.co/6mRfsnDZiP is a great one. I think it would also be interesting to hear a counterpoint about @ApacheParquet that explains actual technical details of that format, the Cathedral vs Bizzaar management, options with Metadata, etc

Our new thrift parser in the Rust @ApacheParquet implementation is a 🎁 that keeps on giving performance wise 🚀 https://t.co/b6lHJbxQzd We are also working on a blog post that has a deeper explanation

Yesterday I learned about the SpatialBench from Sedona https://t.co/I7MYOptkuK Which they based on our tpchgen-rs project: https://t.co/PR0F0AS9SD (BTW I a still looking for some more github watchers on tpchgen-rs so I can get it on homebrew)

BTW if anyone wants a good intro to database storage / Log structured storage (aka LSM trees), the @CMUDB lecture this fall is a good one:

It starts: https://t.co/0fhieCL0BX clfushopt is going to make the worlds fastest tpc-ds generator

I am proud to announce I am now a committer on the @ApacheParquet project. Realistically this likely means more reviews / helping clarify the parquet specs, but I also hope to help more actively evolve the format, especially around new encodings. https://t.co/lnR71Po1yA

I am really proud to announce that we raised €18M in series A. We have got big plans on improving Polars. Great things to come!

BTW I would love some help getting some official DataFusion 50 benchmarks into ClickBench --

DataFusion 50 is released. Read all about it here: https://t.co/B2OHkMPL3h

CloudFlare's Distributed R2 SQL engine's is a pretty good exemplar of how to build a serverless database to process petabytes in seconds using @ApacheDataFusio and @ApacheParquet https://t.co/8QSk6fOwuQ

@ApacheParquet @ApacheDataFusio Check out the follow on from @jcsherin who used these techniques to put full text indexes in parquet:

So cool: @jcsherin added full text indexes into Parquet files using the techniques from our blog https://t.co/t0eDGHeG9c

"Introducing SedonaDB: A single-node analytical database engine with geospatial as a first-class citizen" Built in Rust with @ApacheDataFusio https://t.co/bsneiAJFRv

Example of the level of optimization obsession possible with the @ApacheDataFusio community ❤️ :

We just published an easier to find list of all PMC and committers on @ApacheDataFusio, and it is quite a cool list of people and affiliations if I do say so myself 🤗 https://t.co/OOYNgf58eZ

And we are also adding Geometry to the Rust parquet implementation . Huge thanks to @kylebarron2

It was a great time on Monday at the @ApacheDataFusio meetup in NYC. We heard about distributed query plans, filter pushdown, geospatial support, and VegaFusion. More deets here https://t.co/Axugrv05P3

6 hours to generate TPCH SF750000 dataset using a worker pool of 1000 parallel processes (spread across 25 VMs). BTW SF750000 is 750TB raw / 220 TB parquet. https://t.co/Piqq2ubVIw