Its happening -- DataFusion will (finally) get spilling hash joins. The march to completeness begins.

RT @jonathanc_n: I'd like to start using this platform as a place to post about open source work I do on my off time. To lead it off, I ha….

Join us in Boston at the DataDog offices for the @apachedatafu meetup on Nov 12 for pre 🦃 discussion of databases

🎣 Anyone want to try and help implement a proposed improvement to @ApacheParquet for better Floating point support? Open source fame and glory await . 🙏🙏🙏🙏🙏. [Parquet] Prototype: PARQUET-2249: Introduce IEEE 754 total order & NaN-counts #514 #8156.

It is a common misconception that Parquet requires (slow) reparsing metadata and is limited to built in indexing structures. Using External Indexes, Metadata Stores, Catalogs and Caches to Accelerate Queries on @ApacheParquet with @ApacheDataFusio .

Despite a seemingly common misperception of @ApacheParquet implementation fragmentation, there is a lot of green on the implementation status page:

RT @SpiralDB: It's official: Vortex has been accepted as an Incubation-stage @linuxfoundation project 🍾. The core of the Composable Data St….

RT @pauldix: Congrats to @_willmanning and the @SpiralDB team for getting Vortex hosted by the Linux Foundation! M….

Its starting -- Ed is planning to write a custom @ApacheParquet thrift decoder. I expect a 2-4x improvement in footer parsing with no format changes required. I am pretty stoked to see this.

Refactoring SUM(x), SUM(x+1), SUM(x+2) . is also another dead giveaway of Clickbenchmaxxing:

Benchmaxxing (verb): to add specific optimizations that only impact benchmark results and are not widely applicable to real world use cases.

We are doing another DataFusion meetup in Boston Wednesday Nov 12, 2025

RT @OnlyXuanwo: Some good first issues available in @ApacheOpenDAL, welcome to jump in!.

"EDB claimed the new engine, which pushes queries to open source @ApacheDataFusio , returned queries 30x faster than standard Postgres while tiering offloads cold transactional data to storage is up 18x more cost-efficient.".

Mutli-level merge sort queued up for DataFusion 50.0.0 next month: Thanks to @rluvaton and Yongting You.

@cwi_da @peterabcz @afroozeh3 @ApacheParquet The confusion between format and implementation is common in academic papers, and I think hinders industrial adoption of the technology. For example, the number of format implementation:.BtrBlocks: 1 .FastLanes: 1 .Parquet: 10+ (that *I* can name), an order of magnitude more.

@cwi_da @peterabcz @afroozeh3 @ApacheParquet Finally, the paper several statements about "Parquet" which are really about a particular implementation (probably DuckDB's) -- e.g. "access granularity", "can return compressed vectors", "uses physical sizes for row groups".

@cwi_da @peterabcz @afroozeh3 @ApacheParquet I understand the value of a new format for flexibility while researching but I think the paper's contribution is more broad. I believe the paper would be more impactful if more emphasized that the techniques do not *require* a new file format.

@cwi_da @peterabcz @afroozeh3 @ApacheParquet The paper was a good read: Specifically I think the Encoding Expressions framework is a very nice idea for expressing cascading encodings.

The FastLanes format paper from @afroozeh3 and @peterabcz contains interesting and practical ideas for representing SIMD friendly cascaded encodings. I think almost all of the ideas could be applied to extend @ApacheParquet as a new encoding scheme.