Building generational technology that makes data processing across all modalities simple, reliable and performant regardless of scale.

@databricks @awscloud @puppyquery @dremio @ClickHouseDB @wherobots @streamnativeio @Minio @redpandadata @CelerData @confluentinc @Microsoft @bufbuild @RisingWaveLabs Want to be a part of this grand event? We’re looking for talks! If you’ve been working with Iceberg — whether at scale, in production, or through creative integrations — we’d love for you to share your story 👉

What do you want to know?.

Our biggest meetup yet, featuring cohosts from @databricks, @awscloud, @puppyquery, @dremio, @ClickHouseDB, @wherobots, @streamnativeio, @minio, @redpandadata, @Celerdata, @confluentinc, @Microsoft, @bufbuild, and @RisingWaveLabs!.

Get ready folks for THE Iceberg meetup of the year: October 1st in San Francisco at the Contemporary Jewish Museum.

Leverage the power of data and AI to build the best image intelligence app you can in one night. Register now: Date & time: Thursday Aug 28 @ 4:30pm.Location: Daft HQ in Mission District. Can’t wait to see what you all will build!.

Get your hands dirty with Daft & Daytona at our upcoming Image Intelligence Hack Night next Thursday Aug 28! A chance to win a $200 gift card and 2000 platform credits that you don’t want to miss out on.

RT @lhoestq: Wow the team at @daftengine cooked!. You can now read/write to 🤗Hugging Face with Daft!. > DataFrame engine in 🦀 runs distribu….

RT @essential_ai: Really proud of Essential's collaboration with @daftengine. It simplified web-scale data processing.

RT @ashVaswani: Proud of @RitvikKapila and team’s collaboration with DAFT. It really simplified our web-scale data processing.

3/ The full breakdown of why image processing breaks at scale—and how we fixed it:

2/ Most engineers hit this wall: naive Python works for 100 images, explodes at 1,000. The problem isn't your code. It's that image processing fundamentally doesn't fit traditional data frameworks. We processed 300,000+ images in 30 seconds using the same code that runs.

1/ Here's why processing 1,000 images crashes most machines: Each image inflates 48x during processing. Your 500KB JPEG becomes 24MB in memory. Traditional solution: manual batching, cluster management, constant tuning. Our solution: streaming execution that keeps memory flat

RT @RitvikKapila: 2/Our vLLM inference job was powered by @daftengine, with its massively parallel compute, cloud native I/O and seamless d….

Full case study with more details here: . Why it worked: .• Seamless scaling from local dev → distributed jobs .• Cloud-native I/O + async ops .• Python-first APIs for custom inference logic

Processing 24 trillion tokens without a single crash. @essential_ai built Essential-Web v1.0 using Daft: .• 23.6B LLM queries in 7 days .• 90K AMD MI300X GPU hours .• 32K sustained req/s per VM Scaling vLLM inference at web scale 👇.

Full schedule: Register here: #Daft #Multimodal #AI #DataEngineering #OpenSource #LinuxFoundation.

Whether you're processing terabyte-scale datasets for foundation model training or building real-time inference systems, discover how Daft transforms multimodal data chaos into a streamlined competitive advantage. Check out our session:

Excited to announce that Sammy and Colin will be speaking at #AIDev Europe, 28-29 August in Amsterdam! Check out our session titled “One Pipeline to Rule Them All: Unifying Multimodal and AI Data Processing with Daft.”

@Apple @marimo_io Thrilled to host this incredible community at our office! The energy during talks and networking was amazing. Already looking forward to the next SF Systems meetup (. Maybe we'll have something to share from the stage ourselves soon 👀.

@Apple Next: Akshay Agrawal from @marimo_io presented "Representing Python Notebooks as Dataflow Graphs". Revolutionary approach: modeling notebooks as dataflow graphs enables reproducible execution and callback-less interactive elements. Akshay demonstrated powerful applications:.🔄.

First up: Scott Andreas from @Apple explored "When Faulty Hardware Happens to Flawless Databases" . Key insight: Even in our advanced computing world, any bit can flip. Scott showed how databases can operate safely despite this fundamental challenge. Scott shared incredible.