Late night rant: Spark is an awesome piece of software. But a horrible developer experience. What happened to OSS that was simply `apt install` and 🚀? Why should software be excused for slow local performance because it was built for "production scale"?. So much of "big data".

Ohhhhh yeah data is so back baby.

“What’s your Roman Empire?”. “Daft. URL downloads, dynamic streaming execution….”. 😭😭😭.

Actually crazy. Went through our emails and yes we interviewed him back in 2022 for our first hire at Eventual. Did not make it past our bar, but I remember him being decent. Seems there's a pattern also of him approaching open-source companies.

Like if your natural pace is 100 hours a week… all the more power to you lol.

Interestingly this doesn’t necessarily mean working LESS hours. I actually feel like you could work more hours overall doing this because the work is so much less draining if it isn’t just “busy-work”. See: late night manic coding because of an obsession with quality.

Slow Productivity in the world of tech startups… thoughts?. So far at Eventual the biggest inflection points feel like they’ve come from deliberate thinking and obsessive engineering rather than frenetic 50-calls-a-week-GTM that is perhaps the more common SaaS startup advice

Hold on did I miss something? Querying a long-form SEC 10-k document is still some crazy chained data pipeline in 2025. Something something RAG something embedding something something chunking + LLM calls 🤮🤮. No wonder there’s like 200 document parsing startups.

Incredibly excited to work with the team at @felicis especially @AstasiaMyers who has been an absolute POWERHOUSE for us.

If you’re an ML/AI researcher, let’s work on your data for pre/post-training. If you’re an AI application engineer, let’s talk about your multimodal applications. If you’re an engineer who is frustrated with Spark, let’s get you on a data engine that’s built for 2025.

🐍We're going to accelerate the entire industry towards software that works natively with both multimodal data and LLMs. We're building industry-best solutions for data/AI systems that work on a single machine, on a cluster, on GPUs and also with remote LLMs.

🚀We're growing a team of incredible engineers with both the breadth and depth across domains such as: databases/data systems (ex-Databricks, CMU), distributed systems (ex-AWS, Render), ML/AI (ex-Tesla, Nvidia, Github Copilot) and Product (Stripe, Uber).

🔥 We're now a Series A company with the funding, investors and partners to make our vision a reality.

I couldn't be more proud of our team for this AMAZING milestone today!. To everyone who's followed this journey from the very beginning: a heartfelt thank you, and a promise - there's a revolution coming for multimodal data and AI. We're leading the charge :).

In fact, @desmondcheongzx vibe-coded a custom data sink that would stitch images in the dataframe together into a video and save that as the output of the pipeline. Wacky, but tbh sky’s the limit here :).

Multimodal/unstructured data often means user-defined data. That’s why you’re going to need User-Defined Data Sources and Sinks. This is how you get the best-in-class performance from the daft engine + integration with whatever crazy format you can cook up.

RT @criccomini: .@daftengine is drop-in API compatible with PySpark now! 😈

Daft is now PySpark API-compatible :). Switching your Spark code to Daft is literally 2 lines of code. ```.from daft.pyspark import SparkSession.spark = SparkSession.builder.local().getOrCreate().```. #AntiSparkSocialClub.

Still lots of work that needs to be done here, but I cannot be more excited for this to now be the default execution model in Daft.

Any data engine working with multimodal data needs to be streaming-based and do this intelligent batching. Thus we enjoy the benefits of both parallelism/vectorization as well as memory stability. Otherwise… like in Spark you’re going to have massive OOM issues….

Cannot be understated how much of a paradigm shift this was for @daftengine . In analytics, your data usually gets SMALLER (aggregations, groupby etc). In multimodal/AI… it tends to EXPAND with HUGE heap memory usage. Think: downloading data from urls, running models etc.