- A more scalable approach could involve integrating with open table formats like Delta or Apache Iceberg. (we definitely plan to add Iceberg support, and Studio - our SaaS offers ClickHouse support that scales to billions rows scale).

Key takeaways:. - Managing video metadata is more complex than it seems; DataChain aims to simplify this. - While DataChain offers useful features, the use of SQLite within open source version, raises concerns about scalability for larger datasets, and cross collaboration.

Full review is here

"Metadata marts could play a key role in making video data more accessible and structured for model training and analysis" - Simon Thelin (@synthesiaIO, creator of the DataPains blog) reviewed DataChain 👇.

@PyData @JulianWgs ✅ Finally, metrics, parameters are captured and also attached to Git and iterations - to compare, visualize result;.

@PyData @JulianWgs The usual DVC building blocks are utilized to achieve this:. ✅ DVC data versioning makes sure input data is saved and attached to an iteration and can be restored or access anytime in the future;.✅ Lightweight CLI pipelines declaratively describe and run data processing.

@PyData @JulianWgs suggests using DVC to streamline simulation iterations and track results. DVC keeps it lightweight (no need to run servers and such - CLI, Git, basic Python) while making the whole process way more manageable and scalable.

Full talk on YouTube by @PyData :.

Watch an excellent PyData Berlin talk by @JulianWgs (@VW ) on automating and managing fluid simulations with Python and DVC. 🧵

@photoroom_app @EliotAndres @EliotAndres presentation - at 13:28.

@photoroom_app @EliotAndres DataChain is an open source library 🔗 (and SaaS platform for collaboration and scale) that implements this idea at scale + our goal was make it easy to use by ML teams.

@photoroom_app @EliotAndres 🐘 Smaller scale - DVC + CSV files, or Postgres + some custom ETL to feed it.

Some examples and ideas we've seen:. 🎥 @photoroom_app - see the link to the talk by @EliotAndres below.🧊 Iceberg for metadata (and sometimes binaries) + Spark - is one of the default choices (but usually requires data engineering skills / team).

All efficient AI teams have an excellent data hygiene - they utilize databases, ETLs, custom scripts / tools to effectively build a metadata layer on top of their binary data (videos, images, etc). This is essential to make better datasets.

DataChain organizes and makes your AI data queryable! What does it mean? Why? 👇

RT @DVCorg: 🧵Under the hood DataChain combines power of warehouses with distributed clusters with proper data access patterns to process mi….

🤏 Data access. Pre-fetch, batching, caching, streaming - different workloads require different ways of using data.

⚙️ Distributed compute that runs close to the data to compute Python-based UDFs.

⚡ Use warehouses under the hood (e.g. ClickHouse) to store metadata and perform as many operations inside it (e.g. filters).

☁️ Never copy data. Store references to files instead. (while still preserving versioning, data loading, efficient processing).