4 ms·
Hey, I'm one of the DVC maintainers. DVC pipelines are very different from Airflow. For me the best mental model is a make file for ML projects. It's a way to d
by shcheklein 5y ago
Hey, I'm one of the DVC maintainers. DVC pipelines are very different from Airflow. For me the best mental model is a make file for ML projects. It's a way to describe the project, how to build it (for example to package and an experiment somewhere outside, or embed as a single step in the Airflow DAG). It doesn't replace or compete with Airflow which is more about production to me.
Btw, what do you mean by "handle CI/CD"?
- billconan 5y agoso when I first saw pipeline, the first use case came in mind was data validation and cleaning, like ETL. In my opinion, what's versioned in DVC should have been cleaned up already. So I would offload ETL to airflow and only version the result of it. Thank you for your explanation. that makes sense to me. I think it's like the project.json of a nodejs project, where you can define a set of commands, such as npm build, npm run, npm serve ...