3 ms·
so when I first saw pipeline, the first use case came in mind was data validation and cleaning, like ETL. In my opinion, what's versioned in DVC should have be
by billconan 5y ago
so when I first saw pipeline, the first use case came in mind was data validation and cleaning, like ETL.
In my opinion, what's versioned in DVC should have been cleaned up already. So I would offload ETL to airflow and only version the result of it.
Thank you for your explanation. that makes sense to me. I think it's like the project.json of a nodejs project, where you can define a set of commands, such as npm build, npm run, npm serve ...