2 ms·

Pachyderm is another alternative. Here's a couple articles that compare the two: https://www.pachyderm.com/blog/data-versioning-comparing-dvc-with-pachyderm/ h

Pachyderm is another alternative. Here's a couple articles that compare the two: https://www.pachyderm.com/blog/data-versioning-comparing-dvc-with-pachyderm/ https://www.pachyderm.com/blog/data-versioning-comparing-dvc... https://www.dolthub.com/blog/2022-04-27-data-version-control/ https://www.dolthub.com/blog/2022-04-27-data-version-control... As far as I know, DVC is better than Pachyderm for small datasets, but Pachyderm scales way better

claytonjy 3y ago
I built an ML pipeline in pachyderm years ago, maybe 2016, and loved using it then. But my data was small-to-medium, and I stuck with CSVs because back then you couldn't get many of the benefits with a binary format. I think it's a shame it took Airflow and similar several more years to realize "each step is a docker container" is the right way to build a dag. It's not clear to me why Pachyderm was left behind while Prefect and Dagster became serious contenders, and Airflow/Astronomer started recommending everyone use it just like Pachyderm (container per step).