3 ms·
I've had a great experience using DVC for both data versioning and pipelines before. Can you tell us why Ploomber is a better solution than DVC?
by arjvik 5y ago
I've had a great experience using DVC for both data versioning and pipelines before. Can you tell us why Ploomber is a better solution than DVC?
- edublancas 5y agoI'm not a DVC user so I'll speak for what I've seen in the documentation and the couple examples I ran a while ago. DVC's core is data versioning and the pipeline features are an extension to it. The main difference is that DVC's pipeline feature is agnostic: you define the command, inputs and outputs, and DVC executes pipeline. On the other hand, Ploomber has a deeper integration between your code and the pipeline. For example, our SQL integration allows you to tell Ploomber how to connect to a database and then list a bunch of SQL files as stages in your pipeline (example: https://github.com/ploomber/projects/blob/master/templates/spec-api-sql/pipeline.yaml https://github.com/ploomber/projects/blob/master/templates/s...), this reduces the boilerplate a lot since you only have to write SQL, if you wanted to do the same thing with DVC, you'd have to manage the connections and create bash scripts to submit the queries. The other important difference is that AFAIK, DVC can only run your pipelines locally, and Ploomber can export your pipelines to run in other environments like (Kubernetes, Airflow, AWS, SLURM, Kubeflow), this allows you to run experiments locally but easily move to a distributed environment when you need to train models at a larger scale or want to deploy an ML pipeline.
- esotericpedant 5y agoDVC also runs pipelines in the cloud so you can run experiments remotely at different scales and share results with others.
- edublancas 5y agoInteresting, can you share a link to the docs? I'd like to learn more about their approach.