5 ms·

DVC is (at least as I use it) pretty much just git LFS with multiple backends (guess actually a more simple git annex). It further has some rather MLOps specifi

by riedel 2y ago

DVC is (at least as I use it) pretty much just git LFS with multiple backends (guess actually a more simple git annex). It further has some rather MLOps specific stuff. Is handy if you do versions model training with changing data on S3.

starkparker 2y ago
I've used it for storing rasters alongside georeferencing data in small GIS projects, as an alternative to git LFS. It not only works like git but can integrate with git repos through commit and push/pull hooks, storing DVC pointers and managing .gitignore files while retaining directory structure of the DVC-managed files. It's neat, even if the initial learning curve was a little steep. We used Google Drive as a storage backend and had to grow out of it to a WebDAV backend, and it was nearly trivial to swap them out and migrate.
haensi 2y ago
There’s another thread from October 2022 on that topic. https://news.ycombinator.com/item?id=33047634 https://news.ycombinator.com/item?id=33047634 What makes DVC especially useful for MLOps? Aren’t MLFlow or W&B solving that in a way that’s open source (the former) or just increases the speed and scale massively ( the latter)? Disclaimer: I work at W&B.
- riedel 2y ago
  DVC is much more basic (feels more unix style), integrates really well with any simple CI/CD scripting with git versioning without the need to set up any additional servers. And it is not either or. People actually combine MLFlow and SVC [0] [0] https://data-ai.theodo.com/blog-technique/dvc-pipeline-runs-mlflow https://data-ai.theodo.com/blog-technique/dvc-pipeline-runs-...
matrss 2y ago
Speaking of git-annex, there is another project called DataLad (https://www.datalad.org/ https://www.datalad.org/), which has some overlap with DVC. It uses git-annex under the hood and is domain-agnostic, compared to the ML focus that DVC has.

by riedel 2y ago

starkparker 2y ago

I've used it for storing rasters alongside georeferencing data in small GIS projects, as an alternative to git LFS. It not only works like git but can integrate with git repos through commit and push/pull hooks, storing DVC pointers and managing .gitignore files while retaining directory structure of the DVC-managed files. It's neat, even if the initial learning curve was a little steep. We used Google Drive as a storage backend and had to grow out of it to a WebDAV backend, and it was nearly trivial to swap them out and migrate.

haensi 2y ago

There’s another thread from October 2022 on that topic. https://news.ycombinator.com/item?id=33047634 https://news.ycombinator.com/item?id=33047634 What makes DVC especially useful for MLOps? Aren’t MLFlow or W&B solving that in a way that’s open source (the former) or just increases the speed and scale massively ( the latter)? Disclaimer: I work at W&B.

riedel 2y ago
DVC is much more basic (feels more unix style), integrates really well with any simple CI/CD scripting with git versioning without the need to set up any additional servers. And it is not either or. People actually combine MLFlow and SVC [0] [0] https://data-ai.theodo.com/blog-technique/dvc-pipeline-runs-mlflow https://data-ai.theodo.com/blog-technique/dvc-pipeline-runs-...

matrss 2y ago

Speaking of git-annex, there is another project called DataLad (https://www.datalad.org/ https://www.datalad.org/), which has some overlap with DVC. It uses git-annex under the hood and is domain-agnostic, compared to the ML focus that DVC has.