3 ms·
> better support for ML... such as data and model versioning options? I'm not familiar with the kinds of files ML deals with. But if you want to version contro
by foobarbaz33 5y ago
> better support for ML... such as data and model versioning options?
I'm not familiar with the kinds of files ML deals with. But if you want to version control huge data dumps or non-textual files, git is not ideal. Or use it as a general purpose "share folder" to store random things. A git repo is best fit to store source code for a single project.
SVN might be OK if you insist on using 1 source control system for both code and dumps.
But I don't think you need to force all things into 1 system. It would be acceptable to store pure source code in git. And dumps in something else, maybe using old school backup strategies rather than source control if they are truly massive.
- sepiasaucer 5y agoCouldn’t git lfs handle large non-textual files? How does SVN hand large non-textual files?
- tomrod 5y agoDepending on implementation, model objects can be binary objects or sometimes XML/JSON. Usually the former, and I'm not sure what ONNX uses under the hood. The ideal for data outside of databases in my world is parquet or feather, which again are binary (if I understand the spec right).
- kelseyfrog 5y agoGit+DVC is an option. I haven't personally used it, but I'm prepared to explore it when my team stumbles onto the problem of "code and data need to be in sync or things will break in very terrible ways."