4 ms·
One issue my team is dealing with is how to store our large models. We are using git lfs at the moment but it feels woefully inadequate. It is trivial for a t
by qudat 9y ago
One issue my team is dealing with is how to store our large models. We are using git lfs at the moment but it feels woefully inadequate. It is trivial for a team member to accidentally commit the 500mb file to git and completely destroy our repo size.
Not to derail the post, but what does everyone else use?
- conorliv1 9y agoHave you considered using HDFS? There is a fairly informative post here: https://www.quora.com/How-are-most-data-sets-for-large-scale-machine-learning-stored/answer/Sean-Owen?share=d1ba6d56&srid=uA6AR https://www.quora.com/How-are-most-data-sets-for-large-scale...
- mslate 9y agoUpload the serialized model to S3/GS, configure the URL to be re-pulled--treat model versions like feature flags
- jaz46 9y agoPachyderm (my company) offers versioning large data sets in a similar fashion to git without having to deal with gitLFS. Pachyderm isn't Git under the hood, but we borrow some of the major concepts -- repos as versions data sets, commits, branches, etc. The analogy to Git breaks down as you go a little deeper because building efficient data versioning and management is a surprisingly different problem than code VCS