5 ms·
There are a couple of other contenders in this space. DVC (https://dvc.org/ https://dvc.org/) seems most similar. If you're interested in something you can se
by jrockway 4y ago
There are a couple of other contenders in this space. DVC (https://dvc.org/ https://dvc.org/) seems most similar.
If you're interested in something you can self-host... I work on Pachyderm (https://github.com/pachyderm/pachyderm https://github.com/pachyderm/pachyderm), which doesn't have a Git-like interface, but also implements data versioning. Our approach de-duplicates between files (even very small files), and our storage algorithm doesn't create objects proportional to O(n) directory nesting depth as Xet appears to. (Xet is very much like Git in that respect.)
The data versioning system enables us to run pipelines based on changes to your data; the pipelines declare what files they read, and that allows us to schedule processing jobs that only reprocess new or changed data, while still giving you a full view of what "would" have happened if all the data had been reprocessed. This, to me, is the key advantage of data versioning; you can save hundreds of thousands of dollars on compute. Being able to undo an oopsie is just icing on the cake.
Xet's system for mounting a remote repo as a filesystem is a good idea. We do that too :)
- ylow 4y agoBy the way, our mount mechanism has one very interesting novelty. It does not depend on a FUSE driver on Mac :-)
- jrockway 4y agoThat's smart! I think users have to install a kext still?
- ylow 4y agoNope. No kernel driver needed :-) We wrote an localhost NFS server.
- catiopatio 4y agoBased on unfsd or entirely in-house?
- ylow 4y agoEntirely in house. In Rust!
- catiopatio 4y agoFancy! That’s awesome.
- ArchOversight 4y agoIs this available somewhere open source? This sounds amazing!
- ylow 4y agoWe have found pointer files to be surprisingly efficient as long as you don't have to actually materialize those files. (Git's internals actually very well done). Our mount mechanism does avoid materializing pointer files which makes it pretty fast even for repos with very large number of files.
- unqueued 4y agoFor bigger annex repos with lots of pointer files, I just disable the git-annex smudge filters. Consider whether smudge filters are requirement, or a convenience. The smudge filter interface does not scale that well at all.
- reverius42 4y agoThey're not a requirement! git-xet has a --no-smudge option if you prefer to deal with an unsmudged repo.
- chubot 4y agoIs DVC useful/efficient at storing container images (Docker)? As far as I remember they are just compressed tar files. Does the compression defeat its chunking / differential compression? How about cleaning up old versions?
- ilyt 4y ago> The data versioning system enables us to run pipelines based on changes to your data; the pipelines declare what files they read, and that allows us to schedule processing jobs that only reprocess new or changed data, while still giving you a full view of what "would" have happened if all the data had been reprocessed. This, to me, is the key advantage of data versioning; you can save hundreds of thousands of dollars on compute. Being able to undo an oopsie is just icing on the cake. ...isn't that just parsing git diff --name-only A..B tho ? "Process only files that changed since last commit" is extremely simple problem to solve.
- jrockway 4y agoYeah, that's the basics of it. Just make sure all the output is atomic, scale up the workers, handle inputs that are joins, retry when the workers get rescheduled, etc.