3 ms·
desync is what git-lfs should have been: rolling hash chunk based storage, with local caching of chunks, proxying of remote chunks, etc. When I have big 100MB
by beagle3 5y ago
desync is what git-lfs should have been: rolling hash chunk based storage, with local caching of chunks, proxying of remote chunks, etc.
When I have big 100MB binary files that i want to version, the changes are small (1MB in one place, and a few more KB in others). I also have a few multi GB SQLite databases I would like to version where this would help. (Changes are less than 5% of the data, but index pages throughout the file mean rsync-style partitions still transfer 50% or so of the file. To actually achieve storage efficiency, I store textual dumps, and they also fit better with borg/restic/casync than git or git-lfs
Would have been extra awesome if desync would be able to use a git repo as storage.
- pabs3 5y agogit itself should have been rolling hash chunk based storage. Storing very large text files (like bash history or the Debian security team git repo) in git is very inefficient due to the current design.
- beagle3 5y agoIt is inefficient up until the point where you ask git to do a gc+repack, at which point git will look for deltas - and if it is changes to the same file, it will likely find them and encode them as a delta. For text files with really small changes, this is comparable to a "diff" size, and is better than what a rolling hash would achieve. However, for text files with larger changes, and of course for binary files - a rolling hash is much more effective. Additionally, a rolling hash would easily reuse cross-file similarity, where as the current delta-finding code is likely to only find redundancy in the history of the same file (or similarly named in the same directory).