4 ms·
git itself should have been rolling hash chunk based storage. Storing very large text files (like bash history or the Debian security team git repo) in git is v
by pabs3 5y ago
git itself should have been rolling hash chunk based storage. Storing very large text files (like bash history or the Debian security team git repo) in git is very inefficient due to the current design.
- beagle3 5y agoIt is inefficient up until the point where you ask git to do a gc+repack, at which point git will look for deltas - and if it is changes to the same file, it will likely find them and encode them as a delta. For text files with really small changes, this is comparable to a "diff" size, and is better than what a rolling hash would achieve. However, for text files with larger changes, and of course for binary files - a rolling hash is much more effective. Additionally, a rolling hash would easily reuse cross-file similarity, where as the current delta-finding code is likely to only find redundancy in the history of the same file (or similarly named in the same directory).