3 ms·
I'd like to see a version of this built into things like IPFS. It seems obvious that whenever something is saved into IPFS, there might be a similar object alr
by londons_explore 5y ago
I'd like to see a version of this built into things like IPFS.
It seems obvious that whenever something is saved into IPFS, there might be a similar object already stored. If there is, go make a diff, and only store the diff.
- hcs 5y agoIt should be possible to do this in IPFS already if you use the go-ipfs --chunker option with a content-sensitive chunking algorithm like rabin or buzhash [1]. With this there's a good chance that a file with small changes from something already on IPFS will have some chunks that hash identically, so they'll be shared. [1] https://en.wikipedia.org/wiki/Rolling_hash#Content-based_slicing_using_a_rolling_hash https://en.wikipedia.org/wiki/Rolling_hash#Content-based_sli...
- londons_explore 5y agoBut that isn't quite as good as something like this that can 'understand' diffs in files, rather than simply relying on the fact a bunch of bytes in a row might be the same.
- hcs 5y agoI don't think elfshaker actually does do any binary diffing (e.g. xdelta or bsdiff). It works well because it uses pre-link objects which are built to change as little as possible between versions. Then when it compresses similar files together in a pack, Zstandard can recognize the trivial repeats.
- peterwaller-arm 5y agoAuthor here. This is correct, we set out to do binary diffing but we soon discovered that if you put similar enough object files together in a stream, and then compress the stream, zstandard does a fantastic job at compressing and decompressing quickly with a high compression ratio. The existing binary diffing tools can produce small patches, but they are relatively expensive both to compute the delta and to apply the patches.