3 ms·
If you read the internals description it could just as well be about Borg, very similar principles here, though the application is very different. By the way,
by dom0 9y ago
If you read the internals description it could just as well be about Borg, very similar principles here, though the application is very different.
By the way, both buzhash and SHA-256 are kinda poor choices for a new system, especially one that targets servers.
- rdtsc 9y agoYap borg is the first thing I thought of. It already does a lot of this an more: encryption, configurable encoding, a rolling hash computed by the Buzhash algorithm and so on. Maybe it wasn't geared for CDN delivery during restores but otherwise I've been impressed by borg so far (haven't deployed it in production, only played with it locally though). https://github.com/borgbackup/borg https://github.com/borgbackup/borg This is a description of the internal design: http://borgbackup.readthedocs.io/en/stable/internals.html http://borgbackup.readthedocs.io/en/stable/internals.html
- dom0 9y agoThe "latest" version of that page has seen significant additions: http://borgbackup.readthedocs.io/en/latest/internals.html http://borgbackup.readthedocs.io/en/latest/internals.html
- rdtsc 9y agoThanks. That is a better reference indeed. It's got nice diagrams as well. Can't edit my post any longer, so hopefully others will just see your message.
- dchest 9y agoboth buzhash and SHA-256 are kinda poor choices for a new system Why?
- dom0 9y agoLow software performance
- tomfitz 9y agoSHA-256 seems to score well on https://www.cryptopp.com/benchmarks.html https://www.cryptopp.com/benchmarks.html .
- ktta 9y agoDepends on the context. If there is a lot of hashing, then a faster alternative like BLAKE2b is better.
- dchest 9y agoConsidering that it uses xz for compression, does the performance of SHA-256 matter? (Well, using faster hash function can speed up finding duplicate blocks, which were already packed.) I'm more interested to hear about buzhash, though.
- dom0 9y agoI assume™ that xz won't stay the only choice. I think it's important to understand that in deduplication, you'll pass all data through your hashes one to two times. Regarding buzhash, it can break with byte granularity, and it has a dependency chain that prohibits parallelization. You'll likely never see it go faster than 700-750 MB/s on a desktop CPU (~3.8 GHz Haswell) and it won't profit from non-clock improvements of CPUs. Giving up byte-granularity allows significant improvements in performance, but I don't think anyone comprehensively analysed the impact on deduplication performance. I didn't. (OTOH if your storage is faster than ~200-300 MB/s (buzhash and a hash, naively combined) then there is likely no issue using higher degrees of I/O concurrency, so you can work around these problems).
- dchest 9y agoThanks for explanation. Do you know of any implementations?