4 ms·
> already working in dat (https://datprotocol.github.io/how-dat-works https://datprotocol.github.io/how-dat-works) I just looked over the doc at https://datpro
by oconnor663 7y ago
> already working in dat (https://datprotocol.github.io/how-dat-works https://datprotocol.github.io/how-dat-works)
I just looked over the doc at https://datprotocol.github.io/how-dat-works https://datprotocol.github.io/how-dat-works, particularly the section on the tree hash. It looks pretty reasonable, and definitely similar to BLAKE3.
I think one practical downside of the Dat approach is that root finalization is always done as a separate hashing step. So for example, if I hash the short string "hello world", I have to compute one chunk hash and then one root hash, for a total of two calls to the compression function. Compare that to BLAKE3, which will only do a single compression there, by setting both the chunk flags and the root flag for that one block. This difference is probably totally negligible for Dat's intended use case, where network round trip time is going to completely dwarf hashing time for small inputs. But for a general-purpose hash function you don't want to cut your small input performance in half.
A bigger philosophical difference is that Dat doesn't specify a particular chunk size. Again this makes sense for their use case, since you might want to experiment with fancy things like Rabin fingerprinting. Looks like they discuss that here: https://stackoverflow.com/a/51604110/823869 https://stackoverflow.com/a/51604110/823869. But for a cryptographic hash, it's important that every file has exactly one hash, and a fixed chunk size is required. Another upside of a fixed chunk size is that you really cut down on metadata; the 8-byte length at the front of an encoding or a slice determines the structure of the entire tree.
- rakoo 7y agoIndeed content-based chunking definitely has appeal, I'm thinking about all the backup softwares that use it as a smart way to cut down on total size in a way that leaves all snapshots independent from each other. On the other hand having a well-defined format and the tools to play with like you provide is definitely better to have it spread, so kudos for it.