3 ms·

Thanks for the feedback! I will look into BNTs. > each file is hashed with a random nonce, so as to distinguish files with identical contents Actually, the pu

Thanks for the feedback! I will look into BNTs. > each file is hashed with a random nonce, so as to distinguish files with identical contents Actually, the purpose of the nonce is for privacy. The inclusion proof necessarily contains a leaf node hash, but I didn't want the proof to expose the hash of files it wasn't proving. That's because it could be private information, such as "bob has sensitive file X downloaded". Since this approach makes determinism impossible, I chose to ignore the goroutine determinism part you mention. Storing the tree nodes would likely still be required however, since otherwise the system would be brittle against things like file content and name changes, deletion, etc.

nemo1618 3y ago
Ah, that makes sense. Still, you could use a single nonce per directory, plus an index per file. As for brittleness -- some degree is unavoidable, since if you want to provide an inclusion proof for a file, you need to provide the (unchanged) file itself. I kinda assumed that the directory would be treated as immutable, but maybe that was premature since no one seems to have landed on the exact use case for this tool yet. :P Oh, also -- since a BLAKE3 hash is itself a Merkle root, you could technically extend your proofs down into the files themselves; that is, you could prove that some 1024-byte slice of data was part of a file which was part of the directory. The blake3 package doesn't (currently) provide an API for that, though.
- oconnor663 3y ago
  The BLAKE3 tree structure is designed to give exactly one tree layout for a given file length, so there's not really any flexibility to represent application data directly in the tree structure. You'd need to somehow transform your tree into a flat array of bytes and hash that.
da39a3ee 3y ago
It would be useful to allow the option for determinism (with its security downside) e.g. for people who want to assert that a directory contents hasn't changed? Or is there a simpler/standard way to get a hash of all content under a given directory? Something like fd | sort | xargs cat | md5sum, but including file names.

nemo1618 3y ago

Ah, that makes sense. Still, you could use a single nonce per directory, plus an index per file. As for brittleness -- some degree is unavoidable, since if you want to provide an inclusion proof for a file, you need to provide the (unchanged) file itself. I kinda assumed that the directory would be treated as immutable, but maybe that was premature since no one seems to have landed on the exact use case for this tool yet. :P Oh, also -- since a BLAKE3 hash is itself a Merkle root, you could technically extend your proofs down into the files themselves; that is, you could prove that some 1024-byte slice of data was part of a file which was part of the directory. The blake3 package doesn't (currently) provide an API for that, though.

oconnor663 3y ago
The BLAKE3 tree structure is designed to give exactly one tree layout for a given file length, so there's not really any flexibility to represent application data directly in the tree structure. You'd need to somehow transform your tree into a flat array of bytes and hash that.

da39a3ee 3y ago

It would be useful to allow the option for determinism (with its security downside) e.g. for people who want to assert that a directory contents hasn't changed? Or is there a simpler/standard way to get a hash of all content under a given directory? Something like fd | sort | xargs cat | md5sum, but including file names.