3 ms·

Is this functionally different than sending the hash of a block of bytes when sending a block of bytes? > With a serial hash, the recipient would need to downl

by ebg13 7y ago

Is this functionally different than sending the hash of a block of bytes when sending a block of bytes? > With a serial hash, the recipient would need to download the entire attachment to verify it Rings to me as being trivially untrue, since, if the file is divided into chunks, you can just verify each chunk separately. Alternately, can this be used to generate the complete file hash from chunks received out of order by a download using multiple parallel substreams without writing all of them to a block device in order first?

rfk 7y ago
There are plenty of mischievous things an attacker could do if you just verify each chunk separately, including reordering or omitting chunks, and truncating the file.
- ebg13 7y ago
  An attacker who can do those things could send you a different initial checksum as well. How does this solve that?
  - rfk 7y ago
    Any sort of verification presumes that you have an initial trusted checksum against which to verify, so I don't believe this solves the separate problem of obtaining such a checksum. IIUC your question here boils down to "what is the point of a hash tree?" as opposed to e.g. a list of individual chunk hashes. The answer is that a hash tree lets you verify an individual chunk by looking at the hashes of O(log(N)) chunks rather than having to look at the hash of every chunk.
    - ebg13 7y ago
      Thanks. That is meaningful, but I still don't understand how "the recipient can stream a video attachment, while still verifying each byte as it comes in" isn't basically also true for sequential hashing with periodic chunk hashes.
oconnor663 7y ago
The idea is that you have the hash of some file, and you trust that hash somehow. (For example, maybe you stored it locally in the past, or someone important has signed it, or it's included in the Bitcoin blockchain.) But now you want to fetch the contents of the file from some peer you don't trust. If you just hash each chunk separately, you have a storage problem. Let's say you have a 4 GiB file, about the size of a DVD, and you use 1 KiB chunks like BLAKE3 does. The set of chunk hashes is 128 MiB. You can't store that much data in any blockchain, so maybe you hash it again and store the meta-hash of all the hashes instead (basically a custom 1-level tree mode). Then when it's time to verify, you fetch the set of hashes from the peer as a first step. This is all doable, but now you have to download 128 MiB of chunk hashes before you can verify a single byte. You can try to use a larger chunk size to reduce this overhead, but that increases overhead elsewhere.

by ebg13 7y ago

rfk 7y ago

There are plenty of mischievous things an attacker could do if you just verify each chunk separately, including reordering or omitting chunks, and truncating the file.

ebg13 7y ago
An attacker who can do those things could send you a different initial checksum as well. How does this solve that?
- rfk 7y ago
  Any sort of verification presumes that you have an initial trusted checksum against which to verify, so I don't believe this solves the separate problem of obtaining such a checksum. IIUC your question here boils down to "what is the point of a hash tree?" as opposed to e.g. a list of individual chunk hashes. The answer is that a hash tree lets you verify an individual chunk by looking at the hashes of O(log(N)) chunks rather than having to look at the hash of every chunk.
  - ebg13 7y ago
    Thanks. That is meaningful, but I still don't understand how "the recipient can stream a video attachment, while still verifying each byte as it comes in" isn't basically also true for sequential hashing with periodic chunk hashes.

oconnor663 7y ago

The idea is that you have the hash of some file, and you trust that hash somehow. (For example, maybe you stored it locally in the past, or someone important has signed it, or it's included in the Bitcoin blockchain.) But now you want to fetch the contents of the file from some peer you don't trust. If you just hash each chunk separately, you have a storage problem. Let's say you have a 4 GiB file, about the size of a DVD, and you use 1 KiB chunks like BLAKE3 does. The set of chunk hashes is 128 MiB. You can't store that much data in any blockchain, so maybe you hash it again and store the meta-hash of all the hashes instead (basically a custom 1-level tree mode). Then when it's time to verify, you fetch the set of hashes from the peer as a first step. This is all doable, but now you have to download 128 MiB of chunk hashes before you can verify a single byte. You can try to use a larger chunk size to reduce this overhead, but that increases overhead elsewhere.