4 ms·

Only if blocks have the same offset in the same binary and if they align with the block boundaries. Otherwise, different hashes would be generated. I don't expe

• naikrovek 6y ago
  Well if files in an ISO are aligned to some boundary it would help a lot, similar to how filesystems on disk have a sector size, where all files begin at the start of a sector. However, I don't know if this is true in ISO9660 or any of its extensions.
  • em-bee 6y ago
    ISO is a mountable filesystem so this could be the case if it doesn't have any space saving optimizations that create variable block sizes. also need to watch out for compression
• computerphage 6y ago
  It will still definitely help some use cases: > Identical files will always have the same hash and can more easily be moved from one torrent to another (when creating torrents) without having to re-hash anything.
• aurelian15 6y ago
  There are ways around this. See "content-aware chunking", e.g. implemented using rolling hashes [1]. This is for example what rsync does. The idea is to make blocks (slightly) variable in size. Block boundaries are determined based on a limited window of preceding bytes. This way a change in one location will only have a limited impact on the following blocks. [1] https://en.wikipedia.org/wiki/Rolling_hash https://en.wikipedia.org/wiki/Rolling_hash
  • octoberfranklin 6y ago
    Rolling hashing is really only useful for finding nonaligned duplicates. There isn't a way to advertise some "rolling hash value" in a way that allows other people with a differently-aligned copy to notice that you and them have some duplicated byte ranges. Rolling hashes only work when one person (or two people engaged in a conversation, like rsync) already has both copies.
    • btschaegg 6y ago
      I think you misunderstood how the rolling hash is used in this context. It's not used to address a chunk; you'd use a plain old cryptographic hash function for that. The rolling hash is used to find the chunk boundary: Hash a window before every byte (which is cheap with a rolling hash) and compare it against a defined bit mask. For example: Check if the first 20 bytes are zero. If so, you'd get chunks with about 2^20 bytes (1 MiB) average length. As a good explanation, I'd encourage you to look at borgbackup's internals documentation: https://borgbackup.readthedocs.io/en/stable/internals.html https://borgbackup.readthedocs.io/en/stable/internals.html
      • hinkley 6y ago
        I think they understood just fine. If I discover that the file I want to publish shares a range with an existing file, that does very little because the existing file has already chosen its chunk boundaries and I can’t influence those. That ship has sailed. I can only benefit if the a priori chunks are small enough that some subset of the identified match is still addressable. And then I may only get half of a two thirds of the improvement I was after.
        • tleb_ 6y ago
          The idea is that on none random data, you are able to use a heuristic that would create variable-sized chunks that fit the data. The simplest way seems to detect padding zeros and start a new block on the first following none zero byte. There probably are other ways, knowing the data type should help.