4 ms·

Does H2 feature data integrity (checksums etc)? For me that is one of the best features of ZFS

by blue1 9y ago

mjevans 9y ago
It's a short email post. Yes, they're build in to the inode structure and would be used for de-duplication. (However I'd hope that for a production system there's still a /verify/ pass on the data being de-duped to confirm it actually __is__ a dupe.)
- mrighele 9y ago
  Interestingly, this is not what happens in ZFS [1] (unless the defaults have changed since a few years ago): If you accept the mathematical claim that a secure hash like SHA256 has only a 2\^-256 probability of producing the same output given two different inputs, then it is reasonable to assume that when two blocks have the same checksum, they are in fact the same block. You can trust the hash. An enormous amount of the world's commerce operates on this assumption, including your daily credit card transactions. However, if this makes you uneasy, that's OK: ZFS provies a 'verify' option that performs a full comparison of every incoming block with any alleged duplicate to ensure that they really are the same, and ZFS resolves the conflict if not. To enable this variant of dedup, just specify 'verify' instead of 'on': [1] https://blogs.oracle.com/bonwick/zfs-deduplication-v2 https://blogs.oracle.com/bonwick/zfs-deduplication-v2
  - loeg 9y ago
    And HAMMER2 will use even larger 512 bit hashes for blind dedupe, making the possibility of a collision causing data loss vanishingly small.
    - Dylan16807 9y ago
      Bah, 256 bits is already so vanishingly small it's hard to comprehend. If you put a thousand disks into a pool you might reach 2^40 blocks, leaving you with 256-80=176 bits of margin. That is never going to collide. You could make such a filesystem for each atom on Earth and the odds of a single collision would be less than 0.1% You could put a billion disks in a pool and still have 136 bits of margin. 512 bit hashing is basically placebo.
      - loeg 9y ago
        I agree that 256 bit hashing is totally sufficient. I wouldn't call 512 bits "placebo," though. Just overkill.
        Dylan16807 9y ago
        You hit a point where the longer runtime increases your chance of error by a very small amount that is nevertheless larger than the protection you gain.
  - mjevans 9y ago
    Let me describe a hypothetical attack. E knows of some critical security update that needs to be installed in sensitive locations. E also knows of some attack on the hashing algorithm that is in use by the filesystem to craft a small block containing mostly garbage but some key bits that they would like to control. (Yes this is the hypothetical, but prior algorithms /have/ fallen). E thus arranges to have this 'duplicate' block stored before routine and predictable maintenance patterns. A installs the updates and the 'duplicate' file is now E's datastream, but A's intended credentials. E has caused system corruption, and potentially privilege escalation.

Does H2 feature data integrity (checksums etc)? For me that is one of the best features of ZFS

by blue1 9y ago

Does H2 feature data integrity (checksums etc)? For me that is one of the best features of ZFS

It's a short email post. Yes, they're build in to the inode structure and would be used for de-duplication. (However I'd hope that for a production system there's still a /verify/ pass on the data being de-duped to confirm it actually __is__ a dupe.)

mrighele 9y ago
Interestingly, this is not what happens in ZFS [1] (unless the defaults have changed since a few years ago): If you accept the mathematical claim that a secure hash like SHA256 has only a 2\^-256 probability of producing the same output given two different inputs, then it is reasonable to assume that when two blocks have the same checksum, they are in fact the same block. You can trust the hash. An enormous amount of the world's commerce operates on this assumption, including your daily credit card transactions. However, if this makes you uneasy, that's OK: ZFS provies a 'verify' option that performs a full comparison of every incoming block with any alleged duplicate to ensure that they really are the same, and ZFS resolves the conflict if not. To enable this variant of dedup, just specify 'verify' instead of 'on': [1] https://blogs.oracle.com/bonwick/zfs-deduplication-v2 https://blogs.oracle.com/bonwick/zfs-deduplication-v2
- loeg 9y ago
  And HAMMER2 will use even larger 512 bit hashes for blind dedupe, making the possibility of a collision causing data loss vanishingly small.
  - Dylan16807 9y ago
    Bah, 256 bits is already so vanishingly small it's hard to comprehend. If you put a thousand disks into a pool you might reach 2^40 blocks, leaving you with 256-80=176 bits of margin. That is never going to collide. You could make such a filesystem for each atom on Earth and the odds of a single collision would be less than 0.1% You could put a billion disks in a pool and still have 136 bits of margin. 512 bit hashing is basically placebo.
    - loeg 9y ago
      I agree that 256 bit hashing is totally sufficient. I wouldn't call 512 bits "placebo," though. Just overkill.
      - Dylan16807 9y ago
        You hit a point where the longer runtime increases your chance of error by a very small amount that is nevertheless larger than the protection you gain.
- mjevans 9y ago
  Let me describe a hypothetical attack. E knows of some critical security update that needs to be installed in sensitive locations. E also knows of some attack on the hashing algorithm that is in use by the filesystem to craft a small block containing mostly garbage but some key bits that they would like to control. (Yes this is the hypothetical, but prior algorithms /have/ fallen). E thus arranges to have this 'duplicate' block stored before routine and predictable maintenance patterns. A installs the updates and the 'duplicate' file is now E's datastream, but A's intended credentials. E has caused system corruption, and potentially privilege escalation.