3 ms·

Just because a file is encrypted doesn't mean that a small block doesn't match another encrypted block even if the two are from two different files. This would

Just because a file is encrypted doesn't mean that a small block doesn't match another encrypted block even if the two are from two different files. This would allow them to do deduplication at the block level (see ZFS for example).

rayiner 14y ago
Think about that one for a second. An encrypted block is essentially supposed to look like random data. I.e. if two people encrypt the same file with different keys, you shouldn't be able to tell that they're the same file (or your encryption sucks). So your block-level de-duping then depends on incidental matches between random data. What's the probability of two 4KB (or whatever) blocks of random data being identical? Basically zero even with petabytes of data.
- harryh 14y ago
  see: convergent encryption The encryption on the client doesn't use a random key. The key is a hash of the unencrypted contents of the file.
  - rayiner 14y ago
    Reading list-ed. Interesting.
    - harryh 14y ago
      The issue is that you don't get the full benefits of encryption. If you upload the map to the rayiner family treasure that only you have seen you're good. No one else will be able to read it. But if you upload the latest episode of Modern Family and Disney gets ahold of the same rip you used they (if they can get a government to help them out) can see what you did and charge you with copyright infringement (or whatever the appropriate crime would be).
      - tptacek 14y ago
        After 3-4 years of high-profile CPA-2 attacks on TLS, .NET, Java, and other systems, you'd think we'd all be a lot more skeeved out about cryptosystems that demand known-plaintexts. There's already an obvious conceptual attack (beyond file confirmation) in naive "convergent encryption", which is that you can leverage small amounts of known plaintext to learn unknown plaintext.
nwh 14y ago
In practise any strong encryption (ie, not AES-ECB) is indistinguishable from random noise, and that's by design. Even trying to de-duplicate 4kb blocks of random noise would be a completely fruitless task. If it was possible, storage is probably cheaper than the CPU time to find similar blocks.

rayiner 14y ago

Think about that one for a second. An encrypted block is essentially supposed to look like random data. I.e. if two people encrypt the same file with different keys, you shouldn't be able to tell that they're the same file (or your encryption sucks). So your block-level de-duping then depends on incidental matches between random data. What's the probability of two 4KB (or whatever) blocks of random data being identical? Basically zero even with petabytes of data.

harryh 14y ago
see: convergent encryption The encryption on the client doesn't use a random key. The key is a hash of the unencrypted contents of the file.
- rayiner 14y ago
  Reading list-ed. Interesting.
  - harryh 14y ago
    The issue is that you don't get the full benefits of encryption. If you upload the map to the rayiner family treasure that only you have seen you're good. No one else will be able to read it. But if you upload the latest episode of Modern Family and Disney gets ahold of the same rip you used they (if they can get a government to help them out) can see what you did and charge you with copyright infringement (or whatever the appropriate crime would be).
    - tptacek 14y ago
      After 3-4 years of high-profile CPA-2 attacks on TLS, .NET, Java, and other systems, you'd think we'd all be a lot more skeeved out about cryptosystems that demand known-plaintexts. There's already an obvious conceptual attack (beyond file confirmation) in naive "convergent encryption", which is that you can leverage small amounts of known plaintext to learn unknown plaintext.

nwh 14y ago

In practise any strong encryption (ie, not AES-ECB) is indistinguishable from random noise, and that's by design. Even trying to de-duplicate 4kb blocks of random noise would be a completely fruitless task. If it was possible, storage is probably cheaper than the CPU time to find similar blocks.