3 ms·

The hell it can't. Block-level de-duplication can easily work for encrypted data.

• deleted 15y ago
  [deleted]
  • count 15y ago
    While a different password / key would make the same data decrypt to different data, you don't necessarily have to do de-dupe on blocks of the encryption-cipher-size, so these values can easily overlap. Blocks of data can be treated completely independently of the encryption. The block size is not necessarily the same as the encryption cipher block size (if you're using a block cipher). Say, a chunk of 4000 different files all have a few disk blocks that contain the same pattern of data. You can store those few blocks once, instead of 4000 times. With encryption, this is still possible, but it's going to be slower, as the data segments will be more randomly distributed. It is still, however, possible and will result in savings.
    • esutton 15y ago
      the problem with this theory is that deduplication works so well with file backup as the largest files we have are most often music and video, something that is highly replicated amongst file sharing users. i.e millions of users have the same Beatles album. Yet, when each persons album is encrypted it will be entirely different than every other persons album. If they were in any way similar it was not correctly encrypted. Thus the hit rate will be almost negligible. So than you say OK, but maybe the block level code of the encryption of my Beatles album is similar to the block level code of the encryption of my Eminem album, and than we can save storage there. The problem with this is that encryption algorithms aim to make their input files output as close to a random distribution as possible. Meaning not only are those two copies of the beetles albums going to be as different from each other, but they will be different from any other file. To sum it up, a well encrypted file should be uniformly random -> meaning it is uncompressible - > two different uniformly random files are still random relative to each other and when put together still cannot be compressed.
      • indrax 15y ago
        The easy solution is to segment easily deduplicated data from 'private' data. Perhaps sufficiently deduplicated data doesn't count against your quota. (as amazon does with music)
        • evgen 15y ago
          The problem with this is that the data which you want to de-dupe are the big files that everyone has a copy of; files like mp3s, videos, etc. Unfortunately these are the same files that users are most likely to not want the RIAA/MPAA to know about and keep private. The short version of all of this is that Bitcasa can't actually pull off what they are claiming and still maintain any user security/privacy.
      • count 15y ago
        At the 'infinite' scale of storage that they're discussing, you should still see a savings. It's not going to be 30% or 80%, but it will be savings. De-duplication CAN still take place.
        • esutton 15y ago
          actually it should be close to nothing, encrypted data cannot be compressed as it is supposed to be uniformly random. If it can be compressed, the encryption algorithm is weak. also on stage at tech crunch disrupt the founders were talking about the majority of the data be deduplicated