4 ms·

I dont get it, why is deduping at a chunk level important, can't it just take it as a tradeoff and dedupe at the file level, or does that have the same problems

by lugg 12y ago

I dont get it, why is deduping at a chunk level important, can't it just take it as a tradeoff and dedupe at the file level, or does that have the same problems? And why exactly is deduping important for a personal use case, bandwidth is relatively cheap. In any case, while I want security in these systems to avoid mass survey style attacks, I assume anything I put over the wire to be compromised at least potentially. I keep the kind of files that need that level of encryption on encrypted flash drives.

enscr 12y ago
If you are encrypting each file individually, you are still giving away the file names. Hence you use encrypted file containers. They benefit from block level deduplication. They can be large in size.
- icebraining 12y ago
  If you are encrypting each file individually, you are still giving away the file names. Hence you use encrypted file containers. That doesn't follow - you can solve the problem without containers, e.g. by encrypting or hashing the filenames. Containers protect the file sizes, but they lose the ability to selectively sync certain files to certain nodes.
  - enscr 12y ago
    Hashing doesn't solve the problem completely. You are still giving away information about file count, size & hierarchy. Regarding selective sync : you'll have to organize the files logically into containers. I think even Dropbox doesn't let you sync files selectively, you have to have content separated into folders. I'm not saying that one is better than other, just giving an example of where de-duping at block level is needed as @lugg asked.
    - icebraining 12y ago
      Hashing doesn't solve the problem completely. You are still giving away information about file count, size & hierarchy. Sure; I was just talking about the file names. You can avoid the hierarchy part by transmitting it separately from the files in a secure way, though. Regarding selective sync : you'll have to organize the files logically into containers. I think even Dropbox doesn't let you sync files selectively, you have to have content separated into folders. git-annex does. You can implement rules to decide what files should be synced, which should be dropped locally, etc. For example, an MP3 player running it might have a rule such as filename(*.mp3) or filename(*.ogg) and it'll only sync those files, regardless of their directories.

by lugg 12y ago

enscr 12y ago

If you are encrypting each file individually, you are still giving away the file names. Hence you use encrypted file containers. They benefit from block level deduplication. They can be large in size.

icebraining 12y ago
If you are encrypting each file individually, you are still giving away the file names. Hence you use encrypted file containers. That doesn't follow - you can solve the problem without containers, e.g. by encrypting or hashing the filenames. Containers protect the file sizes, but they lose the ability to selectively sync certain files to certain nodes.
- enscr 12y ago
  Hashing doesn't solve the problem completely. You are still giving away information about file count, size & hierarchy. Regarding selective sync : you'll have to organize the files logically into containers. I think even Dropbox doesn't let you sync files selectively, you have to have content separated into folders. I'm not saying that one is better than other, just giving an example of where de-duping at block level is needed as @lugg asked.
  - icebraining 12y ago
    Hashing doesn't solve the problem completely. You are still giving away information about file count, size & hierarchy. Sure; I was just talking about the file names. You can avoid the hierarchy part by transmitting it separately from the files in a secure way, though. Regarding selective sync : you'll have to organize the files logically into containers. I think even Dropbox doesn't let you sync files selectively, you have to have content separated into folders. git-annex does. You can implement rules to decide what files should be synced, which should be dropped locally, etc. For example, an MP3 player running it might have a rule such as filename(*.mp3) or filename(*.ogg) and it'll only sync those files, regardless of their directories.