5 ms·
>And de-dupe. This is not a contrary aim from the compression, anonymity and encryption bit. From a technical point, they largely are contrary aims.
by drivebyacct2 15y ago
>And de-dupe. This is not a contrary aim from the compression, anonymity and encryption bit.
From a technical point, they largely are contrary aims.
- willvarfar 15y agoThe server need not be party to the key that is used to exchange the filenames themselves between the clients. The content can be compressed in blocks before upload; dynamic, anchored block sizes would increase de-dup. The compressed blocks can be individually encrypted with a key derived from the filename and block. There's still plenty of opportunity for whole-file de-dupe there.
- keyist 15y agoTarsnap gives deduplication, compression, and encryption. Some technical background on how it does this here: http://www.tarsnap.com/efficiency.html http://www.tarsnap.com/efficiency.html There is a detailed breakdown provided as well. Sample output: Total size Compressed size All archives 697 MB 323 MB (unique data) 215 MB 100 MB This archive 148 MB 67 MB New data 17 MB 5.2 MB Edit: Don't want to misrepresent Tarsnap's granularity. The print-stats option gives size in bytes. Above output was generated with the --humanize-numbers option.
- nknight 15y agoI assume this is only deduplication within a single account? Because I don't see anything there on how one would provide dedupe across crypto domains (which, to the best of my knowledge, should be theoretically impossible if the crypto is done right).
- cperciva 15y agoCorrect.
- pbhjpbhj 15y agoJust been looking at Tarsnap. It looks good but I'm curious what happens if the author/creator/owner/maintainer (singular) gets run over by a bus?
- cperciva 15y agoI do my best to avoid busses. (I've always been more of a fan of switched point-to-point connections than busses, to tell the truth.) In all seriousness, if I get hit by a bus Tarsnap probably won't live on without me -- but it runs itself smoothly enough that there should be plenty of time for people to download their data.
- cperciva 15y agoMore sample output, from a heftier machine, in case anyone is curious: Total size Compressed size All archives 3321059166180 1113881916696 (unique data) 116996664943 27182261608 This archive 13831036740 4855451330 New data 336950399 62343192 (Yes, that's 3.3 TB deduplicated and compressed down to 27 GB.)
- plinio_silva 15y agoCan I automatically sync files using it? I work in multiple places and if I have to manually run a sync operation before I leave, I'm certain to forget or be unable to at some occasion, and then I'm screwed.
- cperciva 15y agoTarsnap doesn't do sync. Doing sync while maintaining security is a much harder problem: Tarsnap's design requires each system to know what's on the server before uploading or deleting blocks, and that breaks if you have multiple systems accessing the same storage space.
- JoachimSchipper 15y agoYou can deduplicate within accounts; cross-account deduplication is indeed a problem.