7 ms·
How Dropbox sacrifices user privacy for cost savings
- bayes 16y agoThis makes me worried about hash collisions as well. The article implies that a file whose hash matches something they already have will never even reach their servers - so presumably I just have to keep my fingers crossed that the file they're synchronising to all my machines is the one I uploaded, and not some other user's completely different file that happens to have the same hash?
- schwanksta 16y agoAt first, that was what I thought the flaw would be -- providing a file that has a hash that collides with another file, gets you that file. But it seems to me you would need to know the exact contents of the file in question to get that to happen, making the point moot. Perhaps I'm wrong on that.
- avdempsey 15y agoDo they check filesize too? What are the odds of a hash collision + identical filesize? We might need Carl Sagan to answer that one.
- bdhe 15y ago> What are the odds of a hash collision + identical filesize? If implemented correctly, the additional constraint on filesize being the same is irrelevant. Given one particular hash value, the probability that a second file hashes to the same value is 1/(range of hash function) if the hash function is modeled as an ideal hash function.
- Groxx 15y agoGiven that they do chunks of updates, rather than the whole file, presumably they check multiple hashes to ensure this doesn't happen. I could be wrong, though.
- thomaswmeyer 15y agoSHA256, which Dropbox uses, has around 10^77 possible hashes. That's 100 trillion quadrillion quadrillion quadrillion quadrillion possible values. So I wouldn't worry about hash collisions if I were you. If a hash collision happened easily in SHA256, that would be very big news for the security community, and much more serious services than Dropbox would be affected.
- ceejayoz 15y agoDeduping with hash and filesize in bytes should make a collision unlikely enough.
- sukuriant 15y agoBut still possible. Whenever the number of bits in a file is more than the number of bits in a hash, there are collisions. They could use the Chinese Remainder Theorem, but that would only go so far (maybe far enough to remove substantial doubt? The link below seems to suggest so.) Relavent to the discussion: http://stackoverflow.com/questions/622930/purposely-create-two-files-to-have-the-same-hash http://stackoverflow.com/questions/622930/purposely-create-t...
- gojomo 15y agoThis is a common sentiment, but not really sensible. If you want to store (for example) another 64 bits worth of information, you would always be better off with 64 more bits of some strong hash than 64 bits of filesize.
- slackerIII 15y agoI think the point is that creating a collision when data + fileSize is hashed is much harder than just creating a collision when hashing data alone. Raymond explains it well: http://blogs.msdn.com/b/oldnewthing/archive/2004/05/19/134937.aspx http://blogs.msdn.com/b/oldnewthing/archive/2004/05/19/13493...
- justincormack 15y agoAnd that was written before the md5 collisions were discovered. And no collision has yet been discovered for md5 for files of the same length, they are all extension attacks...
- gojomo 15y agoAbsolutely false. Some MD5 collision-generators specifically find pairs of equally-lengthed inputs with the same hash. See for example hit #2 for [MD5 collisions]: http://www.mscs.dal.ca/~selinger/md5collision/ http://www.mscs.dal.ca/~selinger/md5collision/ 'Extension attacks' are something else, which let you turn one collision into more, or create valid hashes for combinations of unknown text plus a chosen extension – not find an initial collision. See: http://en.wikipedia.org/wiki/Merkle%E2%80%93Damg%C3%A5rd_construction#Security_characteristics http://en.wikipedia.org/wiki/Merkle%E2%80%93Damg%C3%A5rd_con... The 'length extension' property can be helpful, once you find a collision based on 'random' nonsense, in extending that into two documents that are each meaningful-but-different and still colliding, as was done in this 2005 MD5 collision demonstration: http://replay.waybackmachine.org/20050612011328/http://www.cits.rub.de/MD5Collisions/ http://replay.waybackmachine.org/20050612011328/http://www.c...
- guan 16y agoOf course Dropbox knows the keys. If they didn’t, you wouldn’t be able to access your files on so many platforms (web, desktop, iPhone) and you wouldn’t be able to easily share folders with others. Even though it isn’t spelled out, I’ve always suspected that many actual backup services such as Backblaze don’t know the key if I decide to encrypt my backups.
- TomasSedovic 15y agoI don't think the "many clients" argument holds. If done properly, Dropbox would (deterministically) generate the keys from your username and password on the client every time you log in and encrypt/decrypt stuff there. You're right that sharing folders between different accounts would require having a key shared between the clients and thus stored on the Dropbox servers as well. However, they could generate the key for the shared folder, give it to you and your buddies and yet store it encrypted using your master key generated from your username and password. Then it would be accessible to you, but Drobox would not be able to decrypt it without getting your password. It looks like they're not doing that, but hypothetically, they could. And the examples you cite would not be impossible to deal with. Of course, this would require much more work, would be tricky to get right and they'd have Thomas Ptacek on their back for using JavaScript crypto in the browser.
- guan 15y agoYou are absolutely right in terms of what is technically possible. My point was more that it’s not really realistic to use crypto like that with a Dropbox-like service.
- varenc 15y agoWith pure file encryption where the user's password serves as the key you lose...password recovery features, public links for files, shared folders, web access, mobile access (unless you want your phone doing the decryption) All other syncing services do things a pretty similar way
- yoden 15y agoThis is wrong in several ways (as evidenced by other comments in the thread): > public links for files easily done by creating an unencrypted copy > shared folders you can do this by copying, encrypting with new keys for each shared user, and then sending notification of those keys to the user (via some side channel), and then deleting them on your end after they are accessed (you can re-encrypt with the accessor's keys at this point). You can do a bit better if you leave "half open" asymmetric channels (so you can store encrypted messages/keys only the recipient can decrypt), but that might be overkill. > web access you probably will need to use java (or NaCl) to do this, as javascript tends to be too slow to do asymmetric encryption (the the needed bignum support just isn't there). If you're willing to wait, it can be done in javascript in ~5 seconds on a desktop PC. > mobile access Uh, all modern smart phones can do encryption fast enough (well under a second). > password recovery features This is the only salient point. You can kind-of counter it by using the security questions to encrypt the actual encryption key a second time, so that the data can be decrypted if you answer the security questions. But that's obviously less secure.
- rafd 15y agoMy take-away is this: "What this means, is that from the comfort of their desks, law enforcement agencies or copyright trolls can upload contraband files to Dropbox, watch the amount of bandwidth consumed, and then obtain a court order if the amount of data transferred is smaller than the size of the file." This, I think is significant, especially if Dropbox is advertising security, privacy and encryption. As the author mentions, the ToS are being updated to reflect the above possibility ("if Dropbox receives a warrant, it has the ability to remove its own encryption to provide data to law enforcement").
- HerraBRE 15y agoOn the face of it, the security of DropBox encryption seems comparable to that of a padlocked room - where the key to the padlock is kept hidden in a safe place. Except the key is actually kept under the doormat, because people keep going in and out and keeping it elsewhere is just too inconvenient. If you told someone a room like that had military grade security, you would be called a liar. No matter how fancy the padlock. Without knowing the details of DropBox's setup I'm going to refrain from calling them liars, but this all seems pretty fishy to me.
- vladd 15y agoThe analogy you mention is not very good because the hole discovered in their security model is related to duplicate content identified via the same hash value. If you have unique content that's different than anything else, even by 1 bit, then you're secure until someone uploads exactly the same content (this is due to the way in which hash functions work); meanwhile, a key under the doormat would imply a totally different security threat model.
- deleted 15y ago[deleted]
- pacemkr 15y agoI think most of us, if we were developing Dropbox, would have made the same decisions. De-duping at the cost of complete privacy, even in the face of law, is a sound technical and business decision for a service such as Dropbox. When I think Dropbox, I think sharing, I think convenience, I don't think backup and security. For backup I need more space, for security I need to use my own private key (not a password that one can change/recover). Neither of these things is offered by Dropbox. And this is the reason why I never confused Dropbox with, say, CrashPlan. One is a way to share and collaborate, the other is a place to send my private key encrypted bits to. My individual privacy is not compromised by somebody being able to say if a certain file is stored by the entirety of Dropbox user base. The other claim, that, given a court order, Dropbox can be forced to turn over your files or tell the court if you store a certain file _may_ be true, but I don't think Dropbox, the company, has ever promised that level of security.
- earl 15y agoIt's far from obvious that some crusading DA can pick files and force dropbox to tell him or her who has copies of that file.
- JoachimSchipper 15y agoDropbox can certainly get that information - remember, they can access all accounts.
- earl 15y agosorry, that was my point. I use dropbox, and I knew they hashed files, but I didn't think through the implications.
- AlexandrB 15y agoBottom line - you should assume that you cannot trust ANY cloud-based service to keep what you upload safe from hackers (if they're determined enough), and especially from governments. I'm not sure why anyone would be under the illusion that this is the case. Even assuming that hosting such a service would be legally possible today (not sure if it is, IANAL), it could be illegal tomorrow and the service may be compelled to hand over any data. If you want something to stay secret you MUST either: a. Not put it on the internet/cloud. b. Encrypt it yourself before uploading. Yes, there are trust issues with commodity encryption software as well, but these may be mitigated somewhat.
- daganh 15y agoThis really isn't an issue for legitimate users backing up or syncing original data. If you are paranoid about privacy, then you are probably doing something wrong. If for some reason some other user produces the same file that I have, then whats the big deal? They already arrived at that information themselves, so there is still no compromise of data. Of course the government can seize your data whether it is online or not. Don't sync copyrighted material. How many different ways do they have to tell you it's illegal?
- tomkarlo 15y ago"If you are paranoid about privacy, then you are probably doing something wrong." That's a pretty slippery slope to presume that a desire for privacy is tantamount to an admission of guilt. Why would you presume that simply because I have information I want to keep secret, I must be doing something wrong? Conversely, is it ok if we install webcams throughout your house, since apparently you have nothing you'd like to keep private for non-criminal reasons?
- daganh 15y agoIn this instance we are talking about data that is already encrypted. Now tell me why anyone would worry about their encrypted data being identifiable unless it's not their data. Not something I have to worry about and suspect majority of users don't need to worry about this either.
- dude_abides 15y agoHere's a startup idea: A background service that runs on a PC/IPad/Phone, checks for new media files (pictures/mp3s/videos) and automatically re-encodes them, such that the quality, etc. is preserved but the file hash changes and cloud services can no longer deduplicate it. It will be quite valuable for users of services like Dropbox, Amazon Cloud Player, etc.
- evan_ 15y agoWhy would that be valuable?
- dude_abides 15y agoBecause it gives the user "plausible deniability" in case RIAA/MPAA serves him a notice.
- derefr 15y agoWhy bother re-encoding the whole file? Changing a bit or two would be sufficient.
- wulczer 15y agoYeah, and also reencoding from a lossy format always leads to quality loss, that's why the format is called lossy.
- tedunangst 15y agoThere are nonlossy transforms that can be done even with lossy formats, e.g., rotating a jpeg. ok, that's not reencoding.
- bigiain 15y agoHere's another idea: a service that identifies hash values for popular-but-copyright-encumbered files, which you can feed into your (patched copy of) Dropbox client and pretend you're about to upload it. Bam, it appears in your Dropbox account! And sneakier still, a remote hosted bit torrent client that automates that process for you in a way that can provably show you never downloaded the copyrighted file, you just had a number on your drive, which if interpreted as a SHA256 hash key happened to identify a duplicate of the copyright file on Dropbox... (I'd be spectacularly impressed if someone managed to make _that_ trick fly on court in a precedent setting manner!)
- dpcan 15y agoUhm "(if it didn't, it wouldn't be able to detect duplicate data across different accounts" How about comparing a hash of the encrypted data?
- HerraBRE 15y agoYou are able to access the data from elsewhere once you've "uploaded it". But wait, that never happened because it was a dup. That means they can, and will, decrypt somebody else's data to serve your request. Cute, huh?
- orangecat 15y agoEncryption keys would be different for different users.
- AjithAntony 15y agoI don't think anyone suggested the keys were unique to users.
- diegob 15y agoIf you knew a target file's hash, it might be possible to modify the dropbox client to report that file as added, then dropbox would download that file onto your computer. Of course, 10^77 possible hashes makes it unlikely.
- inaequitas 15y agoWithout knowing the internals of how Dropbox operates, my empirical observations are that they employ block-level deduplication, i.e. when you change bits in the middle of the file, the whole thing doesn't get re-uploaded. Which means they keep pointers and have an algorithm that's similar to LBFS (and Rabin fingerprints) This means it's theoretically possible for parts of the file to come from different sources, which means contraband files are 'built' from parts of otherwise legal files.
- cageface 15y agoI've been traveling for most of the last six months, mostly dependent on slow & unreliable hotel wifi. Dropbox's implementation has saved me a ton of time backing up files that would have taken forever to upload in their entirety.
- ikcor 15y agoDropbox has responded to this: http://forums.dropbox.com/topic.php?id=36365 http://forums.dropbox.com/topic.php?id=36365
- arashf 15y agoHi all, Arash from Dropbox here. We understand the concern that the government could try to guess whether a particular file has been uploaded to Dropbox based on processing times and then request that Dropbox identify a user who has access to that file. However, to seek user content information, the government needs to comply with the provisions of the Electronic Communications Privacy Act by obtaining a warrant supported by probable cause (or in some cases a court order from a judge). Those safeguards protect user privacy. De-duplication does not make users any more vulnerable to intrusive government actions. Today, a government agency could ask any online service to provide the names of all users who have a particular file, whether or not the service employs de-duplication. And in that case, the government would also need to support its request with a warrant or court order. The rules that provide a check against unwarranted government snooping apply to online services equally, regardless of their back-end architecture.
- etherael 15y agoGranted, but the point of the article is that other services which do not have this ability are not vulnerable to said orders, they cannot do what they do not have the ability to do.
- JoachimSchipper 15y agoYes, this is key. Tarsnap and other encrypt-then-dedup services simply cannot comply with such an order.
- sigil 15y agoIf you don't mind my asking, what is the percentage savings achieved by de-duplication across all of Dropbox? Some others here have wondered if it was premature optimization.
- ugh 15y agoI know quite a few people who appreciate Dropbox’s de-duplication. It’s not only about saving Dropbox bandwidth.
- 15y ago
- kmfrk 15y agoEarly optimization is the root of all evil, so I understand that an up-and-coming company might do this, but Dropbox has the traction and userbase to make this a very relevant concern. Popularity is also proportional to chance of being targeted by hackers and approached by government or corporation representing intellectual property owners.
- Terretta 15y agoIt's not clear to me why Dropbox would need your keys to de-dupe. He says so in the article, but doesn't say why. Why not compute the file hash on your local machine before encryption, and check that hash against a master dupe list (hash, dupe_count) of all hashes from all users' pre-encrypted local files? Secondly, I cannot see how this requires there to be an index of users hashes. Surely one could store hashes with reference count, increment when a user adds, decrement when a user deletes. The user ID isn't necessary for a reference counter. Not saying Dropbox isn't doing what he says. But he says de-duping proves they can decrypt and proves they have a list of who has the same files. I don't see it from de-dupe alone.
- bgentry 15y agoThe proof is indeed in the deduplication. If Dropbox can skip the upload process of some large file because another user has already uploaded it, they must also be able to decrypt that file in order to sync it with your other machines. Or in order for you to download it through the web interface unencrypted.
- rlpb 15y ago> If Dropbox can skip the upload process of some large file because another user has already uploaded it, they must also be able to decrypt that file in order to sync it with your other machines. Not necessarily. The client could send an encrypted version with only (plaintext) hashes of the pieces. EDIT: no, I'm wrong. > Or in order for you to download it through the web interface unencrypted. This one I will give you, unless they're doing something really weird like client side decryption through Javascript, which I'm not sure is even possible. However, they could in theory not store the key until you actually use the web interface (and you don't have to, so they wouldn't have it), and also not store the key when you do.
- bgentry 15y agoI don't follow. How could the client send an encrypted version of a very large file using only 16 KB on the network?
- kenjackson 15y agoWhy doesn't DropBox just stop doing de-duplication? They must have the money for the storage? The bandwidth savings for users isn't that big of a deal in most cases. I expect that if I have a 2GB file that I'm uploading a 2GB file. I don't cross my fingers that you already have big chunks of it. This just seems like the type of thing that someone much smarter than I can and will exploit in the future.
- wazoox 15y agoDropbox provides lots of storage for free. Therefore they have a strong incentive to provide this free storage at the lowest possible cost, and deduplication definitely makes sense for themselves if not the users.
- podperson 15y agoRather than avoid deduplication (which is technically sound and benefits everyone), perhaps the solution to this is to make it impossible for DropBox itself to know who owns which files. E.g. right now I assume a dropbox user owns a list of file ids with some metadata (e.g. that user's name for those files). If follows that if the government decides file XYZ is illegal then anyone with XYZ in their list is in trouble. The user account could keep track of the total size of all the user's files and use arithmetic to keep it up-to-date, but not actually store the size of individual files except when they are "looked at". So then the user's password (say) which is not itself stored is used to unlock stuff in the user's file table on a per request basis -- i.e. the actual file ids are only computed as needed. The actual mechanism doesn't need to be terribly secure, it just needs to be deniable. In other words without the user's password we simply cannot unambiguously determine which files are his or hers.
- qeorge 15y agoIts not obvious to me this is a price based decision for Dropbox (although the benefit there is obvious). Arguably the best feature of Dropbox for me is binary diffs. If you encrypt the Dropbox this goes out the window, or at least becomes significantly harder to pull off. Am I wrong?
- deleted 15y ago[deleted]
- rarrrrrr 15y agoPrevious HN discussion of SpiderOak's (very different) approach to this same topic: http://news.ycombinator.com/item?id=1640074 http://news.ycombinator.com/item?id=1640074
- wazoox 15y agoSpiderOak definitely looks better from an end user's point of view. How is it going? All of PR love seems to be going to DropBox nowadays.
- kapitalx 15y agotl;dr Dropbox use their own encryption keys to encrypt your data rather than encrypting each user's data using a user provided key. This helps them dedup files and save space/money. This implies that a court could ask to analyse your data. Dropbox will update their privacy policy to say this clearly.
- TheSwede75 15y agoI think it's important to realize that there is a difference between Dropbox, sugarsync etc and companies such as SpiderOak. Dropbox and sugarsync both (as examples) are primarily SYNC and SHARE services that also offer backup. While SpiderOak for example is a secure backup service that also feature sync and sharing capabilities.
- ToastOpt 15y agoCould be worse. Last year when I tried ZumoDrive (a similar service), I noticed it marks the web-browser login cookies as safe for HTTP, and defaults to open session pages via HTTP. All it takes it checking your ZumoDrive once from an unsecured WiFi and your account may be compromised. At least Dropbox gets the endpoint-to-server encryption right.