8 ms·
I just found a weird things on they ToS [1] ... 8. Our service may automatically delete a piece of data you upload or give someone else access to where it dete
by gjulianm 14y ago
I just found a weird things on they ToS [1] ...
8. Our service may automatically delete a piece of data you upload or give someone else access to where it determines that that data is an exact duplicate of original data already on our service. In that case, you will access that original data.
Duplicate check, I get that. But, how do they do it? They say the files are encrypted on the browser, so if I upload file X and other user uploads X too, they can't know they're the same because both uploads are encrypted. So, they can check only for duplicates of the encrypted outcome of each file. But, wouldn't that be inefficient? Probability of collision in encrypted files is (AFAIK) really low, something like 2^(-N), N being the size of the file on bits... If I did it well, it'd be a collision probability of 7.458E-155 for a file of 1MB.
[1] https://mega.co.nz/#terms https://mega.co.nz/#terms
EDIT: Added example.
- arankine 14y agoThis is exactly how megaupload used to work. They use file hashes, i heard reports of some hash collisions meaning entirely different files were identified as duplicates...that was gamed though i believe, real life collisions less likely
- dabeeeenster 14y agoSomething doesn't smell right regarding the browser-based private key encryption. What I think is happening is that they could theoretically decrypt all your content, but the fact that they dont, and would have to jump through some technical hoops to do so, is maybe enough for them to argue that they don't know what is being uploaded?
- deleted 14y ago[deleted]
- dexter313 14y agoThe user's javascript that encrypted the file can also make a hash send it to mega, and then they check if they already have that. EDIT: They never get to see your key or what is in the file.
- gjulianm 14y agoOk, so I try to upload a.exe to Mega. They make a hash, detect someone has already uploaded it. They don't upload my file, and instead they place a link in my account to that "a.exe" of some other user. How can I access it then? Because it's encrypted with a key which is not mine.
- jiggy2011 14y agoThe hashing is done by the client before upload. So if all clients use the same hash algo they will generate the same hash for the same file. So it is encrypted with a key that you know because you have the original file.
- beala 14y agoDeriving the key from the plaintext + the ciphertext is called a known-plaintext attack [1]. AES isn't vulnerable to this. [1] http://en.wikipedia.org/wiki/Known-plaintext_attack#Present_day http://en.wikipedia.org/wiki/Known-plaintext_attack#Present_...
- jiggy2011 14y agoIt is if you know that the key is derived directly and deterministically from the plaintext itself. Of course this will only give you the key to that one particular file, not any other files that you do not have yourself.
- beala 14y agoAh I see this was in reference to your convergent encryption post above. Point taken. Assuming that it works this way, it would allow Mega to figure out if you own a known "bad" file. Just take something like "New_Jay-Z_Album.zip," hash it, and try the hash against your encrypted files. It seems like Kim is trying to avoid this problem.
- g_lined 14y ago
- jiggy2011 14y agoPossibly convergent encryption, basically when you encrypt the file you use a hash of the file as the key. This key can then be encrypted with several different passwords meaning that several people can decrypt this file.
- cryptox 14y agoWouldn't this allow MEGA to also get a copy of the key so they can decrypt your data?
- cynicalkane 14y agoMega would be storing copies of the key that can only be decrypted with a password.
- A1kmm 14y agoBut when someone uploads the same file with a different password, if they want to de-duplicate on content alone, they need to be able to put that new encrypted file hash alongside the original. One solution to that would be to have two hashes of the file, and to use one as the key and one as the index.
- jiggy2011 14y agoOnly if they had a copy of the original plaintext file.
- gjulianm 14y agoWhich in theory they don't have, as they encrypt it on your browser.
- Sami_Lehtinen 14y agoWell, you said "encrypt", what do you mean by encrypt? What's the key? In the Freenet's solution, they encryption key for the data is the data, not any random or user provided key. So the same plaintext is always turned in to same ciphertext. Which we all know, it's very bad idea, it ruins encryption. In this case, it's quite easy to spot out all of the users having the same content in their account. This is only reasonable method when ever you can assume that every plaintext is unique, which also makes point of deduplicationg data absolutely pointess.
- tommoor 14y agoGot to wonder if this is a bit of copy and paste from the old terms, although you would think they would have checked this quite thoroughly! Good spot.
- caw 14y agoMaybe they use block deduplication on the storage arrays? Particularly looking at the "our service may automatically delete a piece of data you upload". In other words, you already uploaded it encrypted and they're just serving out the de-duplicated bits.
- dexter313 14y agoPiece is the keyword here.
- bchallenor 14y agoThere wouldn't be much point in doing that. Encrypted data is indistinguishable from random data, so it is incompressible. The chances of finding anything significant to deduplicate are so low that it's not worth trying.
- politician 14y ago"A node token ("magic cookie") grants access to a subtree of the issuing user's filesystem. An associated symmetric key is required to decrypt and/or store decryptable data." https://mega.co.nz/#developers https://mega.co.nz/#developers Apparently, they use unsalted symmetric key encryption which allows them to discover [hash(file), password] duplicates. By comparison, the old Megaupload would apparently deduplicate based on [hash(file)] matches. Suppose Alice and Bob have files [D, E, F] and [F, G, H], respectively. If MEGA discovers that Alice and Bob share a duplicate file F and Alice to reveals her password (through password frequency analysis or to the Government), then all of Bob's files are compromised. I would personally feel unsafe storing private documents on Mega due to the lack of public/private key encryption, but that's me.
- jiggy2011 14y agoI'm not entirely sure what they mean by a "magic cookie" here. The document does say it uses public/private keys for data transfer. You would not usually use public/private for data storage because of the huge keysizes required.
- politician 14y agoI think the cookie bit is just an authorization credential. If they use asymmetric encryption for data transfer, then how does that work as part of a convergent encryption scheme? Wouldn't all of the file hashes be different at that point?
- jiggy2011 14y agoI would assume the asymmetric crypto is just for the transfer. In other words you encrypt the data and then send it wrapped in another layer of public/private crypto. Not entirely sure though.
- jlgreco 14y ago> I would personally feel unsafe storing private documents on Mega due to the lack of public/private key encryption, but that's me. It is probably worth pointing out that all of this encryption stuff is only for them. Nomatter how they brand it, the purpose is to give them plausible deniability, which they lacked with the previous Mega. If you want encryption meant for you, then you should probably just encrypt your files yourself before you let Mega touch them.
- AhtiK 14y agoFrom dev docs it looks like file integrity (and thereby duplicate check) is done with CBC-MAC: "File integrity is verified using chunked CBC-MAC. Chunk sizes start at 128 KB and increase to 1 MB, which is a reasonable balance between space required to store the chunk MACs and the average overhead for integrity-checking partial reads."
- jiggy2011 14y ago"Each file and each folder node uses its own randomly generated 128 bit key. File nodes use the same key for the attribute block and the file data, plus a 64 bit random counter start value and a 64 bit meta MAC to verify the file's integrity." Not sure I understand this, how can you deduplicate if every file is encrypted via a random key?
- kbaker 14y agoMaybe on file upload, they encrypt it with the file hash then chunk those encrypted files and store those with dedup. Then, on the user side, they store an per-user encrypted index (random, counter, MAC) to those individual chunks to represent the file. That way, they can only see giant encrypted blocks of data, and per-user encrypted indexes to data. But it is all encrypted. They would need to hack into accounts by keylogging passwords to decrypt the indexes and see what files users can actually access. Public links could be shared by giving out a key in the URL that is a file containing indexes to other blocks. So whoever knows the URL, knows the index, and can get the data. That is the way I'd design it, at least... :) edit:typo
- jiggy2011 14y agoNot sure, it doesn't seem to say anything about encrypting with the file hash and implies that metadata and actual data use the same key.
- huhtenberg 14y agoPerhaps they de-duplicate using smaller block sizes? Something like 256 bytes. Also, if you believe Microsoft Outlook, there's something called "compressible encryption", which implies there are encryption schemes that aren't exactly random, meaning in turn that not all N-bit blocks are equally likely.
- lubos 14y agoeven 256 bytes block is big enough to make collisions so rare that it's not even worth. not to mention metadata associated with such a small block would be probably more than the size of the block itself.
- doe88 14y agoIt could be achieved through convergent encryption see this presentation from last week http://crypto.stanford.edu/RealWorldCrypto/slides/tom.pdf http://crypto.stanford.edu/RealWorldCrypto/slides/tom.pdf
- tlogan 14y agoI guess that is generally done using "convergent encryption". There are many variations and they were done by some peer-to-peer DHT filesystem (I was kinda involved in some "skunk" projects developing it). The idea is simple: if you have data, then you can generate the key from data itself to encrypt and decrypt than data. Then you use hash of encrypted data to look up if server (or just other side) has the same data. If hash exists, no need to send to server - just tell server to bump references. If hash does not exists, just upload data. The key derived from data needs to be stored locally - if you lose it, you will lose data too.
- cryptox 14y agoAnd deduplication would work since the 2nd person would derive the same key for the data and thus can decrypt it?
- dlss 14y agoYes exactly. key = f(data); upload(encrypt(data, key)); store_key(filename, key); Which is to say that every file has a globally 1-to-1 mapping to its encrypted version. I'm not sure how they are storing the (User, [(Filename,Key)]) data, but this is ideally encrypted on a per user basis, making any sort of per-user lookup attacks moot.
- mrb 14y agoPerhaps, a much simpler possibility than the other proposed in this thread, is that, in addition to uploading the encrypted file, the service uploads hashes of chunks of plaintext data. That way the service can just compares hashes of data, just like a regular dedup implementation.
- Giszmo 14y agomaybe they store a hash along your strongly encrypted file. this way they could go after copy-righted files accross the system once reported at the cost of quasi-decrypting some of your files. your really unique files would still be save although you might not want them to be able to timestamp proof your possession of these neither.
- tlrobinson 14y agoHere's one solution: http://news.ycombinator.com/item?id=2461713 http://news.ycombinator.com/item?id=2461713 However given a file Mega would be able to tell whether they stored that file, eliminating the plausible deniability they're aiming for.
- Sami_Lehtinen 14y agoRead Freenet documentation. They encrypt everything, yet they use very efficient deduplication. I really like Freenet's design. Encryption key is based on the payload, so if you don't know what he payload is, you can't decrypt the packet. Of course decryption keys can be delivered using different encrypted tree of keys, which is used when you deliver download link. For that reason, when ever I'm sharing anything I usually encrypt files with my recipients public keys before sending those out. Just to make sure that data is really private and keys are known only to my selected peers. In some cases when I want to make stuff even more private, I encrypt data separately with each recipients public key, so you can't even see list of public key ID's which are required to decrypt the data. I also have 'secure work station' which is hardened and not connected to internet. That's the workstation I use to decrypt, handle and encrypt data. Only encrypted and signed data is allowed to come and go to that workstation.