6 ms·

The encryption technique sounds like convergent encryption. Convergent encryption is a fairly standard way storing encrypted data that might duplicate across u

The encryption technique sounds like convergent encryption. Convergent encryption is a fairly standard way storing encrypted data that might duplicate across users. It allows for deduping of data while in an encrypted state. https://en.wikipedia.org/wiki/Convergent_encryption https://en.wikipedia.org/wiki/Convergent_encryption

matt_wulfeck 9y ago
Doesn’t this in some way betray the encryption? For example, if multiple users had illegal.csv in their accounts, getting access to one user will allow you to prosecute any of them without decrypting their data because they will all share the same sha?
- simias 9y ago
  It is (as the Wikipedia article points out). It's basically equivalent to storing passwords hashed without a salt, it makes makes it obvious when a password is reused, or in this case the same file stored twice. That means that the provider always knows if a file is shared across multiple users (even if they don't know what the file is) and given a cleartext file they can always check if somebody has it stored on the service. It's not ideal at all if you want good privacy. SpiderOak is explicitly supposed not to do that[1] to avoid these issues, however I refuse to recommend them until they finally decide to release an open source client and 2FA support so caveat emptor. Anyway, in this case it's irrelevant because Apple has access to the keys anyway, it's only supposed to prevent the third party (Google Cloud in this case) from having access to the files. [1] https://spideroak.com/resources/encryption-white-paper https://spideroak.com/resources/encryption-white-paper in the "Data Deduplication" section.
  - Someone1234 9y ago
    It should be pointed out that this "security flaw" is the only way these cloud file storage platforms are cost effective. Since it allows the same anonymous blob stored by several dozen users to only take up one unit of space instead of one unit per user. This is particularly space saving if a particular file type was chunked cleverly so that static parts of the file's structure were stored away from dynamic parts (Microsoft's Office XML formats for example could definitely be split this way).
    - kolpa 9y ago
      It's not the "only" way. Another way: * Offer to charge less usage if a duplicate block is detected, but allow users to pay full price for privately-salted storage. This is similar in principle to how Data Saver works on your mobile device (opt-in to allow a man-in-the-middle to compress or downsample your data)
      - avar 9y ago
        I thought they same reading this thread, but there's two huge problems with that: a) Once you know the chunk size, you can determine based on pricing whether another customer has that data, which can have huge privacy implications. E.g. let's say we both work at the same company and get the same salary statement PDF aside from the dollar number & your name (which I know). I can simply brute-force craft a file that changes that number around and upload it to iCloud, when I stop paying for storage I know I've cracked what's on your drive. In any case, I'd be surprised if Apple's not already leaking this information due to caching in a way that could be revealed via timing attacks. b) It'll lead to hugely erratic pricing for consumers. E.g. let's say you download 100TB of movies from BitTorrent, now you pay almost nothing for it, but if everyone else deletes their copies pricing for you will go up. Apple could mitigate that by never raising the price on a given chunk, but that just leaves them paying for it, and it's easily abused. Open two accounts, upload the same data, then delete it from one account, pay 1/2 for storage.
      - jbuild 9y ago
        That’s a great solution, but at what point is it easier to just store your data yourself? If someone worried about their provider knowing some user has the same file as another user, that person shouldn’t really be trusting Google or Apple with anything.
      - rmrfrmrf 9y ago
        Sounds like a great side-channel attack vector for law enforcement.
    - avar 9y ago
      I don't believe that. Apple charges $10/mo for 1TB of data. That's $120/yr. You can get 1TB drives commercially for under $50 these days, you'd need a bit more than 1 drive per user for reliability & sharding, but still. Not de-duplicating the data would clearly be profitable anyway. Hetzner charges less than half of that for their storage solution, and that's without de-duplication[1]. Does decreasing the security of their users & not offering end-to-end encryption save them even more money? Sure, but I don't see how it couldn't be profitable without it, seeing as you can easily buy non-de-duplicated cloud storage for way less from other providers. 1. https://www.hetzner.com/storage-box?country=us https://www.hetzner.com/storage-box?country=us
  - Someone 9y ago
    ”in this case it's irrelevant because Apple has access to the keys anyway” Not completely, I would think. Google may be able to discover whether any user stores file F in iCloud by creating an iCloud account for themselves and uploading the file to it. If that doesn’t create a new file, it already was there before. Depending on what exactly Apple stores on iCloud, they may even be able to detect how many users or (unlikely) even which users store the file. I don’t see how they could use it, and don’t think they would use it, but Google also has a large data set of email messages and attachments sent between iCloud and gmail accounts that they could somehow use to correlate activity between their gmail servers and the “iCloud on Google cloud” servers.
  - mygo 9y ago
    Even with a salt you can still tell if a password is reused. You just have to use the same salt.
    - 5x50 9y ago
      If you are using the same salt on multiple passwords you are not doing it properly. It is supposed to be almost a nonce. 1 random salt per password.
      - mygo 9y ago
        That is the standard way of doing it, yes. However not every non-standard implementation is insecure.
- snuxoll 9y ago
  The encryption is to protect data from snooping by the cloud provider Apple hosts the chunks on, not provide end-to-end privacy/anonymity to users from courts or Apple itself. Apple has and WILL comply with subpoena's to gain access to the contents of an iCloud account (sans Keychain where Apple has literally no way of decrypting the data themselves), as such you should never assume data in iCloud is safe from a DA or prosecutor.
- discussedbefore 9y ago
  https://news.ycombinator.com/item?id=16427557#16428434 https://news.ycombinator.com/item?id=16427557#16428434 (2 days ago)
- dunham 9y ago
  It leaks this information if you use the same key for all users. It's not clear that Apple is doing this. There is some benefit to convergent encryption with a different key per user (e.g. you dedup across multiple backups from the same user).
- tombert 9y ago
  This is why I like to GPG encrypt my stuff before I upload it to Dropbox. This way, it's pretty much guaranteed not to register as something easy to correlate against, and I can use Dropbox as just a free file store. I assume some other people do this as well.
- djrogers 9y ago
  Data is broken in to chunks, then encrypted, then stored. Given that the AWS keys vary by user, and it’s not done file-by-file, any de-duping would be entirely coincidence and not leak anything.
- kochthesecond 9y ago
  The checksums might be scoped per user in some way, or they are quite brave

matt_wulfeck 9y ago

Doesn’t this in some way betray the encryption? For example, if multiple users had illegal.csv in their accounts, getting access to one user will allow you to prosecute any of them without decrypting their data because they will all share the same sha?

simias 9y ago
It is (as the Wikipedia article points out). It's basically equivalent to storing passwords hashed without a salt, it makes makes it obvious when a password is reused, or in this case the same file stored twice. That means that the provider always knows if a file is shared across multiple users (even if they don't know what the file is) and given a cleartext file they can always check if somebody has it stored on the service. It's not ideal at all if you want good privacy. SpiderOak is explicitly supposed not to do that[1] to avoid these issues, however I refuse to recommend them until they finally decide to release an open source client and 2FA support so caveat emptor. Anyway, in this case it's irrelevant because Apple has access to the keys anyway, it's only supposed to prevent the third party (Google Cloud in this case) from having access to the files. [1] https://spideroak.com/resources/encryption-white-paper https://spideroak.com/resources/encryption-white-paper in the "Data Deduplication" section.
- Someone1234 9y ago
  It should be pointed out that this "security flaw" is the only way these cloud file storage platforms are cost effective. Since it allows the same anonymous blob stored by several dozen users to only take up one unit of space instead of one unit per user. This is particularly space saving if a particular file type was chunked cleverly so that static parts of the file's structure were stored away from dynamic parts (Microsoft's Office XML formats for example could definitely be split this way).
  - kolpa 9y ago
    It's not the "only" way. Another way: * Offer to charge less usage if a duplicate block is detected, but allow users to pay full price for privately-salted storage. This is similar in principle to how Data Saver works on your mobile device (opt-in to allow a man-in-the-middle to compress or downsample your data)
    - avar 9y ago
      I thought they same reading this thread, but there's two huge problems with that: a) Once you know the chunk size, you can determine based on pricing whether another customer has that data, which can have huge privacy implications. E.g. let's say we both work at the same company and get the same salary statement PDF aside from the dollar number & your name (which I know). I can simply brute-force craft a file that changes that number around and upload it to iCloud, when I stop paying for storage I know I've cracked what's on your drive. In any case, I'd be surprised if Apple's not already leaking this information due to caching in a way that could be revealed via timing attacks. b) It'll lead to hugely erratic pricing for consumers. E.g. let's say you download 100TB of movies from BitTorrent, now you pay almost nothing for it, but if everyone else deletes their copies pricing for you will go up. Apple could mitigate that by never raising the price on a given chunk, but that just leaves them paying for it, and it's easily abused. Open two accounts, upload the same data, then delete it from one account, pay 1/2 for storage.
    - jbuild 9y ago
      That’s a great solution, but at what point is it easier to just store your data yourself? If someone worried about their provider knowing some user has the same file as another user, that person shouldn’t really be trusting Google or Apple with anything.
    - rmrfrmrf 9y ago
      Sounds like a great side-channel attack vector for law enforcement.
  - avar 9y ago
    I don't believe that. Apple charges $10/mo for 1TB of data. That's $120/yr. You can get 1TB drives commercially for under $50 these days, you'd need a bit more than 1 drive per user for reliability & sharding, but still. Not de-duplicating the data would clearly be profitable anyway. Hetzner charges less than half of that for their storage solution, and that's without de-duplication[1]. Does decreasing the security of their users & not offering end-to-end encryption save them even more money? Sure, but I don't see how it couldn't be profitable without it, seeing as you can easily buy non-de-duplicated cloud storage for way less from other providers. 1. https://www.hetzner.com/storage-box?country=us https://www.hetzner.com/storage-box?country=us
- Someone 9y ago
  ”in this case it's irrelevant because Apple has access to the keys anyway” Not completely, I would think. Google may be able to discover whether any user stores file F in iCloud by creating an iCloud account for themselves and uploading the file to it. If that doesn’t create a new file, it already was there before. Depending on what exactly Apple stores on iCloud, they may even be able to detect how many users or (unlikely) even which users store the file. I don’t see how they could use it, and don’t think they would use it, but Google also has a large data set of email messages and attachments sent between iCloud and gmail accounts that they could somehow use to correlate activity between their gmail servers and the “iCloud on Google cloud” servers.
- mygo 9y ago
  Even with a salt you can still tell if a password is reused. You just have to use the same salt.
  - 5x50 9y ago
    If you are using the same salt on multiple passwords you are not doing it properly. It is supposed to be almost a nonce. 1 random salt per password.
    - mygo 9y ago
      That is the standard way of doing it, yes. However not every non-standard implementation is insecure.
snuxoll 9y ago
The encryption is to protect data from snooping by the cloud provider Apple hosts the chunks on, not provide end-to-end privacy/anonymity to users from courts or Apple itself. Apple has and WILL comply with subpoena's to gain access to the contents of an iCloud account (sans Keychain where Apple has literally no way of decrypting the data themselves), as such you should never assume data in iCloud is safe from a DA or prosecutor.
discussedbefore 9y ago
https://news.ycombinator.com/item?id=16427557#16428434 https://news.ycombinator.com/item?id=16427557#16428434 (2 days ago)
dunham 9y ago
It leaks this information if you use the same key for all users. It's not clear that Apple is doing this. There is some benefit to convergent encryption with a different key per user (e.g. you dedup across multiple backups from the same user).
tombert 9y ago
This is why I like to GPG encrypt my stuff before I upload it to Dropbox. This way, it's pretty much guaranteed not to register as something easy to correlate against, and I can use Dropbox as just a free file store. I assume some other people do this as well.
djrogers 9y ago
Data is broken in to chunks, then encrypted, then stored. Given that the AWS keys vary by user, and it’s not done file-by-file, any de-duping would be entirely coincidence and not leak anything.
kochthesecond 9y ago
The checksums might be scoped per user in some way, or they are quite brave