4 ms·

I think the best approach is never save unencrypted data on cloud. Always encrypted on client first. But by that way we lost dedup capability, so we have to do

I think the best approach is never save unencrypted data on cloud. Always encrypted on client first. But by that way we lost dedup capability, so we have to do everything, such as encryption, dedup and compression on client side. I made an in-app file system dedicated for that purpose. https://github.com/zboxfs/zbox https://github.com/zboxfs/zbox

philips 7y ago
I think this is a really compelling approach! What sorts of applications have started to adopt this? I also thought a different layer to start at would be sqlite databases since I understand that many mobile application use that. How do applications handle conflicts? It looks like there is a version on files but when is a new version created? On close? Do you see GDPR or any other compelling event that will cause applications to consider this sort of cloud storage?
- burmecia 7y ago
  I think any applications need store confidential files on client or remote can adopt it. Web and mobile app might be the best to use it at this moment. As there is transaction control, the conflict handling should be straightforward. That is, the thread got write lock can write the file exclusively and each write is a transaction and commit will form a new permanent version. GDPR might be a good reason, but I think it can be more general. Any apps need store confidential data can use this, no matter the data is on local or cloud.
viraptor 7y ago
> But by that way we lost dedup capability This depends on how secret do you want your data to be. You could use block-based encryption/compression and backup. That way you can still dedup encrypted result. If anyone can inject data into your system and monitor the backup, they could learn when they hit collisions, but for most personal backup cases that's irrelevant.
- kochthesecond 7y ago
  if you encrypt in a way that enables the service to do dedupe, you are either reusing IVs and encryption keys across items (bad) and leaking information that two items are the same item.
  - viraptor 7y ago
    You must not reuse IV between different blocks, but that does not stop you from using the same IV for the same block. Yes, you leak information about matching blocks - it's up to your use case whether you care about it.
    - kochthesecond 7y ago
      Yes, sorry I was imprecise. I meant reusing IVs for equal blocks only, to be able to see duplicate data.
- rzzzt 7y ago
  Turning two blocks of cleartext with the same content into equal blocks of cyphertext is not great for encryption: https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Electronic_Codebook_(ECB) https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation...
- burmecia 7y ago
  I don't think the encrypt-then-dedup is a safe way to protect data privacy. In this case, identical blocks need to produce same cipher text, this will actually leak your data pattern even though it is encrypted. A better way I think is using randomly-seeded derived keys to encrypt each block, thus the identical blocks' cipher text will always be different.
  - viraptor 7y ago
    Yes, if that's more important to you than dedup savings, then you should definitely do that.
- 7y ago

philips 7y ago

I think this is a really compelling approach! What sorts of applications have started to adopt this? I also thought a different layer to start at would be sqlite databases since I understand that many mobile application use that. How do applications handle conflicts? It looks like there is a version on files but when is a new version created? On close? Do you see GDPR or any other compelling event that will cause applications to consider this sort of cloud storage?

burmecia 7y ago
I think any applications need store confidential files on client or remote can adopt it. Web and mobile app might be the best to use it at this moment. As there is transaction control, the conflict handling should be straightforward. That is, the thread got write lock can write the file exclusively and each write is a transaction and commit will form a new permanent version. GDPR might be a good reason, but I think it can be more general. Any apps need store confidential data can use this, no matter the data is on local or cloud.

viraptor 7y ago

> But by that way we lost dedup capability This depends on how secret do you want your data to be. You could use block-based encryption/compression and backup. That way you can still dedup encrypted result. If anyone can inject data into your system and monitor the backup, they could learn when they hit collisions, but for most personal backup cases that's irrelevant.

kochthesecond 7y ago
if you encrypt in a way that enables the service to do dedupe, you are either reusing IVs and encryption keys across items (bad) and leaking information that two items are the same item.
- viraptor 7y ago
  You must not reuse IV between different blocks, but that does not stop you from using the same IV for the same block. Yes, you leak information about matching blocks - it's up to your use case whether you care about it.
  - kochthesecond 7y ago
    Yes, sorry I was imprecise. I meant reusing IVs for equal blocks only, to be able to see duplicate data.
rzzzt 7y ago
Turning two blocks of cleartext with the same content into equal blocks of cyphertext is not great for encryption: https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Electronic_Codebook_(ECB) https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation...
burmecia 7y ago
I don't think the encrypt-then-dedup is a safe way to protect data privacy. In this case, identical blocks need to produce same cipher text, this will actually leak your data pattern even though it is encrypted. A better way I think is using randomly-seeded derived keys to encrypt each block, thus the identical blocks' cipher text will always be different.
- viraptor 7y ago
  Yes, if that's more important to you than dedup savings, then you should definitely do that.
7y ago