4 ms·

How much does encryption defeat the rsync algorithm? I don't see anything in the article that addresses that. That is, say you change one line in 1 MB text fi

by chubot 13y ago

How much does encryption defeat the rsync algorithm? I don't see anything in the article that addresses that. That is, say you change one line in 1 MB text file. If you rsync plain text, it will transfer a handful of bytes. If you encrypt and then rsync encrypted text, then presumably a lot more bytes will be scrambled and you will have a bigger diff to transfer. I guess it depends on the block size.

rsync 13y ago
It depends. If the resulting file is truly random vs. the last time you mounted it, then you (of course) need to retransmit everything. However, imagine that every time you closed your TC volume or unmounted filevault it had to rewrite an entirely new multi-GB file ? Of course that would take forever and would be difficult to use. So most encryption image software organizes the image internally such that new data written only affects portions of the file. And then, in theory, rsync with options like --partial --inplace --whatever will then transfer it efficiently. BUT, our experience is this seldom works properly. The amount of internally changed data changes dramatically form usage to usage, and often has very little to do with the amount of actual data you changed. We just never got it to work well, consistently.
- nsomaru 13y ago
  Does this mean you guys found that a large portion of the encrypted file had to be retransmitted as compared with the number of bits actually changed? How does one consistently store encrypted data using rsync.net as a backend, whilst maintaining the benefits of the rsync protocol? This seems to rule out the benefits of using rsync w/ encrypted files. Is there something I'm missing?
lcedp 13y ago
The thing is to stay secure you need to re-encrypt the whole file. Otherwise you will get this: https://en.wikipedia.org/wiki/Block_cipher_modes_of_operation#Electronic_codebook_.28ECB.29 https://en.wikipedia.org/wiki/Block_cipher_modes_of_operatio... That's why encfs uses the second mode in the list (chaining).
- nsomaru 13y ago
  Could you expand a bit? I've been thinking of building a bandwidth efficient encrypted syncing system, and this article (almost) hits the spot. How is the best way to get this done, security considerations as against bandwidth?
  - lcedp 13y ago
    I'm not highly knowledgeable in security, but the gist is if a block cipher uses the simplest mode of operation (ECB) each block is encrypted independently - good for sync bandwidth but from that the attacker can derive the structure of the file which will help to recover it or it can be used to check if known file in present in encrypted data(watermark attack [1]). More advanced block cipher modes "blend" to some extent blocks hiding file structure information (but if you change single bit a bunch (or all) of the blocks is changed). Also there are stream ciphers, which don't use blocks at all. If you change 1 bit stream cipher will change 1 bit in its output. The problem is it's insecure to use the same key twice with stream cyphers (reused key attack - [2])- e.g. if files with known text is encrypted (think English text, xml files, jpeg headers). Of course it can be overcome by using initialization vector ("seed" which is used with the key), but that IV should be different for each file and you should somehow save it to be able to decrypt it. [1] https://en.wikipedia.org/wiki/Watermarking_attack https://en.wikipedia.org/wiki/Watermarking_attack [2] https://en.wikipedia.org/wiki/Stream_cipher_attack https://en.wikipedia.org/wiki/Stream_cipher_attack
  - chubot 13y ago
    Try doing the matasano challenges :). I completed the first two and it was my first exposure to block cipher weaknesses. I had almost zero crypto knowledge before that. All you need to know is programming and basic arithmetic.

by chubot 13y ago

rsync 13y ago

It depends. If the resulting file is truly random vs. the last time you mounted it, then you (of course) need to retransmit everything. However, imagine that every time you closed your TC volume or unmounted filevault it had to rewrite an entirely new multi-GB file ? Of course that would take forever and would be difficult to use. So most encryption image software organizes the image internally such that new data written only affects portions of the file. And then, in theory, rsync with options like --partial --inplace --whatever will then transfer it efficiently. BUT, our experience is this seldom works properly. The amount of internally changed data changes dramatically form usage to usage, and often has very little to do with the amount of actual data you changed. We just never got it to work well, consistently.

nsomaru 13y ago
Does this mean you guys found that a large portion of the encrypted file had to be retransmitted as compared with the number of bits actually changed? How does one consistently store encrypted data using rsync.net as a backend, whilst maintaining the benefits of the rsync protocol? This seems to rule out the benefits of using rsync w/ encrypted files. Is there something I'm missing?

lcedp 13y ago

The thing is to stay secure you need to re-encrypt the whole file. Otherwise you will get this: https://en.wikipedia.org/wiki/Block_cipher_modes_of_operation#Electronic_codebook_.28ECB.29 https://en.wikipedia.org/wiki/Block_cipher_modes_of_operatio... That's why encfs uses the second mode in the list (chaining).

nsomaru 13y ago
Could you expand a bit? I've been thinking of building a bandwidth efficient encrypted syncing system, and this article (almost) hits the spot. How is the best way to get this done, security considerations as against bandwidth?
- lcedp 13y ago
  I'm not highly knowledgeable in security, but the gist is if a block cipher uses the simplest mode of operation (ECB) each block is encrypted independently - good for sync bandwidth but from that the attacker can derive the structure of the file which will help to recover it or it can be used to check if known file in present in encrypted data(watermark attack [1]). More advanced block cipher modes "blend" to some extent blocks hiding file structure information (but if you change single bit a bunch (or all) of the blocks is changed). Also there are stream ciphers, which don't use blocks at all. If you change 1 bit stream cipher will change 1 bit in its output. The problem is it's insecure to use the same key twice with stream cyphers (reused key attack - [2])- e.g. if files with known text is encrypted (think English text, xml files, jpeg headers). Of course it can be overcome by using initialization vector ("seed" which is used with the key), but that IV should be different for each file and you should somehow save it to be able to decrypt it. [1] https://en.wikipedia.org/wiki/Watermarking_attack https://en.wikipedia.org/wiki/Watermarking_attack [2] https://en.wikipedia.org/wiki/Stream_cipher_attack https://en.wikipedia.org/wiki/Stream_cipher_attack
- chubot 13y ago
  Try doing the matasano challenges :). I completed the first two and it was my first exposure to block cipher weaknesses. I had almost zero crypto knowledge before that. All you need to know is programming and basic arithmetic.