3 ms·

Using a random text generator http://pasted.co/encrypted.txt http://pasted.co/encrypted.txt ( 10,002 bytes ) Then compressed it with winrar http://pasted.co/c

Using a random text generator http://pasted.co/encrypted.txt http://pasted.co/encrypted.txt ( 10,002 bytes ) Then compressed it with winrar http://pasted.co/compressed.rar http://pasted.co/compressed.rar ( 7,743 bytes ) Roughly 20% compression isnt meaningless- so I fail to see why you are just giving a flat 'no' when its obvious what you are saying is untrue.

charleslmunger 10y ago
Your random text isn't uniformly random - any byte that isn't a letter or number never shows up.
- DanBlake 10y ago
  Here it is encrypted with AES256 - Pass is wasteoftime http://pasted.co/wasteoftime.txt http://pasted.co/wasteoftime.txt 4,732 bytes http://pasted.co/wasteoftime.rar http://pasted.co/wasteoftime.rar 3,753 bytes Also, if there is issues with using the 'wrong' encryption, I feel thats kind of a straw man argument. Please feel free to upload a file over 5MB which cant be reduced in size through any of the various compression tools. Also, keep in mind that I never said it would be a huge benefit at all- I only said that SOME compression was possible SOME of the time.
  - mikeash 10y ago
    That's not encrypted with AES256, that's encrypted with AES256 and then encoded with base64. Do you understand the difference between base64 and binary data? Base64 only uses six bits per byte. It's designed to allow data to transit cleanly through places which only allow text, such as JSON. Binary data is eight bits per byte. Binary data is what encryption and compression algorithms output. Naturally, if you take data which only uses six bits per byte, and run it through a compression algorithm which is able to use eight bits per byte, you can achieve good compression. But this is illusory, because you expanded the original by 33% when you applied base64 in the first place! All your attempts at compression can be beaten by simply decoding the base64 into the original binary data. It doesn't matter what magnitude of compression you specified. Random data cannot be compressed at all on average. This is a simple mathematical certainty with a straightforward proof. Encrypted data looks and acts like random data. If you find a way to reliably compress the output of a modern, secure encryption algorithm (not the base64 encoding, but the original binary) then you'll have found a massive security hole in it which will make you famous.
    - Dylan16807 10y ago
      And as a note: It's fine to use base64 if you want. But then you also need to output the compressed file as base64. And you'll see that 3753 octects equate to 5004 base64 characters, and the file is actually significantly larger than it was before compression.
  - klodolph 10y ago
    > Also, keep in mind that I never said it would be a huge benefit at all- I only said that SOME compression was possible SOME of the time. This is a common misconception. The pigeonhole principle here tells us that any scheme that ever achieves some compression also achieves some expansion. The only reason compression algorithms work at all is because they are more likely to compress than they are to expand, because we know something about the probability distribution of the plaintexts. The pigeonhole argument treats compression as a black box with input and output, and makes no assumption about how the compression works. Your idea—to only compress some inputs and not others—doesn't change the fact that your algorithm can be treated as if it is a black box with inputs and outputs. So the pigeonhole principle still applies. Many people before you have made this argument before, so we are very familiar with it, and very familiar with the reason why it is wrong. FURTHERMORE, compression, by its very nature, works very poorly on data which is apparently uniformly distributed, as encrypted data is. This is why you are wrong.
hinkley 10y ago
That's a base64 encoding of random data, which by definition has only 6 bits of entropy per byte. You should be able to compress it 25% but you only managed 22.6%
deleted 10y ago
[deleted]
r1ch 10y ago
You aren't performing your test properly. Encryption doesn't output random text, it outputs bytes of data (0-255). It should be obvious that compressing text is possible. $ dd if=/dev/urandom of=test bs=1k count=10 $ cat test | gzip -9 > test.gz $ cat test | bzip2 -9 > test.bz2 $ ls -al test* -rw-r--r-- 1 r1ch r1ch 10240 Jun 28 16:30 test -rw-r--r-- 1 r1ch r1ch 10739 Jun 28 16:31 test.bz2 -rw-r--r-- 1 r1ch r1ch 10263 Jun 28 16:31 test.gz Random data is not compressible. $ dd if=/dev/zero of=test bs=1k count=10 $ openssl enc -in test -aes-256-ctr -out test.encrypted $ cat test.encrypted | gzip -9 > test.encrypted.gz $ cat test.encrypted | bzip2 -9 > test.encrypted.bz2 $ ls -al test* -rw-r--r-- 1 r1ch r1ch 10240 Jun 28 16:32 test -rw-r--r-- 1 r1ch r1ch 10256 Jun 28 16:33 test.encrypted -rw-r--r-- 1 r1ch r1ch 10737 Jun 28 16:34 test.encrypted.bz2 -rw-r--r-- 1 r1ch r1ch 10279 Jun 28 16:33 test.encrypted.gz Encrypted zeroes are not compressible.
appleflaxen 10y ago
this difference is because your random text values are not random binary values. you are trying to bring empirical evidence to an information theoretic fight :(

charleslmunger 10y ago

Your random text isn't uniformly random - any byte that isn't a letter or number never shows up.

DanBlake 10y ago
Here it is encrypted with AES256 - Pass is wasteoftime http://pasted.co/wasteoftime.txt http://pasted.co/wasteoftime.txt 4,732 bytes http://pasted.co/wasteoftime.rar http://pasted.co/wasteoftime.rar 3,753 bytes Also, if there is issues with using the 'wrong' encryption, I feel thats kind of a straw man argument. Please feel free to upload a file over 5MB which cant be reduced in size through any of the various compression tools. Also, keep in mind that I never said it would be a huge benefit at all- I only said that SOME compression was possible SOME of the time.
- mikeash 10y ago
  That's not encrypted with AES256, that's encrypted with AES256 and then encoded with base64. Do you understand the difference between base64 and binary data? Base64 only uses six bits per byte. It's designed to allow data to transit cleanly through places which only allow text, such as JSON. Binary data is eight bits per byte. Binary data is what encryption and compression algorithms output. Naturally, if you take data which only uses six bits per byte, and run it through a compression algorithm which is able to use eight bits per byte, you can achieve good compression. But this is illusory, because you expanded the original by 33% when you applied base64 in the first place! All your attempts at compression can be beaten by simply decoding the base64 into the original binary data. It doesn't matter what magnitude of compression you specified. Random data cannot be compressed at all on average. This is a simple mathematical certainty with a straightforward proof. Encrypted data looks and acts like random data. If you find a way to reliably compress the output of a modern, secure encryption algorithm (not the base64 encoding, but the original binary) then you'll have found a massive security hole in it which will make you famous.
  - Dylan16807 10y ago
    And as a note: It's fine to use base64 if you want. But then you also need to output the compressed file as base64. And you'll see that 3753 octects equate to 5004 base64 characters, and the file is actually significantly larger than it was before compression.
- klodolph 10y ago
  > Also, keep in mind that I never said it would be a huge benefit at all- I only said that SOME compression was possible SOME of the time. This is a common misconception. The pigeonhole principle here tells us that any scheme that ever achieves some compression also achieves some expansion. The only reason compression algorithms work at all is because they are more likely to compress than they are to expand, because we know something about the probability distribution of the plaintexts. The pigeonhole argument treats compression as a black box with input and output, and makes no assumption about how the compression works. Your idea—to only compress some inputs and not others—doesn't change the fact that your algorithm can be treated as if it is a black box with inputs and outputs. So the pigeonhole principle still applies. Many people before you have made this argument before, so we are very familiar with it, and very familiar with the reason why it is wrong. FURTHERMORE, compression, by its very nature, works very poorly on data which is apparently uniformly distributed, as encrypted data is. This is why you are wrong.

hinkley 10y ago

That's a base64 encoding of random data, which by definition has only 6 bits of entropy per byte. You should be able to compress it 25% but you only managed 22.6%

deleted 10y ago

[deleted]

r1ch 10y ago

You aren't performing your test properly. Encryption doesn't output random text, it outputs bytes of data (0-255). It should be obvious that compressing text is possible. $ dd if=/dev/urandom of=test bs=1k count=10 $ cat test | gzip -9 > test.gz $ cat test | bzip2 -9 > test.bz2 $ ls -al test* -rw-r--r-- 1 r1ch r1ch 10240 Jun 28 16:30 test -rw-r--r-- 1 r1ch r1ch 10739 Jun 28 16:31 test.bz2 -rw-r--r-- 1 r1ch r1ch 10263 Jun 28 16:31 test.gz Random data is not compressible. $ dd if=/dev/zero of=test bs=1k count=10 $ openssl enc -in test -aes-256-ctr -out test.encrypted $ cat test.encrypted | gzip -9 > test.encrypted.gz $ cat test.encrypted | bzip2 -9 > test.encrypted.bz2 $ ls -al test* -rw-r--r-- 1 r1ch r1ch 10240 Jun 28 16:32 test -rw-r--r-- 1 r1ch r1ch 10256 Jun 28 16:33 test.encrypted -rw-r--r-- 1 r1ch r1ch 10737 Jun 28 16:34 test.encrypted.bz2 -rw-r--r-- 1 r1ch r1ch 10279 Jun 28 16:33 test.encrypted.gz Encrypted zeroes are not compressible.

appleflaxen 10y ago

this difference is because your random text values are not random binary values. you are trying to bring empirical evidence to an information theoretic fight :(