3 ms·

What? How can they tell the difference between an encrypted file vs. some binary format they simply don't understand?

underdeserver 3y ago
There aren't really all that many common unknown binary formats, in terms of GBs in people's drives. Mostly it's photos, pdfs, word documents, executables, videos, etc. Aside from that there are mathematical tools you can use to detect whether a binary blob is compressed/encrypted or not, like entropy calculation.
NavinF 3y ago
Incompressible files that don't use a standard compression format (like a video codec) are usually encrypted
codedokode 3y ago
Let me share a Chinese Firewall (according to rumours) algorithm for detecting cryptography: count number of 0 and 1 bits. If the number is almost equal, it is likely encrypted content.
- rkagerer 3y ago
  Sounds like a cat and mouse game - hasn't someone tweaked an encryption algorithm yet to produce a less even distribution?
- ajb 3y ago
  That also applies to any good compression though.
  - freedomben 3y ago
    True, but since very, very few files are compressed with unique or unknown compression algorithms, you can read the metadata in the file and see if it's really compressed or if the whole thing is just seemingly random. That gives me an idea. Might be fun to write a tool that will encrypt a file but make it look like a valid .zip file that just unpacks to meaningless noise. There's gotta be something like that already out there... Although you could recursively unpack/evaluate and eventually you should get to the point where it's non-random data. If you never do, then it's encrypted.
    - ajb 3y ago
      Makes sense. There are some academic papers (I think one by Ross Anderson) which state that steganography relies on there being some randomness in the data you are hiding the message inside. So, to make the bandwidth available for for steganographic data hiding as large as possible, those needing to conceal data need incentives for innocents to publish files constructed using as large as possible quantities of random bits. It looks like Stable Diffusion is providing this incentive!

There aren't really all that many common unknown binary formats, in terms of GBs in people's drives. Mostly it's photos, pdfs, word documents, executables, videos, etc. Aside from that there are mathematical tools you can use to detect whether a binary blob is compressed/encrypted or not, like entropy calculation.

NavinF 3y ago

Incompressible files that don't use a standard compression format (like a video codec) are usually encrypted

codedokode 3y ago

Let me share a Chinese Firewall (according to rumours) algorithm for detecting cryptography: count number of 0 and 1 bits. If the number is almost equal, it is likely encrypted content.

rkagerer 3y ago
Sounds like a cat and mouse game - hasn't someone tweaked an encryption algorithm yet to produce a less even distribution?
ajb 3y ago
That also applies to any good compression though.
- freedomben 3y ago
  True, but since very, very few files are compressed with unique or unknown compression algorithms, you can read the metadata in the file and see if it's really compressed or if the whole thing is just seemingly random. That gives me an idea. Might be fun to write a tool that will encrypt a file but make it look like a valid .zip file that just unpacks to meaningless noise. There's gotta be something like that already out there... Although you could recursively unpack/evaluate and eventually you should get to the point where it's non-random data. If you never do, then it's encrypted.
  - ajb 3y ago
    Makes sense. There are some academic papers (I think one by Ross Anderson) which state that steganography relies on there being some randomness in the data you are hiding the message inside. So, to make the bandwidth available for for steganographic data hiding as large as possible, those needing to conceal data need incentives for innocents to publish files constructed using as large as possible quantities of random bits. It looks like Stable Diffusion is providing this incentive!