3 ms·
I have a compression issue much closer to my heart that is not being properly addresses: AES-256 encrypted plaintext files. Seriously! If you just figure out th
by beagle3 11y ago
I have a compression issue much closer to my heart that is not being properly addresses: AES-256 encrypted plaintext files. Seriously! If you just figure out the 256-bit key, it would compress well with any compressor - there's a huge class of files which would only need 256 additional bits! Why isn't there any advance on this front? /s
Entropy is not a broken measurement - in fact, it's the perfect measurement (proof: Shannon's source coding theorem and its converse), its only downside is that it is only defined with respect to a model.
If you measure it on the output of one model (permutations) with respect to another model (i.i.d) then, duh, it's wrong. And general permutations are so rare, that they might bug the author but there's no incentive to consider them. Non general permutations - e.g., addition-modulo permutations, often ARE reasonably well compressed by markov/tree/fsmx model algorithms and their delta counterparts. (And they're still too rare to consider on their own).
Kolmogorov complexity is interesting, but it is too vast and complicated to consider as a whole (proof: equivalence of computing Chaitin's omega to the halting problem), other than as a very useful reasoning device. Therefore, like with the halting problem, we focus on specific subsets that seem interesting.
Frankly, this rant reads like "But I don't understand why they don't yet have a car that runs on water". Compression is math, and it's a part of math we understand quite well. BWT, for example, is an amazing algorithm - but if you take a step back, it's just an efficient implementation of a compression scheme that was known before. (IIRC, the last gap in understanding the equivalence was closed by Effros et al in 2002)
- raverbashing 11y agoWell, that's why you compress then encrypt first Entropy is a good measurement but it's not the goal. The goal is a smaller file size Yeah, I guess that for 'generic compression' the advances are going to be slow and few. Without any knowledge of the type of file it's only bytes so you have to work with that. Also disk space and network speeds are getting cheaper, so what might have needed compression in the past might not need today (even though compression got much cheaper)
- detaro 11y ago> Well, that's why you compress then encrypt first Just be careful of CRIME-style attacks...
- pella 11y agowiki: "CRIME ("Compression Ratio Info-leak Made Easy") is a security exploit against secret web cookies over connections using the HTTPS and SPDY protocols that also use data compression." https://en.wikipedia.org/wiki/CRIME https://en.wikipedia.org/wiki/CRIME
- hannob 11y agoIt is a design goal of symmetric crypto algorithms to produce an output that is indistinguishable from random with any reasonable effort. Thereby what you're asking for is a compression algorithm that is so good that you can use it to attack AES. As long as we consider AES as safe that won't happen. (And I don't expect that AES gets anywhere near being broken within decades to come.)
- beagle3 11y agoYes, that is exactly what I was hinting at. The /s at the end of my first paragraph stands for "and without sarcasm from here on...".
- deleted 11y ago[deleted]