3 ms·
Note that in practice fancy frequency analysis is usually overkill, and may even lead you astray against texts that don't match your assumptions. A cheesy match
by tialaramex 6y ago
Note that in practice fancy frequency analysis is usually overkill, and may even lead you astray against texts that don't match your assumptions. A cheesy match against ASCII alphanumerics plus space is both simple and often effective.
- redkinght99 6y agoYou wouldn't happen to have any of examples of this "cheesey matching" would you? :D
- tialaramex 6y agoI really mean as simple as checking whether a byte is 0x41 to 0x5A (upper case Latin letters) 0x61 to 0x7A (lower case) or 0x30 to 0x39 (numerals) or 0x20 (space) With XOR you've got 8 bits to correct, let's look at each in turn to see why my cheesy approach is enough on a chunk of likely ASCII or ASCII-compatible text: 0x80 the most significant bit - If this is wrong nothing will match because 0x80 isn't set in any of our matches 0x40 - If this is wrong digit zero and space won't match, upper case letters won't match, and lower case 'a' through 'o' plus 'z' won't match. 0x20 - If this is wrong space and numerals don't match. It also "flips" the case which means if you're just fishing for the plaintext and there were no spaces or numerals it's still easily readable. 0x10 - If this is wrong all numerals except zero don't match, and letters 'K' through 'O' in either case don't match 0x08 - If this is wrong space doesn't match. Digits 2 through 7, letters 'S' through 'W' in either case don't match 0x04 - If this is wrong space doesn't match. Digits 8 and 9 don't match. Letters 'X' through 'Z' in either case don't match. 0x02 - If this is wrong space doesn't match. Digits 8 and 9 again, and this time only the letter 'Y' (either case) doesn't match. 0x01 least significant - Only space doesn't match. You can see that space is really doing a lot of work for you here, but on the other hand humans really like space. If the text you thought would be French poetry is actually Python it's likely still full of spaces even though the letter frequencies are way off.
- exmadscientist 6y agoDon't forget line endings! At least for texts using it, the CR-LF combination can make removing XOR obfuscation the work of just a few minutes. But not as easy as binaries... I've seen a few binaries in the wild that were "protected" with XOR "encryption". Sure, you can go for the header and figure out the XOR constant that way. But no need to waste your time on that! Binaries are often full of long stretches of zeros... and guess what happens to those under XOR?
- saagarjha 6y agoFor XOR you only need one letter to match to figure out the rest, and for any reasonable length plaintext that isn’t just padded with Qs or something is extremely likely to have E be the most common letter. So in this case it’s not a bad idea; in slightly more complicated scenarios (such as a substitution cipher) frequency analysis generally loses its accuracy past a couple letters.
- tptacek 6y agoThe most frequent character in normal plaintext is not in fact 'e'.