6 ms·
How does it kill compression?
by _pmf_ 7y ago
How does it kill compression?
- londons_explore 7y agoTry it... wget http://mattmahoney.net/dc/enwik8.zip -O - | gunzip | head -c 1000000 | gzip -9 | wc -c 355350 wget http://mattmahoney.net/dc/enwik8.zip -O - | gunzip | head -c 1000000 | base64 | gzip -9 | wc -c 528781 base64 is 48.8% larger after compression on english text, whereas it would only be 33.3% larger without compression. The reason is because compression finds and eliminates repeating patterns, but base64 can make the same input data look totally different depending on which of the 4 input alignments it has.
- xxs 7y agooddly enough lz/deflate are as ancient as base64. 16bit deflate is quite poor and slow, yet predominant.
- alboy 7y agoMost compression algos operate on single-byte chunks of data and base64 encoding messes up the original byte alignment making the input appear more random than it is.
- CamperBob2 7y agoFor one thing, it removes opportunities for more efficient content-aware compression elsewhere in the data path.
- MaxBarraclough 7y agoIf we're going the content-aware route, can't the compression scheme just be Base64-aware?
- CamperBob2 7y agoHow exactly do you make a base64-aware image compressor, though?
- MaxBarraclough 7y agoI don't follow. An image compressor is guaranteed not to be given Base64 input. A general-purpose compression scheme can easily detect when its input is Base64, so I don't see why Base64 should be particularly hard to compress, in principle at least.