4 ms·
In theory, yes. In practice, no. At my previous job, I wrote a short Python script that took /usr/dict/words and gzipped it and also converted to UTF-16LE (in
by KMag 4y ago
In theory, yes. In practice, no. At my previous job, I wrote a short Python script that took /usr/dict/words and gzipped it and also converted to UTF-16LE (inserting a null byte before every character) and gzipping that. The information content is the same, but the compressed UTF-16LE ends up a bit bigger. IIRC, the difference was more than 1%, but less than 20%.
My use case was to show the flaw in the logic a colleague was using to assert that gzipped JSON should be the same size as gzipped MessagePack for the same data, because the information content was the same. It was a quick 5-minute script without having to deal with coming up with a suitable JSON corpus to convert to MessagePack.
Among other things, the zlib compression window only holds half as many characters if your characters are twice as big.
- KMag 4y agoFor anyone still reading, out of curiosity, I reran the experiment on my Debian box: $ </usr/share/dict/words gzip --best | wc -c 261255 $ </usr/share/dict/words iconv -f utf-8 -t utf-16le | gzip --best | wc -c 303404 A bit over a 16% size increase form converting the "wamerican" dictionary to UTF-16LE and then compressing.
- teddyh 4y agoFor the large dictionary it’s a tiny bit worse, but still rounds to 16%: $ < /usr/share/dict/american-english-insane gzip --best | wc --bytes 1778330 $ < /usr/share/dict/american-english-insane iconv -f utf-8 -t utf-16le | gzip --best | wc --bytes 2061457