4 ms·
> For instance, if it were the case that it did a bad job of encoding Mandarin I don't know if you picked this example on purpose, but using UTF-8 to encode Ch
by kccqzy 3y ago
> For instance, if it were the case that it did a bad job of encoding Mandarin
I don't know if you picked this example on purpose, but using UTF-8 to encode Chinese is 50% larger than the old encoding (GB2312). I remember people cared about this like twenty years ago. I don't know of anyone that still cares about this encoding inefficiency. Any compression algorithm is able to remove such encoding inefficiency while using negligible CPU to decompress.
- PeterisP 3y agoA key aspect is that nowadays we rarely encode pure text - while other encodings are more efficient for encoding pure Mandarin, nowadays a "Mandarin document" may be an HTML or JSON or XML file where less than half of the characters are from CJK codespace, and the rest come from all the formatting overhead which is in the 7-bit ASCII range, and UTF-8 works great for such combined content.
- tomoyoirl 3y agoTo say nothing of mod_gzip
- kstrauser 3y agoThat doesn't seem like the worst issue imaginable. I doubt there are too many cases where every byte counts, text uses a significant portion of the available space, and compression is unavailable or inefficient. If we were still cramming floppies full of text files destined for very slow computers, that'd be one thing. Web pages full of uncompressed text are still either so small that it's a moot point or so huge with JS, images, and fonts that the relative text size isn't that significant. Which is all to say that you're right, but I can't imagine that it's more than a theoretical nuisance outside some extremely niche cases.
- bawolff 3y ago> Web pages full of uncompressed text Are basically non existent. Almost all modern web servers transparently compress html. Sending uncompressed text over the wire is extremely rare.
- iraqmtpizza 3y agoThey shouldn't be non-existent. Zip-then-encrypt is not secure due to information leakage. EDIT: also, it's not safe—message length is dependent on the values of the plaintext bytes, period. i'm not saying don't live dangerously, i'm just saying live dangerously knowing
- fanf2 3y agoThe information leakage problem occurs when compression is done in the TLS layer, because then the compression context includes both headers (with cookies) and bodies (containing potentially attacker-controlled data). But if you do compression at the HTTP layer using its Transfer-Encoding then the compression context only covers the body, which is safe.
- rocqua 3y agoIt can still leak data if attackers can get their input reflected. I.e. I send you a word, and then I get to observe a compressed and encrypted message including my word and sensitive data. If my word matches the sensitive data, the cyphertext will be smaller. Hence I can learn things about the cipgertext. That is no longer good encryption.
- kstrauser 3y agoDo you often send raw bitmaps for the same reason?
- iraqmtpizza 3y agoDo you often get completely pwned and have your encrypted calls transcribed by people eating doughnuts because you thought it was safe to compress sensitive data before encrypting? https://web.archive.org/web/20080901185111/https://technology.newscientist.com/channel/tech/dn14124-compressed-web-phone-calls-are-easy-to-bug.html https://web.archive.org/web/20080901185111/https://technolog...
- bawolff 3y ago
- gary_0 3y agoFor Asian languages, UTF-8 is basically the same size as any other encoding when compressed[0] (and you should be using compression if you care about space) so in practice there is no data size advantage to using non-standard encodings. [0] https://utf8everywhere.org/#asian https://utf8everywhere.org/#asian
- neild 3y agoIn addition, Chinese characters encode more information than English letters, so a text written in Chinese will generally consume fewer bytes than the same text in English even when using UTF-8. (Consider: Horse is five letters, but 馬 is one character. Even at three bytes per character, Chinese wins.)
- kstrauser 3y agoTrue, but even then you wouldn’t want to store it egregiously badly.
- Panzer04 3y agoPresumably that derives from the overhead of encoding an english character as a full byte? Given there's only 26 characters normally, you could fit that into 5 bits instead, which funnily enough does actually line up with the chinese character encoding (5x5 vs 1x24).
- kps 3y agoYes. It's the non-Latin alphabets that lose with either UTF-8 or UTF-16, compared with stateful ISO 2022 page switching.