4 ms·
I'll throw this out there too: UTF-8 isn't necessarily better than UTF-16; they both support the entirety of the Unicode character space. UTF-8 is convenient o
by chungy 2y ago
I'll throw this out there too: UTF-8 isn't necessarily better than UTF-16; they both support the entirety of the Unicode character space.
UTF-8 is convenient on Unix systems since it fits into 8-bit character slots that were already in place; file systems have traditionally only forbidden the NULL byte and forward-slash, and all other characters are valid. From this fact, you can use UTF-8 in file names with ease on legacy systems, you don't need any operating system support for it.
UTF-8 is "space optimized" for ASCII text, while most extra-ASCII Latin, Cyrillic, Greek, Arabic characters need two bytes each (same as UTF-16); most of Chinese/Japanese/Korean script in the BMP requires three bytes in UTF-8, whereas you still only need two bytes in UTF-16. To go further beyond, all SMP characters (eg, most emoji) require four bytes each in both systems.
Essentially, UTF-8 is good space-wise for mostly-ASCII text. It remains on-par with UTF-16 for most western languages, and only becomes more inefficient than UTF-16 for east-Asian languages (in such regions, UTF-16 is already dominant).
- bawolff 2y agoSpace savings are irrelavent. Text is small, and after gzip its going to be the same anyways. Seriously, when was the last time where you both cared about saving a single kb, but compression was not an option. I'm pretty sure the answer is never. It was never in the 90s, it is extra never now that we have hard drives hundreds of gb big. UTF-8 is better because bytes are a natural unit. You dont have to worry about surrogates. You dont have to worry about de-synchronization issues. You dont have to worry about byte order. Backwards compatibility with ascii and basically every interface ever, also helps. (Well not 7bit smtp..). The fact is, ascii is everywhere. Being able to treat it as just a subset of utf-8 makes a lot of things easier. > (eg, most emoji) require four bytes each in both systems. This is misleading because most emoiji are not just a single astral character but a combination of many.
- nmz 2y agoSaving disk space and synchronization is only important for network transmission. At the local level, you will need to convert to something where you can get/know positioning, utf8 does not allow for this given its variability, this means a lot of operations are more expensive and you will have to convert to utf16 anyway.
- mmoskal 2y agoInterestingly in a Javascript or similar runtime most of text that hits the caches where the size actually matters is still ASCII even in far east because of identifiers. Utf8 for the win!