3 ms·
> Why use both [UTF-8 and UTF-16]? Western languages typically are most efficiently encoded with UTF-8 (since most characters would be represented with 1 byte o
by ken 8y ago
> Why use both [UTF-8 and UTF-16]? Western languages typically are most efficiently encoded with UTF-8 (since most characters would be represented with 1 byte only), while Asian languages can usually produce smaller files when using UTF-16 as encoding.
The second sentence is technically correct, but it's a strange followup here because it's not why UTF-8 and UTF-16 exist today. I don't know any Asian webpages that use UTF-16 to save bandwidth, e.g., Japanese Wikipedia is still UTF-8.
The major use of UTF-16 in 2019, AFAICT, is for legacy operating system interfaces.
- ameliaquining 8y agoAlso "legacy" language runtimes. "Legacy" being in scare quotes because JavaScript, the JVM, and the CLR all work this way and are all very much in widespread use today.
- ken 8y agoYes, indeed. I didn't mean "legacy" to imply that they aren't still useful. Simply that many of them predate the creation of codepoints which don't fit in 16 bits -- and it's harder to upgrade a binary API distributed in a client OS, compared to changing a web server's preferred encoding. JS/JVM/CLR all work fine, but I imagine if they were created today, their strings would not be based on UTF-16.
- sebazzz 8y agoUtf8String is in the works for C#/.NET. It would still be pretty much bolted on though.
- Someone 8y agoJDK 9 introduced “compact strings” (https://bugs.openjdk.java.net/browse/JDK-8054307 https://bugs.openjdk.java.net/browse/JDK-8054307). That stores a string’s characters in a byte array, with either the traditional two bytes per ‘char’ (encoding the entire string as UTF16) or, if possible, one (encoding the entire string as ISO-8859-1/Latin-1). They probably didn’t use UTF-8 because it would break the fact that string indexing is O(1).
- int_19h 8y agoThat's just internal representation, though. Semantically, strings are still sequences of chars, and char is still 16-bit, so the API is still UTF-16.
- mjevans 8y agoWell, the second sentence isn't often correct. It'd probably be true for pure TEXT files IIRC, however the markup elements of webpages and javascript also take up space and /those/ are western elements. On actual web pages UTF-8 is actually the smallest choice.
- profmonocle 8y agoI imagine compression negates most of the overhead from encoding Asian web pages in UTF-8. Plus, considering how image/markup/script heavy modern web pages are, the actual human-readable text is a tiny sliver of most sites' bandwidth costs.