2 ms·

> Indexing & substrings are common, too. Indexing code points in both UTF-8 and UTF-16 requires reading the whole string up to index location. Substrings are t

> Indexing & substrings are common, too. Indexing code points in both UTF-8 and UTF-16 requires reading the whole string up to index location. Substrings are the same as well. > Right, and for 1 billion Chinese speaking people UTF16 is 2 bytes/character, UTF8 is 3 bytes/character. That's true for a text file without markup. But most text is not like that in 2017. HTML is probably the most common text format nowadays. So let's see how a popular Chinese language website does. curl http://language.chinadaily.com.cn/ --silent | wc -c 52678 curl http://language.chinadaily.com.cn/ --silent | iconv -f utf8 -t utf-16le | wc -c 93368 So UTF-8 seems to be quite a bit more efficient in this case, 52678 bytes. When converted to UTF-16, same page was 93368 bytes.

Const-me 9y ago
Even in 2017, not everyone is a Web or Electron developer. I certainly am not. I don’t advocate using UTF16 for the web, but people still code native desktop apps, mobile apps, embedded software, videogames, store stuff in various databases, etc. For such use, markup is irrelevant.
- banthar 9y ago
  Even outside Web, you still have mostly-ASCII: * filenames * identifiers * config files * text protocols * host names, email addresses * embedded scripts (including SQL and OpenGL shaders) * command line interfaces * translations for languages using Latin alphabets I don't think 2/3 size reduction for some languages will offset the cost in all the other places.
  - pjmlp 9y ago
    If English is your world yeah. Some of us use other languages and like to use them everywhere we can.
  - Const-me 9y ago
    For some of these things we don’t have much choice, because the encoding is part of some lower-level API (file system, OpenGL, CLI), which usually don’t accept arbitrary encoding. They accept only one, and unless you want to waste time converting, you better use that exact encoding. Other stuff like IDs, shaders before GL 4.2, and many text protocols aren’t Unicode at all. For configs I usually use UTF-8 myself, because I don’t like writing parsers for custom formats and just use XML, and any standard-compliant parser supports all of them.
Ded7xSEoPKYNsDd 9y ago
> Indexing code points in both UTF-8 and UTF-16 requires reading the whole string up to index location. Substrings are the same as well. Java's String functions don't index by Unicode code points, though. Java strings are encoded in UCS-2, or at least the API needs to pretend that they are.
- vardump 9y ago
  That can sure lead to interesting bugs!
- Const-me 9y ago
  Right. Same in C#, C++ STL, and in Apple’s obj-c/swift.
  - tjalfi 9y ago
    Swift takes a different approach than Objective C[0]. [0] https://www.mikeash.com/pyblog/friday-qa-2015-11-06-why-is-swifts-string-api-so-hard.html https://www.mikeash.com/pyblog/friday-qa-2015-11-06-why-is-s...

Const-me 9y ago

Even in 2017, not everyone is a Web or Electron developer. I certainly am not. I don’t advocate using UTF16 for the web, but people still code native desktop apps, mobile apps, embedded software, videogames, store stuff in various databases, etc. For such use, markup is irrelevant.

banthar 9y ago
Even outside Web, you still have mostly-ASCII: * filenames * identifiers * config files * text protocols * host names, email addresses * embedded scripts (including SQL and OpenGL shaders) * command line interfaces * translations for languages using Latin alphabets I don't think 2/3 size reduction for some languages will offset the cost in all the other places.
- pjmlp 9y ago
  If English is your world yeah. Some of us use other languages and like to use them everywhere we can.
- Const-me 9y ago
  For some of these things we don’t have much choice, because the encoding is part of some lower-level API (file system, OpenGL, CLI), which usually don’t accept arbitrary encoding. They accept only one, and unless you want to waste time converting, you better use that exact encoding. Other stuff like IDs, shaders before GL 4.2, and many text protocols aren’t Unicode at all. For configs I usually use UTF-8 myself, because I don’t like writing parsers for custom formats and just use XML, and any standard-compliant parser supports all of them.

Ded7xSEoPKYNsDd 9y ago

> Indexing code points in both UTF-8 and UTF-16 requires reading the whole string up to index location. Substrings are the same as well. Java's String functions don't index by Unicode code points, though. Java strings are encoded in UCS-2, or at least the API needs to pretend that they are.

vardump 9y ago
That can sure lead to interesting bugs!
Const-me 9y ago
Right. Same in C#, C++ STL, and in Apple’s obj-c/swift.
- tjalfi 9y ago
  Swift takes a different approach than Objective C[0]. [0] https://www.mikeash.com/pyblog/friday-qa-2015-11-06-why-is-swifts-string-api-so-hard.html https://www.mikeash.com/pyblog/friday-qa-2015-11-06-why-is-s...