4 ms·

> what are their arguments? It’s often faster. When you process non-English text in UTF8, branch prediction fails all the time because all characters have rand

> what are their arguments? It’s often faster. When you process non-English text in UTF8, branch prediction fails all the time because all characters have random size from 1 to 3 bytes. When you process non-English text in UTF16, branch prediction predicts all the time: surrogate pairs are extremely rare so the 99.9% of the stream is 2 bytes/character. > it's not compatible with ascii and C-strings like UTF-8 In modern C++, wchar_t is a built-in fundamental type: http://www.cplusplus.com/reference/cwchar/wchar_t/ http://www.cplusplus.com/reference/cwchar/wchar_t/

jmgao 8y ago
> In modern C++, wchar_t is a built-in fundamental type: http://www.cplusplus.com/reference/cwchar/wchar_t/ http://www.cplusplus.com/reference/cwchar/wchar_t/ But with the slight problem that it actually means UTF-32 on everything on non-Windows (there's char16_t (C++11) and char32_t (C++20), but somewhat obnoxiously, they're new types, so you need explicit casts for pointer conversion)
- int_19h 8y ago
  It's even worse - it means "wide character", which is not necessarily Unicode. E.g. on all BSDs, if your locale is something like zh_TW-Big5, then wchar_t stores Big5 codepoints! You can use it to store UTF-32, of course, but all the wcs* standard functions won't treat it as such. Now, there's __STDC_ISO_10646__, which, if it is defined by an implementation, means that wchar_t is guaranteed to be a Unicode codepoint. So e.g. glibc defines that, and on Linux you can therefore assume wchar_t as UTF-32 regardless of locale. But BSDs don't. And on Windows, nothing that uses 16-bit wchar_t can use that define, since 16 bits aren't wide enough to be conforming.
kupiakos 8y ago
Wow, branch prediction is actually a pretty good point I hadn't thought about.
- wahern 8y ago
  It's common, though, to decode to an internal 16- or even 32-bit representation. Combined with clever UTF-8 decoding state machines the overhead is minimal, whether you do this all at once or in a streaming, segmented fashion. Rare is the circumstance where you can just load UTF-16 directly into an array and go to town without preprocessing. In most application scenarios you need to translate to a normalization form (i.e. NFC, NFD, etc), which you can simply combine with your UTF-8 decoding. Such tight loops of arithmetic code are blazingly fast, and performance can improve dramatically by batching these steps separately rather than intermixing different phases of text processing in your application. Perl6 decodes to NFG form which provides a 1:1 mapping between characters and scalar values.[1] That makes Unicode string munging almost as easy as ASCII.[2] But it requires always preprocessing and intern'ing your strings. [1] Multi-codepoint graphemes are dynamically assigned a unique internal codepoint. [2] I say almost because some Unicode text rules, like those for word and paragraph breaks, can never be implemented as simple as in ASCII.
udp 8y ago
Correct me if I'm wrong: UTF-16 can be faster if you're iterating through characters, but for parsing iterating through bytes is usually sufficient. It's fine to use something like plain old strchr to search for a { or a < in a UTF-8 string. Characters only get really important when you want to print the string.
- Const-me 8y ago
  > It's fine to use something like plain old strchr to search for a { or a < in a UTF-8 string. Correct. > only get really important when you want to print the string. Print, split into fixed-length pieces, render, layout, typeset — GUI apps do that a lot with their strings. Even web browser do, probably that’s why JavaScript strings are UTF-16 despite web in general is mostly UTF-8, see section 6.1.4 “The String Type” on page 67: http://www.ecma-international.org/publications/files/ECMA-ST/Ecma-262.pdf http://www.ecma-international.org/publications/files/ECMA-ST...
  - udp 8y ago
    Very true, but once you're at the level of complexity where you're rendering a GUI are you really concerned about cache invalidation due to incorrect branch prediction iterating strings? GUI rendering is very expensive for lots of reasons, but I don't believe that's one of them - even in the UTF-8 case. You're ultimately either drawing thousands of pixels or communicating over a (relatively compared to the CPU cache) slow bus with the GPU.
    - Const-me 8y ago
      > You're ultimately either drawing thousands of pixels or communicating over a (relatively compared to the CPU cache) slow bus with the GPU. In modern software, GPU draws pixels. Before it does that, CPU lays out these glyphs. Because GPUs are ridiculously fast these days, the layout step is typically slower than painting. I use MS edge browser. The built-in profiler said this comments page took 14ms to layout and only 6ms to paint. This page contains just a few tiny images, the majority of the content is text. I think the layout step spent most of the time iterating over the characters, looking up glyphs and measuring various blocks of text on this page.

jmgao 8y ago

> In modern C++, wchar_t is a built-in fundamental type: http://www.cplusplus.com/reference/cwchar/wchar_t/ http://www.cplusplus.com/reference/cwchar/wchar_t/ But with the slight problem that it actually means UTF-32 on everything on non-Windows (there's char16_t (C++11) and char32_t (C++20), but somewhat obnoxiously, they're new types, so you need explicit casts for pointer conversion)

int_19h 8y ago
It's even worse - it means "wide character", which is not necessarily Unicode. E.g. on all BSDs, if your locale is something like zh_TW-Big5, then wchar_t stores Big5 codepoints! You can use it to store UTF-32, of course, but all the wcs* standard functions won't treat it as such. Now, there's __STDC_ISO_10646__, which, if it is defined by an implementation, means that wchar_t is guaranteed to be a Unicode codepoint. So e.g. glibc defines that, and on Linux you can therefore assume wchar_t as UTF-32 regardless of locale. But BSDs don't. And on Windows, nothing that uses 16-bit wchar_t can use that define, since 16 bits aren't wide enough to be conforming.

kupiakos 8y ago

Wow, branch prediction is actually a pretty good point I hadn't thought about.

wahern 8y ago
It's common, though, to decode to an internal 16- or even 32-bit representation. Combined with clever UTF-8 decoding state machines the overhead is minimal, whether you do this all at once or in a streaming, segmented fashion. Rare is the circumstance where you can just load UTF-16 directly into an array and go to town without preprocessing. In most application scenarios you need to translate to a normalization form (i.e. NFC, NFD, etc), which you can simply combine with your UTF-8 decoding. Such tight loops of arithmetic code are blazingly fast, and performance can improve dramatically by batching these steps separately rather than intermixing different phases of text processing in your application. Perl6 decodes to NFG form which provides a 1:1 mapping between characters and scalar values.[1] That makes Unicode string munging almost as easy as ASCII.[2] But it requires always preprocessing and intern'ing your strings. [1] Multi-codepoint graphemes are dynamically assigned a unique internal codepoint. [2] I say almost because some Unicode text rules, like those for word and paragraph breaks, can never be implemented as simple as in ASCII.

udp 8y ago

Correct me if I'm wrong: UTF-16 can be faster if you're iterating through characters, but for parsing iterating through bytes is usually sufficient. It's fine to use something like plain old strchr to search for a { or a < in a UTF-8 string. Characters only get really important when you want to print the string.

Const-me 8y ago
> It's fine to use something like plain old strchr to search for a { or a < in a UTF-8 string. Correct. > only get really important when you want to print the string. Print, split into fixed-length pieces, render, layout, typeset — GUI apps do that a lot with their strings. Even web browser do, probably that’s why JavaScript strings are UTF-16 despite web in general is mostly UTF-8, see section 6.1.4 “The String Type” on page 67: http://www.ecma-international.org/publications/files/ECMA-ST/Ecma-262.pdf http://www.ecma-international.org/publications/files/ECMA-ST...
- udp 8y ago
  Very true, but once you're at the level of complexity where you're rendering a GUI are you really concerned about cache invalidation due to incorrect branch prediction iterating strings? GUI rendering is very expensive for lots of reasons, but I don't believe that's one of them - even in the UTF-8 case. You're ultimately either drawing thousands of pixels or communicating over a (relatively compared to the CPU cache) slow bus with the GPU.
  - Const-me 8y ago
    > You're ultimately either drawing thousands of pixels or communicating over a (relatively compared to the CPU cache) slow bus with the GPU. In modern software, GPU draws pixels. Before it does that, CPU lays out these glyphs. Because GPUs are ridiculously fast these days, the layout step is typically slower than painting. I use MS edge browser. The built-in profiler said this comments page took 14ms to layout and only 6ms to paint. This page contains just a few tiny images, the majority of the content is text. I think the layout step spent most of the time iterating over the characters, looking up glyphs and measuring various blocks of text on this page.