3 ms·

Wow, branch prediction is actually a pretty good point I hadn't thought about.

• wahern 8y ago
  It's common, though, to decode to an internal 16- or even 32-bit representation. Combined with clever UTF-8 decoding state machines the overhead is minimal, whether you do this all at once or in a streaming, segmented fashion. Rare is the circumstance where you can just load UTF-16 directly into an array and go to town without preprocessing. In most application scenarios you need to translate to a normalization form (i.e. NFC, NFD, etc), which you can simply combine with your UTF-8 decoding. Such tight loops of arithmetic code are blazingly fast, and performance can improve dramatically by batching these steps separately rather than intermixing different phases of text processing in your application. Perl6 decodes to NFG form which provides a 1:1 mapping between characters and scalar values.[1] That makes Unicode string munging almost as easy as ASCII.[2] But it requires always preprocessing and intern'ing your strings. [1] Multi-codepoint graphemes are dynamically assigned a unique internal codepoint. [2] I say almost because some Unicode text rules, like those for word and paragraph breaks, can never be implemented as simple as in ASCII.