3 ms·
Unrelated, but is it my imagination or all string algorithms would benefit from UTF-32 over UTF-8? UTF-32 always uses 32 bits per character, so you can easily
by AlienRobot 3y ago
Unrelated, but is it my imagination or all string algorithms would benefit from UTF-32 over UTF-8?
UTF-32 always uses 32 bits per character, so you can easily paralelize it because you can easily tell where a character starts in a bit stream.
UTF-16 can have 16 or 32 bits, so if you have a sequence 2 characters of 16-32 bits, or in UTF-8's case, a sequence of 3 characters that are 8-16-16 bits, they won't fit neatly in a box. You can't just say char_ptr + 32 to land on the 32nd character, making it much harder to paralelize.
- layer8 3y agoRead https://unicode.org/faq/utf_bom.html#utf32-2 https://unicode.org/faq/utf_bom.html#utf32-2. In addition, UTF-32 can cause issues on the network due to byte ordering (little endian vs. big endian). There’s really two encodings, UTF-32LE and UTF-32BE. For all of these reasons, the world has largely standardized on UTF-8, plus for historical reasons UTF-16 in the JVM, CLR, Windows, and JavaScript.