4 ms·
They work well as long as you're fine working with bytes. For "characters" which a user sees on the screen, that is, graphemes, you need an entirely new layer.
by nine_k 3y ago
They work well as long as you're fine working with bytes. For "characters" which a user sees on the screen, that is, graphemes, you need an entirely new layer. Take some word, e.g. "éclair". How long is it? What are its first three characters? How do you uppercase it?
- kragen 3y agothose are library functions, and they work fine on utf-8 strings, though graphemes in particular are difficult and context-dependent in unicode in a way that is exactly the same in c and in java
- amluto 3y agoThey are library functions for which a good library does not exist. I recently needed to convert probably-UTF-8 data to definitely valid UTF-8 with errors replaced. This was not an enjoyable experience in C++. (The ztd proposal is IMO a big step in the right direction.)
- flohofwoe 3y agoStuff like this is handled in a (3rd-party) UNICODE library in the C/C++ world, which should ideally work on UTF-8 encoded byte arrays, provided by another (3rd-party) UTF-8 encoding/decoding library. Other then that high-level UNICODE stuff (like finding grapheme cluster boundaries) UTF-8 itself really works fine in C/C++ anywhere than Windows in the sense that I can write a foreign-language "Hello World!" and it "just works" (e.g. if the whole source file is UTF-8 encoded anyway, than C string literals are also automatically valid UTF-8 strings). UNICODE on Windows is still a bigger mess than it should be because of its UCS-2 / UTF-16 heritage.
- slaymaker1907 3y agoStuff gets really nasty when you start trying to reason about case insensitive string comparisons. For example, the following might return something different depending on what your locale is: "π".localeCompare("Π", undefined, { sensitivity: "accent" }) My machine says these are equal, but I've seen cases where network stacks consider domains as different if they feature the same Greek letter in a different case even though domain names are supposed to be case-insensitive.
- samus 3y agoTo answer all of these, you need `libicu`, not just a mere UTF-8 decoder. Java also doesn't include full facilities: it's `BreakIterator` is based on an ancient Unicode version.