4 ms·

> right now the thinking about strings for most of the world is very C-brained, very Python 2 Is it? Doesn't pretty much every language have a unicode string t

> right now the thinking about strings for most of the world is very C-brained, very Python 2 Is it? Doesn't pretty much every language have a unicode string type (be that UTF16 in older languages or UTF8 in newer one) that is the default goto type for dealing with text these days? C and C++ being the notable exceptions I suppose.

kragen 3y ago
utf-8 strings work fine in c and c++, as they have since utf-8 was introduced; that was the major design objective of utf-8 in fact
- nine_k 3y ago
  They work well as long as you're fine working with bytes. For "characters" which a user sees on the screen, that is, graphemes, you need an entirely new layer. Take some word, e.g. "éclair". How long is it? What are its first three characters? How do you uppercase it?
  - kragen 3y ago
    those are library functions, and they work fine on utf-8 strings, though graphemes in particular are difficult and context-dependent in unicode in a way that is exactly the same in c and in java
    - amluto 3y ago
      They are library functions for which a good library does not exist. I recently needed to convert probably-UTF-8 data to definitely valid UTF-8 with errors replaced. This was not an enjoyable experience in C++. (The ztd proposal is IMO a big step in the right direction.)
  - flohofwoe 3y ago
    Stuff like this is handled in a (3rd-party) UNICODE library in the C/C++ world, which should ideally work on UTF-8 encoded byte arrays, provided by another (3rd-party) UTF-8 encoding/decoding library. Other then that high-level UNICODE stuff (like finding grapheme cluster boundaries) UTF-8 itself really works fine in C/C++ anywhere than Windows in the sense that I can write a foreign-language "Hello World!" and it "just works" (e.g. if the whole source file is UTF-8 encoded anyway, than C string literals are also automatically valid UTF-8 strings). UNICODE on Windows is still a bigger mess than it should be because of its UCS-2 / UTF-16 heritage.
    - slaymaker1907 3y ago
      Stuff gets really nasty when you start trying to reason about case insensitive string comparisons. For example, the following might return something different depending on what your locale is: "π".localeCompare("Π", undefined, { sensitivity: "accent" }) My machine says these are equal, but I've seen cases where network stacks consider domains as different if they feature the same Greek letter in a different case even though domain names are supposed to be case-insensitive.
  - samus 3y ago
    To answer all of these, you need `libicu`, not just a mere UTF-8 decoder. Java also doesn't include full facilities: it's `BreakIterator` is based on an ancient Unicode version.
microtherion 3y ago
C++ has had std::u8string, std::u16string and std::u32string since C++11

kragen 3y ago

utf-8 strings work fine in c and c++, as they have since utf-8 was introduced; that was the major design objective of utf-8 in fact

nine_k 3y ago
They work well as long as you're fine working with bytes. For "characters" which a user sees on the screen, that is, graphemes, you need an entirely new layer. Take some word, e.g. "éclair". How long is it? What are its first three characters? How do you uppercase it?
- kragen 3y ago
  those are library functions, and they work fine on utf-8 strings, though graphemes in particular are difficult and context-dependent in unicode in a way that is exactly the same in c and in java
  - amluto 3y ago
    They are library functions for which a good library does not exist. I recently needed to convert probably-UTF-8 data to definitely valid UTF-8 with errors replaced. This was not an enjoyable experience in C++. (The ztd proposal is IMO a big step in the right direction.)
- flohofwoe 3y ago
  Stuff like this is handled in a (3rd-party) UNICODE library in the C/C++ world, which should ideally work on UTF-8 encoded byte arrays, provided by another (3rd-party) UTF-8 encoding/decoding library. Other then that high-level UNICODE stuff (like finding grapheme cluster boundaries) UTF-8 itself really works fine in C/C++ anywhere than Windows in the sense that I can write a foreign-language "Hello World!" and it "just works" (e.g. if the whole source file is UTF-8 encoded anyway, than C string literals are also automatically valid UTF-8 strings). UNICODE on Windows is still a bigger mess than it should be because of its UCS-2 / UTF-16 heritage.
  - slaymaker1907 3y ago
    Stuff gets really nasty when you start trying to reason about case insensitive string comparisons. For example, the following might return something different depending on what your locale is: "π".localeCompare("Π", undefined, { sensitivity: "accent" }) My machine says these are equal, but I've seen cases where network stacks consider domains as different if they feature the same Greek letter in a different case even though domain names are supposed to be case-insensitive.
- samus 3y ago
  To answer all of these, you need `libicu`, not just a mere UTF-8 decoder. Java also doesn't include full facilities: it's `BreakIterator` is based on an ancient Unicode version.

microtherion 3y ago

C++ has had std::u8string, std::u16string and std::u32string since C++11