5 ms·

I don't think it's a C++ problem. You just can't transform anything developed in "ancient" times to unicode aware in a single swoop. On the other hand, libicu

I don't think it's a C++ problem. You just can't transform anything developed in "ancient" times to unicode aware in a single swoop. On the other hand, libicu is 37MB by itself, so it's not something someone can write in a weekend and ship. Any tool which is old enough will have a thousand ways to do something. This is the inevitability of software and programming languages. In the domain of C++, which has a size mammoth now, everyone expects this huge pony to learn new tricks, but everybody has a different idea of the "new tricks", so more features are added on top of its already impressive and very long list of features and capabilities. You want libICU built-in? There must be other folks who want that too. So you may need to find them and work with them to make your dream a reality. So, C++ is doing fine. It's not that they omitted Unicode during the design phase. Unicode has arrived later, and it has to be integrated by other means. This is what libraries for.

pornel 2y ago
Being developed in, and having to stay compatible with, ancient times is a real problem of C++. The now-invalid assumptions couldn't have been avoided 50 years ago. Fixing them now in C++ is difficult or impossible, but still, the end result is a ton of brokenness baked into C++. Languages developed in the 21st century typically have some at least half-decent Unicode support built-in. Unicode is big and complex, but there's a lot that a language can do to at least not silently destroy the encoding.
- cm2187 2y ago
  That explains why there are two functions, one for ascii and one for unicode. That doesn't explain why the unicode functions are hard to use (per the article).
  - BoringTimesGang 2y ago
    Because human language is hard to boil down to a simple computing model and the problem is underdefined, based on naive assumptions. Or perhaps I should say naïve.
    - cm2187 2y ago
      Well pretty much every other more recent language solved that problem.
      - kccqzy 2y ago
        Almost no programming language, perhaps other than Swift, solved that problem. Just use the article's examples as test cases. It's just as wrong as the C++ version in the article, except it's wrong with nicer syntax.
        zahlman 2y ago
        Python's strings have uppercase, lowercase and case-folding methods that don't choke on this. They don't use UTF-16 internally (they can use UCS-2 for strings whose code points will fit in that range; while a string might store code points from the surrogate-pair range, they're never interpreted as surrogate pairs, but instead as an error encoding so that e.g. invalid UTF-8 can be round-tripped) so they're never worried about surrogate pairs, and it knows a few things about localized text casing: >>> 'ß'.upper() 'SS' >>> 'ß'.lower() 'ß' >>> 'ß'.casefold() 'ss' There are a lot of really complicated tasks for Unicode strings. String casing isn't really one of them. (No, Python can't turn 'SS' back into 'ß'. But doing that requires metadata about language that a string simply doesn't represent.)
        kccqzy 2y ago
        Still breaks on, for example, Turkish i vs İ. It's impossible to do correctly without language information. > (No, Python can't turn 'SS' back into 'ß'. But doing that requires metadata about language that a string simply doesn't represent.) Yes that's my point. Because in typical languages strings don't store language metadata, this is impossible to do correctly in general.
        zahlman 2y ago
        I'm not seeing anything in the Swift documentation about strings carrying language metadata, either, though?
  - deleted 2y ago
    [deleted]
- account42 2y ago
  No, strong backwards compatiblity a real strength of C++. In fact, it's probably it's main strength these days.
relaxing 2y ago
It’s been 30 years. Unicode predates C++98. Java saw the writing on the wall. There’s no excuse.
- bayindirh 2y ago
  > There’s no excuse. I politely disagree. None of the programming languages which started integrating Unicode was targeting from bare metal to GUI, incl. embedded and OS development at the same time. C++ has a great target area when compared to other programming languages. There are widely used libraries which compile correctly on PDP-11s, even if they are updated constantly. You can't just say "I'll be just making everything Unicode aware, backwards compatibility be damned, eh".
  - blenderob 2y ago
    But we don't have to make everything Unicode aware. Backward compatibility is indeed very important in C++. Like you rightly said, it still has to work for PDP-11 without breaking anything. But the C++ overlords could always add a new type that is Unicode-aware. Converting one Unicode string to another is a purely in-memory, in-CPU operation. It does not need any I/O and it does not need any interaction with peripherals. So one can dream that such a type along with its conversion routines could be added to an updated standard library without breaking existing code that compiles correctly on PDP-11s.
    - bayindirh 2y ago
      > Converting one Unicode string to another is a purely in-memory, in-CPU operation. ...but it's a complex operation. This is what libICU is mostly for. You can't just look-up a single table and convert a string to another like you work on ASCII table or any other simple encoding. Germans have their ß to S (or capital ß depending on the year), Turkish has ı/I and i/İ pairs, and tons of other languages have other rules. Esp, this I/ı and İ/i pairs break tons of applications in very unexpected ways. I don't remember how many bugs I reported, and how many workarounds I have implemented in my systems. Adding a type is nice, but the surrounding machinery is so big, it brings tons of work with itself. Unicode is such a complicated system, that I read that even you need two UTF-16 characters (4 bytes in total) to encode a single character. This is insane (as in complexity, I guess they have their reasons).
akira2501 2y ago
> libicu is 37MB by itself, so it's not something someone can write in a weekend and ship. Isn't that mostly just from tables derived from the Unicode standard?
zahlman 2y ago
>You just can't transform anything developed in "ancient" times to unicode aware in a single swoop. Even for Python it took well over a decade, and people still complain about the fact that they don't get to treat byte-sequences transparently as text any more - as if they want to wrestle with the `basestring` supertype, getting `UnicodeDecodeError` from an encoding operation or vice-versa, trying to guess the encoding of someone else's data instead of expecting it to be decoded on the other side.... But in C++ (and in C), you have the additional problem that the 8-bit integer type was named for the concept of a character of text, even though it clearly cannot actually represent any such thing. (Not to mention the whole bit about `char` being a separate type from both `signed char` and `unsigned char`, without defined signedness.)
ectospheno 2y ago
> Any tool which is old enough will have a thousand ways to do something. Only because of the strange desire of programmers to never stop. Not every program is a never ending story. Most are short stories their authors bludgeon into a novel. Programming languages bloat into stupidity for the same reason. Nothing is ever removed. Programmers need editors.
- fluoridation 2y ago
  So how do you design a language that accommodates both the people who need a codebase to be stable for decades and the people who want the bleeding edge all the time, backwards compatibility be damned?
  - the_gorilla 2y ago
    You don't. Any language that tries to do both turns into an unusable abomination like C++. Good languages are stable and the bleeding edge is just the "new thing" and not necessarily better than the old thing.
    - fluoridation 2y ago
      C++ doesn't try to do that. It aims to remain as backwards compatible as possible, which is what the GP is complaining about.

pornel 2y ago

Being developed in, and having to stay compatible with, ancient times is a real problem of C++. The now-invalid assumptions couldn't have been avoided 50 years ago. Fixing them now in C++ is difficult or impossible, but still, the end result is a ton of brokenness baked into C++. Languages developed in the 21st century typically have some at least half-decent Unicode support built-in. Unicode is big and complex, but there's a lot that a language can do to at least not silently destroy the encoding.

cm2187 2y ago
That explains why there are two functions, one for ascii and one for unicode. That doesn't explain why the unicode functions are hard to use (per the article).
- BoringTimesGang 2y ago
  Because human language is hard to boil down to a simple computing model and the problem is underdefined, based on naive assumptions. Or perhaps I should say naïve.
  - cm2187 2y ago
    Well pretty much every other more recent language solved that problem.
    - kccqzy 2y ago
      Almost no programming language, perhaps other than Swift, solved that problem. Just use the article's examples as test cases. It's just as wrong as the C++ version in the article, except it's wrong with nicer syntax.
      - zahlman 2y ago
        Python's strings have uppercase, lowercase and case-folding methods that don't choke on this. They don't use UTF-16 internally (they can use UCS-2 for strings whose code points will fit in that range; while a string might store code points from the surrogate-pair range, they're never interpreted as surrogate pairs, but instead as an error encoding so that e.g. invalid UTF-8 can be round-tripped) so they're never worried about surrogate pairs, and it knows a few things about localized text casing: >>> 'ß'.upper() 'SS' >>> 'ß'.lower() 'ß' >>> 'ß'.casefold() 'ss' There are a lot of really complicated tasks for Unicode strings. String casing isn't really one of them. (No, Python can't turn 'SS' back into 'ß'. But doing that requires metadata about language that a string simply doesn't represent.)
        kccqzy 2y ago
        Still breaks on, for example, Turkish i vs İ. It's impossible to do correctly without language information. > (No, Python can't turn 'SS' back into 'ß'. But doing that requires metadata about language that a string simply doesn't represent.) Yes that's my point. Because in typical languages strings don't store language metadata, this is impossible to do correctly in general.
        zahlman 2y ago
        I'm not seeing anything in the Swift documentation about strings carrying language metadata, either, though?
- deleted 2y ago
  [deleted]
account42 2y ago
No, strong backwards compatiblity a real strength of C++. In fact, it's probably it's main strength these days.

relaxing 2y ago

It’s been 30 years. Unicode predates C++98. Java saw the writing on the wall. There’s no excuse.

bayindirh 2y ago
> There’s no excuse. I politely disagree. None of the programming languages which started integrating Unicode was targeting from bare metal to GUI, incl. embedded and OS development at the same time. C++ has a great target area when compared to other programming languages. There are widely used libraries which compile correctly on PDP-11s, even if they are updated constantly. You can't just say "I'll be just making everything Unicode aware, backwards compatibility be damned, eh".
- blenderob 2y ago
  But we don't have to make everything Unicode aware. Backward compatibility is indeed very important in C++. Like you rightly said, it still has to work for PDP-11 without breaking anything. But the C++ overlords could always add a new type that is Unicode-aware. Converting one Unicode string to another is a purely in-memory, in-CPU operation. It does not need any I/O and it does not need any interaction with peripherals. So one can dream that such a type along with its conversion routines could be added to an updated standard library without breaking existing code that compiles correctly on PDP-11s.
  - bayindirh 2y ago
    > Converting one Unicode string to another is a purely in-memory, in-CPU operation. ...but it's a complex operation. This is what libICU is mostly for. You can't just look-up a single table and convert a string to another like you work on ASCII table or any other simple encoding. Germans have their ß to S (or capital ß depending on the year), Turkish has ı/I and i/İ pairs, and tons of other languages have other rules. Esp, this I/ı and İ/i pairs break tons of applications in very unexpected ways. I don't remember how many bugs I reported, and how many workarounds I have implemented in my systems. Adding a type is nice, but the surrounding machinery is so big, it brings tons of work with itself. Unicode is such a complicated system, that I read that even you need two UTF-16 characters (4 bytes in total) to encode a single character. This is insane (as in complexity, I guess they have their reasons).

akira2501 2y ago

> libicu is 37MB by itself, so it's not something someone can write in a weekend and ship. Isn't that mostly just from tables derived from the Unicode standard?

zahlman 2y ago

>You just can't transform anything developed in "ancient" times to unicode aware in a single swoop. Even for Python it took well over a decade, and people still complain about the fact that they don't get to treat byte-sequences transparently as text any more - as if they want to wrestle with the `basestring` supertype, getting `UnicodeDecodeError` from an encoding operation or vice-versa, trying to guess the encoding of someone else's data instead of expecting it to be decoded on the other side.... But in C++ (and in C), you have the additional problem that the 8-bit integer type was named for the concept of a character of text, even though it clearly cannot actually represent any such thing. (Not to mention the whole bit about `char` being a separate type from both `signed char` and `unsigned char`, without defined signedness.)

ectospheno 2y ago

> Any tool which is old enough will have a thousand ways to do something. Only because of the strange desire of programmers to never stop. Not every program is a never ending story. Most are short stories their authors bludgeon into a novel. Programming languages bloat into stupidity for the same reason. Nothing is ever removed. Programmers need editors.

fluoridation 2y ago
So how do you design a language that accommodates both the people who need a codebase to be stable for decades and the people who want the bleeding edge all the time, backwards compatibility be damned?
- the_gorilla 2y ago
  You don't. Any language that tries to do both turns into an unusable abomination like C++. Good languages are stable and the bleeding edge is just the "new thing" and not necessarily better than the old thing.
  - fluoridation 2y ago
    C++ doesn't try to do that. It aims to remain as backwards compatible as possible, which is what the GP is complaining about.