12 ms·

It is issues like this due to which I gave up on C++. There are so many ways to do something and every way is freaking wrong! An acceptable solution is given a

It is issues like this due to which I gave up on C++. There are so many ways to do something and every way is freaking wrong! An acceptable solution is given at the end of the article: > If you use the International Components for Unicode (ICU) library, you can use u_strToUpper and u_strToLower. Makes you wonder why this isn't part of the C++ standard library itself. Every revision of the C++ standard brings with itself more syntax and more complexity in the language. But as a user of C++ I don't need more syntax and more complexity in the language. But I do need more standard library functions that solves these ordinary real-world programming problems.

BoringTimesGang 2y ago
>It is issues like this due to which I gave up on C++. There are so many ways to do something and every way is freaking wrong! These are mostly unicode or linguistics problems.
- tralarpa 2y ago
  The fact that the standard library works against you doesn't help (to_lower takes an int, but only kind of works (sometimes) correctly on unsigned char, and wchar_t is implicitly promoted to int).
  - BoringTimesGang 2y ago
    to_lower is in the std namespace but is actually just part of the C89 standard, meaning it predates both UTF8 and UTF16. Is the alternative that it should be made unusable, and more existing code broken? A modern user has to include one of the c-prefix headers to use it, already hinting to them that 'here be dragons'. But there are always dragons. It's strings. The mere assumption that they can be transformed int-by-int, irrespective of encoding, is wrong. As is the assumption that a sensible transformation to lower case without error handling exists.
    - account42 2y ago
      > Is the alternative that it should be made unusable, and more existing code broken? It should be marked [[deprecated]], yes. There is no good reason to use std::tolower/toupper anywhere - they can neither do unicode properly nor are they anywhere close to efficient for ASCII. And their behavior depends on the process-global locale.
pistoleer 2y ago
> There are so many ways to do something and every way is freaking wrong! That's life! The perfect way does not exist. The best you can do is be aware of the tradeoffs, and languages like C++ absolutely throw them in your face at every single opportunity. It's fatiguing, and writing in javascript or python allows us to uphold the facade that everything is okay and that we don't have to worry about a thing.
- deleted 2y ago
  [deleted]
- pornel 2y ago
  JS and Python are still old enough to have been created when Unicode was in its infancy, so they have their own share of problems from using UCS-2 (such as indexing strings by what is now a UTF-16 code unit, rather than by a codepoint or a grapheme cluster). Swift has been developed in the modern times, and it's able to tackle Unicode properly, e.g. makes distinction between codepoints and grapheme clusters, and steers users away from random-access indexing and having a single (incorrect) notion of a string length.
bayindirh 2y ago
I don't think it's a C++ problem. You just can't transform anything developed in "ancient" times to unicode aware in a single swoop. On the other hand, libicu is 37MB by itself, so it's not something someone can write in a weekend and ship. Any tool which is old enough will have a thousand ways to do something. This is the inevitability of software and programming languages. In the domain of C++, which has a size mammoth now, everyone expects this huge pony to learn new tricks, but everybody has a different idea of the "new tricks", so more features are added on top of its already impressive and very long list of features and capabilities. You want libICU built-in? There must be other folks who want that too. So you may need to find them and work with them to make your dream a reality. So, C++ is doing fine. It's not that they omitted Unicode during the design phase. Unicode has arrived later, and it has to be integrated by other means. This is what libraries for.
- pornel 2y ago
  Being developed in, and having to stay compatible with, ancient times is a real problem of C++. The now-invalid assumptions couldn't have been avoided 50 years ago. Fixing them now in C++ is difficult or impossible, but still, the end result is a ton of brokenness baked into C++. Languages developed in the 21st century typically have some at least half-decent Unicode support built-in. Unicode is big and complex, but there's a lot that a language can do to at least not silently destroy the encoding.
  - cm2187 2y ago
    That explains why there are two functions, one for ascii and one for unicode. That doesn't explain why the unicode functions are hard to use (per the article).
    - BoringTimesGang 2y ago
      Because human language is hard to boil down to a simple computing model and the problem is underdefined, based on naive assumptions. Or perhaps I should say naïve.
deleted 2y ago
[deleted]
Muromec 2y ago
Well, the only time you can do str lower where unicode locale awareness will be a problem is when you do it on the user input, like names. How about you just dont? If it's a constant in your code, you probably use ASCII anyway or can do a static mapping. If it's user user input -- just don't str lower / str upper it.
- account42 2y ago
  Yes, except when it is not your choice. If the requirements are to display some strings in lower/uppercase then you need to find a way to do that. That doesn't have to be using the standard library though.
pjmlp 2y ago
Because it is a fight to put anything on a ISO managed language, and only the strongest persevere long enough to make it happen. Regardless of what ISO language we are talking about.
- gpderetta 2y ago
  Yes, significantly smaller libraries had an hard time getting onto the standard. Getting the equivalent of ICU would be almost impossible. And good luck keeping it up to date.
- account42 2y ago
  If anything it should be harder to add things to the language. Too many new additions have been half-arsed like and needed to be changed or deprecated soon after.
wslh 2y ago
Me too, how is case conversion perfectly done in modern languages such as Zig [1], Rust, or Swift? [1] Ended up looking at https://github.com/JakubSzark/zig-string https://github.com/JakubSzark/zig-string
- steveklabnik 2y ago
  In Rust, the APIs are clear if they're ASCII only or unicode aware. https://doc.rust-lang.org/stable/std/primitive.str.html#method.to_lowercase https://doc.rust-lang.org/stable/std/primitive.str.html#meth... > ‘Lowercase’ is defined according to the terms of the Unicode Derived Core Property Lowercase. https://doc.rust-lang.org/stable/std/primitive.str.html#method.to_ascii_lowercase https://doc.rust-lang.org/stable/std/primitive.str.html#meth... > ASCII letters ‘A’ to ‘Z’ are mapped to ‘a’ to ‘z’, but non-ASCII letters are unchanged. Now, "perfectly" is very strong. For example, the Turkish i problem. That is not solved. But 99% of Unicode stuff is handled correctly by default.
  - oguz-ismail 2y ago
    > 99% of Unicode stuff Does that include context-dependent conversion rules like o'reilly -> O'Reilly?
    - PaulDavisThe1st 2y ago
      that is neither up-casing nor-downcasing, but (de)capitalization, which is a significantly more complex task (which ultimately requires up- or down-casing, but a whole lot more before then).
      - oguz-ismail 2y ago
        So it doesn't. If Unicode doesn't cover non-trivial forms of case-folding, 99% of Unicode doesn't mean anything.
        PaulDavisThe1st 2y ago
dennis_jeeves2 2y ago
> There are so many ways to do something and every way is freaking wrong! Stroustrup, laugheth!
hoseja 2y ago
>Makes you wonder why this isn't part of the C++ standard library itself. Because then every change in Unicode would need to be standardized in C++ as well. Yup. Can't have Unicode due to committee friction.
account42 2y ago
> Makes you wonder why this isn't part of the C++ standard library itself. Because the C++ standard library cares about binary size and backwards compatiblity, both of with are incompatible with a full Unicode implementation. Putting this in the stdlib means everyone has to pay for it even when you don't need it. Libraries are fine, not everything needs to be defined by the language itself.
Night_Thastus 2y ago
As a C++ dev, I have never run into the problem the post is describing. Upper and lowercase conversion has always worked just fine. Though then again, I don't fiddle with mixed unicode and non-unicode situations.
Netch 2y ago
> Makes you wonder why this isn't part of the C++ standard library itself. Plainly no need if there is a separate easily attachable library (and with permissible license). What C++ had to do - provide character (char{8,16,32}_t) and string types - it has done.

BoringTimesGang 2y ago

>It is issues like this due to which I gave up on C++. There are so many ways to do something and every way is freaking wrong! These are mostly unicode or linguistics problems.

tralarpa 2y ago
The fact that the standard library works against you doesn't help (to_lower takes an int, but only kind of works (sometimes) correctly on unsigned char, and wchar_t is implicitly promoted to int).
- BoringTimesGang 2y ago
  to_lower is in the std namespace but is actually just part of the C89 standard, meaning it predates both UTF8 and UTF16. Is the alternative that it should be made unusable, and more existing code broken? A modern user has to include one of the c-prefix headers to use it, already hinting to them that 'here be dragons'. But there are always dragons. It's strings. The mere assumption that they can be transformed int-by-int, irrespective of encoding, is wrong. As is the assumption that a sensible transformation to lower case without error handling exists.
  - account42 2y ago
    > Is the alternative that it should be made unusable, and more existing code broken? It should be marked [[deprecated]], yes. There is no good reason to use std::tolower/toupper anywhere - they can neither do unicode properly nor are they anywhere close to efficient for ASCII. And their behavior depends on the process-global locale.

pistoleer 2y ago

> There are so many ways to do something and every way is freaking wrong! That's life! The perfect way does not exist. The best you can do is be aware of the tradeoffs, and languages like C++ absolutely throw them in your face at every single opportunity. It's fatiguing, and writing in javascript or python allows us to uphold the facade that everything is okay and that we don't have to worry about a thing.

deleted 2y ago
[deleted]
pornel 2y ago
JS and Python are still old enough to have been created when Unicode was in its infancy, so they have their own share of problems from using UCS-2 (such as indexing strings by what is now a UTF-16 code unit, rather than by a codepoint or a grapheme cluster). Swift has been developed in the modern times, and it's able to tackle Unicode properly, e.g. makes distinction between codepoints and grapheme clusters, and steers users away from random-access indexing and having a single (incorrect) notion of a string length.

bayindirh 2y ago

I don't think it's a C++ problem. You just can't transform anything developed in "ancient" times to unicode aware in a single swoop. On the other hand, libicu is 37MB by itself, so it's not something someone can write in a weekend and ship. Any tool which is old enough will have a thousand ways to do something. This is the inevitability of software and programming languages. In the domain of C++, which has a size mammoth now, everyone expects this huge pony to learn new tricks, but everybody has a different idea of the "new tricks", so more features are added on top of its already impressive and very long list of features and capabilities. You want libICU built-in? There must be other folks who want that too. So you may need to find them and work with them to make your dream a reality. So, C++ is doing fine. It's not that they omitted Unicode during the design phase. Unicode has arrived later, and it has to be integrated by other means. This is what libraries for.

pornel 2y ago
Being developed in, and having to stay compatible with, ancient times is a real problem of C++. The now-invalid assumptions couldn't have been avoided 50 years ago. Fixing them now in C++ is difficult or impossible, but still, the end result is a ton of brokenness baked into C++. Languages developed in the 21st century typically have some at least half-decent Unicode support built-in. Unicode is big and complex, but there's a lot that a language can do to at least not silently destroy the encoding.
- cm2187 2y ago
  That explains why there are two functions, one for ascii and one for unicode. That doesn't explain why the unicode functions are hard to use (per the article).
  - BoringTimesGang 2y ago
    Because human language is hard to boil down to a simple computing model and the problem is underdefined, based on naive assumptions. Or perhaps I should say naïve.

deleted 2y ago

[deleted]

Muromec 2y ago

Well, the only time you can do str lower where unicode locale awareness will be a problem is when you do it on the user input, like names. How about you just dont? If it's a constant in your code, you probably use ASCII anyway or can do a static mapping. If it's user user input -- just don't str lower / str upper it.

account42 2y ago
Yes, except when it is not your choice. If the requirements are to display some strings in lower/uppercase then you need to find a way to do that. That doesn't have to be using the standard library though.

pjmlp 2y ago

Because it is a fight to put anything on a ISO managed language, and only the strongest persevere long enough to make it happen. Regardless of what ISO language we are talking about.

gpderetta 2y ago
Yes, significantly smaller libraries had an hard time getting onto the standard. Getting the equivalent of ICU would be almost impossible. And good luck keeping it up to date.
account42 2y ago
If anything it should be harder to add things to the language. Too many new additions have been half-arsed like and needed to be changed or deprecated soon after.

wslh 2y ago

Me too, how is case conversion perfectly done in modern languages such as Zig [1], Rust, or Swift? [1] Ended up looking at https://github.com/JakubSzark/zig-string https://github.com/JakubSzark/zig-string

steveklabnik 2y ago
In Rust, the APIs are clear if they're ASCII only or unicode aware. https://doc.rust-lang.org/stable/std/primitive.str.html#method.to_lowercase https://doc.rust-lang.org/stable/std/primitive.str.html#meth... > ‘Lowercase’ is defined according to the terms of the Unicode Derived Core Property Lowercase. https://doc.rust-lang.org/stable/std/primitive.str.html#method.to_ascii_lowercase https://doc.rust-lang.org/stable/std/primitive.str.html#meth... > ASCII letters ‘A’ to ‘Z’ are mapped to ‘a’ to ‘z’, but non-ASCII letters are unchanged. Now, "perfectly" is very strong. For example, the Turkish i problem. That is not solved. But 99% of Unicode stuff is handled correctly by default.
- oguz-ismail 2y ago
  > 99% of Unicode stuff Does that include context-dependent conversion rules like o'reilly -> O'Reilly?
  - PaulDavisThe1st 2y ago
    that is neither up-casing nor-downcasing, but (de)capitalization, which is a significantly more complex task (which ultimately requires up- or down-casing, but a whole lot more before then).
    - oguz-ismail 2y ago
      So it doesn't. If Unicode doesn't cover non-trivial forms of case-folding, 99% of Unicode doesn't mean anything.
      - PaulDavisThe1st 2y ago

dennis_jeeves2 2y ago

> There are so many ways to do something and every way is freaking wrong! Stroustrup, laugheth!

hoseja 2y ago

>Makes you wonder why this isn't part of the C++ standard library itself. Because then every change in Unicode would need to be standardized in C++ as well. Yup. Can't have Unicode due to committee friction.

account42 2y ago

> Makes you wonder why this isn't part of the C++ standard library itself. Because the C++ standard library cares about binary size and backwards compatiblity, both of with are incompatible with a full Unicode implementation. Putting this in the stdlib means everyone has to pay for it even when you don't need it. Libraries are fine, not everything needs to be defined by the language itself.

Night_Thastus 2y ago

As a C++ dev, I have never run into the problem the post is describing. Upper and lowercase conversion has always worked just fine. Though then again, I don't fiddle with mixed unicode and non-unicode situations.

Netch 2y ago

> Makes you wonder why this isn't part of the C++ standard library itself. Plainly no need if there is a separate easily attachable library (and with permissible license). What C++ had to do - provide character (char{8,16,32}_t) and string types - it has done.