3 ms·

I feel your pain. Last week I just gave up and wrote my UTF-8 to UTF-32 conversion routine. It took me far less to do that than I spent looking for a standard s

I feel your pain. Last week I just gave up and wrote my UTF-8 to UTF-32 conversion routine. It took me far less to do that than I spent looking for a standard solution.

usrnm 4y ago
Conversion is the easy part, though. Now try to count the number of "characters" in a string. You will quickly realise that it's much easier to just reach for icu, or at least some parts of it
- mlindner 4y ago
  That's not something you should be doing anyway. It's not a useful thing to talk about and doesn't even work for all languages. If you are writing some software that requires counting the number of characters in a string you should stop and rethink why you're trying to do what you're doing.
- lultimouomo 4y ago
  Sure, but a) conversion was exactly what was provided but C++11, before they deprecated it b) plenty of staff that the standard library does is complicated! Some of it also seems less fundamental than handling plain text.
- int_19h 4y ago
  In practice, counting the number of "characters" turns out to be not all that common, though. This usually comes up doing low-level text rendering or editing, but how often would you do something like that yourself instead of delegating to some library? (and then that library might still reach for ICU - but that's another story)
Findecanor 4y ago
Does it perform validation? Byte sequences too short or too long are obviously invalid, but codes encoded in too many bytes than necessary are too - which isn't immediately obvious.
- lultimouomo 4y ago
  It does very limited validation. It is very purpose specific, its ultimate objective to check whether the first character of a UTF-8 string is RTL or LTR, for layout purporses. To do this we convert N UTF-8 bytes to a single UTF-32 code pointer, after which we check its value against a simple table. If we can't decode the code point from the UTF-8 encoding, we return NULL which falls (for our purposes) in the LTR category. The UTF-8 string is left untouched and unvalidated and passed as-is outside of our library, where it's someone else's problem. (The function is defined as static in the cpp file where RTL detection is done, with a warning that is not general purpose)

I feel your pain. Last week I just gave up and wrote my UTF-8 to UTF-32 conversion routine. It took me far less to do that than I spent looking for a standard solution.

usrnm 4y ago

Conversion is the easy part, though. Now try to count the number of "characters" in a string. You will quickly realise that it's much easier to just reach for icu, or at least some parts of it

mlindner 4y ago
That's not something you should be doing anyway. It's not a useful thing to talk about and doesn't even work for all languages. If you are writing some software that requires counting the number of characters in a string you should stop and rethink why you're trying to do what you're doing.
lultimouomo 4y ago
Sure, but a) conversion was exactly what was provided but C++11, before they deprecated it b) plenty of staff that the standard library does is complicated! Some of it also seems less fundamental than handling plain text.
int_19h 4y ago
In practice, counting the number of "characters" turns out to be not all that common, though. This usually comes up doing low-level text rendering or editing, but how often would you do something like that yourself instead of delegating to some library? (and then that library might still reach for ICU - but that's another story)

Findecanor 4y ago

Does it perform validation? Byte sequences too short or too long are obviously invalid, but codes encoded in too many bytes than necessary are too - which isn't immediately obvious.

lultimouomo 4y ago
It does very limited validation. It is very purpose specific, its ultimate objective to check whether the first character of a UTF-8 string is RTL or LTR, for layout purporses. To do this we convert N UTF-8 bytes to a single UTF-32 code pointer, after which we check its value against a simple table. If we can't decode the code point from the UTF-8 encoding, we return NULL which falls (for our purposes) in the LTR category. The UTF-8 string is left untouched and unvalidated and passed as-is outside of our library, where it's someone else's problem. (The function is defined as static in the cpp file where RTL detection is done, with a warning that is not general purpose)