3 ms·

> Standard C’s primary deficiency is its constant clinging to and dependency upon the “multibyte” encoding and the “wide character” encoding. What? No, UTF-8

> Standard C’s primary deficiency is its constant clinging to and dependency upon the “multibyte” encoding and the “wide character” encoding. What? No, UTF-8 won for a reason, and that reason is not just that C has a deficiency in this area but, rather, that UTF-8 is: a) simple and much saner than UTF-16, b) self-synchronizing in both directions, c) as -or even more- space efficient than UTF-16 on average even for non-Latin text, d) even UTF-32 doesn't make it possible to turn logical character string indices into 32-bit word string indices. (d) is the real killer. One just cannot assume that a character or glyph requires just one codepoint to express, therefore one can't assume that a character or glyph will require some fixed number of code units to express, therefore one might as well use UTF-8 because it's saner and more efficient than the other UTFs, therefore... <drumroll/> a "constant clinging to and dependency upon the “multibyte” encoding" is NOT a deficiency of C but an advantage of C. C does have problems here though, namely all the usual problems it has: - C strings (NUL-terminated) suck - C doesn't have a first class string type, only pointer to char - C doesn't have a first class string type that indicates what encoding the string uses Now, C's wchar_t is not even a deficiency but a disaster.

Taywee 3y ago
I don't think that's what the complaint is. The complaint is that "multibyte" is not necessarily UTF-8. You can't just blindly convert to multibyte assuming that it's UTF-8, because it might not be. You can't convert between two encodings by just going through "multibyte", because it might actually not support all characters you might need to support. So it really is a deficiency in C. It's nearly useless to have a "multibyte" or "wide character" encoding when those can mean anything. Having conversion between UTF-8 and UTF-32 is useful. Having conversion between "implemetation and platform dependent 'multibyte'" and "implementation and platform dependent 'wide character'" strings is nearly useless.
- cryptonector 3y ago
  > You can't just blindly convert to multibyte assuming that it's UTF-8, because it might not be. I mentioned that. TFA didn't say that specifically.
  - Taywee 3y ago
    It sort of did, but in a completely different place past the critique section: > But, rather than using them and needing to praying to the heaven’s the internal Multibyte C Encoding is UTF-8 (like with the aforementioned wcrtomb -> mbrtoc8/16/32 style of conversions), we’ll just provide a direction conversion routine and cut out the wchar_t encoding/multibyte encoding middle man. Not sure why it wasn't mentioned up top. When trying to convert between UTF-8 and UTF-16 without doing it myself or pulling in external dependencies, this was the most annoying thing that slapped me in the face. This is the problem that makes reliable charset conversions between specific encodings actually impossible using just the stdlib functions.
    - cryptonector 3y ago
      Standards-wise the only answer to this is to deprecate all non-UTF-8 locales and leave non-UTF-8 codesets outside the scope of C. Basically, non-Unicode needs to always be at the edge, while in the middle everything needs to be Unicode. From an application perspective it's easy: document that it only works in UTF-8 locales. Really, that is my position for my software. Anything else is ETOOHARD.
      - Taywee 3y ago
        I just want reliable conversions. In my situation (duct taping a very old service to a newer one), I needed to read structured files with UTF-16 fields, and process them into an eventual UTF-8 file written to a different location. The host this needed to run on did not have any unicode locales installed (and incidentally, I hate changing locales for my software because it's a program-global switch to flip, and most of my program still wants to run in the user's locale). I found it ridiculous that there was no way to just convert UTF-16 to UTF-8 without either reinventing that wheel, pulling in an external dependency, or changing global state and having the right system locales installed (as well as knowing the name of at least one of those locales, and guessing a language along with it), despite having the latest C and C++ compilers at my disposal.
- kps 3y ago
  C multibyte, I believe, was designed around ISO2022-style stateful code switching. It predates Unicode.
nunuvit 3y ago
If I understand you correctly, you think that statement is knocking utf-8. And if I understand the article correctly, it's not. Something here is incongruous, and it may very well be me. I believe the author is knocking how the C language poorly supports utf-8 due to legacy issues with supporting wide characters, which aligns with your criticism.

Taywee 3y ago

I don't think that's what the complaint is. The complaint is that "multibyte" is not necessarily UTF-8. You can't just blindly convert to multibyte assuming that it's UTF-8, because it might not be. You can't convert between two encodings by just going through "multibyte", because it might actually not support all characters you might need to support. So it really is a deficiency in C. It's nearly useless to have a "multibyte" or "wide character" encoding when those can mean anything. Having conversion between UTF-8 and UTF-32 is useful. Having conversion between "implemetation and platform dependent 'multibyte'" and "implementation and platform dependent 'wide character'" strings is nearly useless.

cryptonector 3y ago
> You can't just blindly convert to multibyte assuming that it's UTF-8, because it might not be. I mentioned that. TFA didn't say that specifically.
- Taywee 3y ago
  It sort of did, but in a completely different place past the critique section: > But, rather than using them and needing to praying to the heaven’s the internal Multibyte C Encoding is UTF-8 (like with the aforementioned wcrtomb -> mbrtoc8/16/32 style of conversions), we’ll just provide a direction conversion routine and cut out the wchar_t encoding/multibyte encoding middle man. Not sure why it wasn't mentioned up top. When trying to convert between UTF-8 and UTF-16 without doing it myself or pulling in external dependencies, this was the most annoying thing that slapped me in the face. This is the problem that makes reliable charset conversions between specific encodings actually impossible using just the stdlib functions.
  - cryptonector 3y ago
    Standards-wise the only answer to this is to deprecate all non-UTF-8 locales and leave non-UTF-8 codesets outside the scope of C. Basically, non-Unicode needs to always be at the edge, while in the middle everything needs to be Unicode. From an application perspective it's easy: document that it only works in UTF-8 locales. Really, that is my position for my software. Anything else is ETOOHARD.
    - Taywee 3y ago
      I just want reliable conversions. In my situation (duct taping a very old service to a newer one), I needed to read structured files with UTF-16 fields, and process them into an eventual UTF-8 file written to a different location. The host this needed to run on did not have any unicode locales installed (and incidentally, I hate changing locales for my software because it's a program-global switch to flip, and most of my program still wants to run in the user's locale). I found it ridiculous that there was no way to just convert UTF-16 to UTF-8 without either reinventing that wheel, pulling in an external dependency, or changing global state and having the right system locales installed (as well as knowing the name of at least one of those locales, and guessing a language along with it), despite having the latest C and C++ compilers at my disposal.
kps 3y ago
C multibyte, I believe, was designed around ISO2022-style stateful code switching. It predates Unicode.

nunuvit 3y ago

If I understand you correctly, you think that statement is knocking utf-8. And if I understand the article correctly, it's not. Something here is incongruous, and it may very well be me. I believe the author is knocking how the C language poorly supports utf-8 due to legacy issues with supporting wide characters, which aligns with your criticism.