3 ms·

The default locale for pretty much any language on most Linux distros is utf8 at least. Legacy systems exist, as you point out especially in those CJK countries

The default locale for pretty much any language on most Linux distros is utf8 at least. Legacy systems exist, as you point out especially in those CJK countries, I guess the transition to Unicode was much more annoying there. But incidentally those old MBCS charsets are those where isalnum et al never made sense in the first place. I'd argue the usefulness of all that glibc code is very close to zero. Sure, the glibc folks can pat themselves on the back for covering the POSIX spec so well but I really prefer the Linux approach here; follow POSIX where it makes sense but omit the insanity.

bonzini 6y ago
They would pat themselves on the back for writing useful code when it was useful. They didn't write it in 2010, it dates back to the 80s. The comparison to Linux also makes little sense; talking about locales, Linux still supports Microsoft code pages for the FAT file system.
- iforgotpassword 6y ago
  Then it is time to consider removing this code. If you need it use an old glibc. Your comparison to Linux makes no sense either. It's not that the code pages themselves are the problem, you can use them for conversion and whatnot. But pretending a collection of functions that is unable to do anything meaningful in the present day and crashes for extra points is fine, because it was written long ago just isn't. The musl solution is correct. If you need to handle strings in a locale-aware manner, use a dedicated lib. Better yet don't use C.
  - bonzini 6y ago
    With musl, an Italian person wouldn't be able to use "grep -w ciò" and a Greek person wouldn't be able to use "grep -w" on Greek text at all. So okay it's not a big deal, a Thai person cannot use "grep -w" on Thai text either, but the musl solution is correct for specific use cases where locale doesn't matter. And the crash is a problem with C but it's not a problem with the function. There's a different function, wisalnum, that doesn't crash with out of bounds values, but it also doesn't do Unicode on musl.
    - iforgotpassword 6y ago
      grep -w is pretty useless either way. Ok I might sound like I just try to brush off the issue here but it's not intuitive anyways that it needs to know the locale to properly match in the first place. The Italian person might be caught off guard when trying to find a Swedish word in a file, because you need to remember to switch the locale first. And if grep actually uses isalnum for -w then the Italian and Greek and Thai people will be out of luck either way since they all use utf8 today and ò is two bytes. Good look if you work with a multi-language file and have to constantly switch. But that's not even a problem specifically with POSIX or glibc but language in general; there are languages that don't have any separators between words so you already lost. The crash is a problem with the function, specifically with glibc. You might even put into question why POSIX defines out of range values as UB instead of requiring it to return false but that's yet another can of worms we better not open here. I fully blame not doing any range checks and doing an oob array access on glibc. UB doesn't force you to create an implementation that crash and burns... Returning false would be great. Maybe abort if you're anal. Have it return something random if you're concerned about speed. But don't freaking crash! The libc should work with the dev, not against them.

bonzini 6y ago

They would pat themselves on the back for writing useful code when it was useful. They didn't write it in 2010, it dates back to the 80s. The comparison to Linux also makes little sense; talking about locales, Linux still supports Microsoft code pages for the FAT file system.

iforgotpassword 6y ago
Then it is time to consider removing this code. If you need it use an old glibc. Your comparison to Linux makes no sense either. It's not that the code pages themselves are the problem, you can use them for conversion and whatnot. But pretending a collection of functions that is unable to do anything meaningful in the present day and crashes for extra points is fine, because it was written long ago just isn't. The musl solution is correct. If you need to handle strings in a locale-aware manner, use a dedicated lib. Better yet don't use C.
- bonzini 6y ago
  With musl, an Italian person wouldn't be able to use "grep -w ciò" and a Greek person wouldn't be able to use "grep -w" on Greek text at all. So okay it's not a big deal, a Thai person cannot use "grep -w" on Thai text either, but the musl solution is correct for specific use cases where locale doesn't matter. And the crash is a problem with C but it's not a problem with the function. There's a different function, wisalnum, that doesn't crash with out of bounds values, but it also doesn't do Unicode on musl.
  - iforgotpassword 6y ago
    grep -w is pretty useless either way. Ok I might sound like I just try to brush off the issue here but it's not intuitive anyways that it needs to know the locale to properly match in the first place. The Italian person might be caught off guard when trying to find a Swedish word in a file, because you need to remember to switch the locale first. And if grep actually uses isalnum for -w then the Italian and Greek and Thai people will be out of luck either way since they all use utf8 today and ò is two bytes. Good look if you work with a multi-language file and have to constantly switch. But that's not even a problem specifically with POSIX or glibc but language in general; there are languages that don't have any separators between words so you already lost. The crash is a problem with the function, specifically with glibc. You might even put into question why POSIX defines out of range values as UB instead of requiring it to return false but that's yet another can of worms we better not open here. I fully blame not doing any range checks and doing an oob array access on glibc. UB doesn't force you to create an implementation that crash and burns... Returning false would be great. Maybe abort if you're anal. Have it return something random if you're concerned about speed. But don't freaking crash! The libc should work with the dev, not against them.