4 ms·

I'm willing to bet a large amount that non UTF-8 encoding were broken and nobody cared enough to bother fixing them. OpenBSD does not hesitate to nuke legacy s

by throwaway2048 11y ago

I'm willing to bet a large amount that non UTF-8 encoding were broken and nobody cared enough to bother fixing them. OpenBSD does not hesitate to nuke legacy stuff that gets broken. Which i feel is ultimately for the best, because half-assed support that barely functions is worse than no support at all many times.

kragen 11y ago
If I had to guess, using my mental model of OpenBSD: (a) most non-UTF-8-or-UTF-16 locales will choke (crash or corrupt data) in the rare case that they try to encode text outside their encoding range (the mirror image of the problem UTF-8B fixes in UTF-8); (b) codecs have to be fast and handle untrusted strings of somewhat unpredictable lengths, making them a likely source of security holes; (c) possible subtle bugs in a codec enable "cloaking attacks" where different parts of a system parse the same string differently; these have existed in the past with UTF-8, but would have to be rooted out of every codec; (d) encoding text with one codec and decoding it with another also corrupts it. So there are lots of good reasons to require the system to default to UTF-8 and use other codecs only in special cases involving backwards compatibility. I hope you can still get reasonable performance and sensible ordering by setting LC_COLLATE=C.
busterarm 11y ago
Having sat in on a BUG meeting where this was discussed by one of the devs responsible, I believe it was basically "UTF-8 won, it's time to not pretend otherwise, we're going to move forward with this."
stsp 11y ago
It was in fact intentionally broken to find out where removing single-byte locales hurts our users most. We have a hackathon coming up with devs committed to making UTF-8 work in more base utilities. If that works out, and the most sore points of latin1/koi-8/etc users have been adequately addressed, 5.9 will ship with only the UTF-8 locale (and of course the default "C" locale -- ASCII). If this approach turns out to be wrong because we cannot get regressions fixed, 5.9 will ship like 5.7 and 5.8 (with UTF-8 and single byte locales).
- opk 11y ago
  My first thought was, what about the "C" locale so good to see that question already answered. I really wish there was some sort of standard "U" locale that would be the same as "C" but UTF-8, and ISO rather than US format dates.
  - kazinator 11y ago
    That locale pseudo-exists. It's called "don't call the evil setlocale function, write in C90 as much as possible, do your own UTF-8 encoding and decoding, and implement the exact default date format you want with your own strftime string or whatever."
    - Dylan16807 11y ago
      That doesn't exactly help me as a user, and possibly makes things worse as some things respect locale and some don't.
  - caf 11y ago
    There has been some talk both in glibc and musl of shipping such a "C-but-UTF-8" locale.
- kragen 11y ago
  Oh, I didn't realize you weren't removing "C"! Thank you for explaining!

kragen 11y ago

If I had to guess, using my mental model of OpenBSD: (a) most non-UTF-8-or-UTF-16 locales will choke (crash or corrupt data) in the rare case that they try to encode text outside their encoding range (the mirror image of the problem UTF-8B fixes in UTF-8); (b) codecs have to be fast and handle untrusted strings of somewhat unpredictable lengths, making them a likely source of security holes; (c) possible subtle bugs in a codec enable "cloaking attacks" where different parts of a system parse the same string differently; these have existed in the past with UTF-8, but would have to be rooted out of every codec; (d) encoding text with one codec and decoding it with another also corrupts it. So there are lots of good reasons to require the system to default to UTF-8 and use other codecs only in special cases involving backwards compatibility. I hope you can still get reasonable performance and sensible ordering by setting LC_COLLATE=C.

busterarm 11y ago

Having sat in on a BUG meeting where this was discussed by one of the devs responsible, I believe it was basically "UTF-8 won, it's time to not pretend otherwise, we're going to move forward with this."

stsp 11y ago

It was in fact intentionally broken to find out where removing single-byte locales hurts our users most. We have a hackathon coming up with devs committed to making UTF-8 work in more base utilities. If that works out, and the most sore points of latin1/koi-8/etc users have been adequately addressed, 5.9 will ship with only the UTF-8 locale (and of course the default "C" locale -- ASCII). If this approach turns out to be wrong because we cannot get regressions fixed, 5.9 will ship like 5.7 and 5.8 (with UTF-8 and single byte locales).

opk 11y ago
My first thought was, what about the "C" locale so good to see that question already answered. I really wish there was some sort of standard "U" locale that would be the same as "C" but UTF-8, and ISO rather than US format dates.
- kazinator 11y ago
  That locale pseudo-exists. It's called "don't call the evil setlocale function, write in C90 as much as possible, do your own UTF-8 encoding and decoding, and implement the exact default date format you want with your own strftime string or whatever."
  - Dylan16807 11y ago
    That doesn't exactly help me as a user, and possibly makes things worse as some things respect locale and some don't.
- caf 11y ago
  There has been some talk both in glibc and musl of shipping such a "C-but-UTF-8" locale.
kragen 11y ago
Oh, I didn't realize you weren't removing "C"! Thank you for explaining!