4 ms·

C locales are one of those optimistic features that we have inherited that in retrospect ended up being misguided and more trouble than worth; any program that

by zokier 3y ago

C locales are one of those optimistic features that we have inherited that in retrospect ended up being misguided and more trouble than worth; any program that really needs to deal with localization probably will end up pulling something like ICU to deal with all sorts of cases, and on the other hand locales cause all sorts of weird issues for programs not really expecting to be localized. As a bonus locale support incurs heavy performance hit. In this case its extra-awkward to have an attempt of having unicode support on a function that takes a single char as an input; it can't actually handle arbitrary unicode codepoints anyways. I feel a common theme with these sort of things is the thinking that difficult problems can be made tractable by presenting a "simple" naive interface while fudging things behind the scenes. Those supposedly simple interfaces actually become complex to think about once you start asking difficult questions about correctness, error handling, and edge cases.

Someone 3y ago
> In this case its extra-awkward to have an attempt of having unicode support on a function that takes a single char as an input Nitpick: it doesn’t take a char; it takes an int that must either be representable as unsigned char or be equal to EOF (https://en.cppreference.com/w/c/string/byte/isspace https://en.cppreference.com/w/c/string/byte/isspace) Given that description, I don’t think anybody attempted to have unicode support for isspace. IMO, the bug is to call isspace for bytes extracted from utf-8 data.
tialaramex 3y ago
There are a few of these "optimistic features", and presumably there will be more in today's languages as the state of the art moves on. Another existing example in C: The idea of "wide" characters and strings, which was intended to be maybe the right shape for UCS-2, but it turns out UCS-2 wasn't the future, UCS-4 (approximately UTF-32) is way too bloated to be reasonable, and so we're all going to speak UTF-8 which ironically C was already at least somewhat suitable for. An example from Java: The Java concurrent memory model. As shipped Java 1.0 assumed the way forward is you put locks on data structures, that way concurrent access "just works" correctly. It's amazing. It's also ludicrously expensive and you're paying for it even if you don't ever do concurrent access. Today our locks are much cheaper, but they aren't so cheap that you can afford to just lock all your data structures like it's no big deal. On a modern system taking and giving back the lock costs one atomic operation and maybe a fence - cheap but far from free. So modern Java data structures are mostly just unsafe to use concurrently because that's the trade everybody else took in other languages. A cursed C++ example: In the 1980s, unless you were a hash table expert, the hash table you probably knew how to write was a separate chaining hash table, even by 1989 when ANSI C was standardised, experts might grudgingly admit this was a reasonable choice - there was promising alternatives, but hey separate chaining works and it has well understood behaviour. Somehow though, C++ ended up standardising this bad old data structure as std::unordered_map not in 1998 (when it'd have seemed rather outmoded) but in 2011 (by which point it was beginning to smell bad). On the other hand it's also easy to make the mistake of assuming something is a variable when it's actually locked down by the time you're standardising. 'A' is 65. It was 65 in ASCII in the 1960s and it's still 65 in UTF-8 (as a result of grandfathering the entire ASCII 7-bit code) today. If you carefully wrote code so that 'A' needn't be 65 that was a waste of your time. Clearly in 1965 it was not safe to just assume 'A' is 65, it might change, and in 2023 it's crazy to make that adjustable, it won't change, but somewhere it shaded over and I'm sure some people got caught.
- kps 3y ago
  C still does not require letters to be in order or contiguous. (In EBCDIC they're not contiguous.)
  - account42 3y ago
    I think today it's fair to say that if people want to have EBCDIC as the execution character set then its their problem.
- Dylan16807 3y ago
  > but it turns out UCS-2 wasn't the future, UCS-4 (approximately UTF-32) is way too bloated to be reasonable, and so we're all going to speak UTF-8 which ironically C was already at least somewhat suitable for. For UCS-4 I think it's less about the bloat and more about realizing that it's a hard problem. Not only was going to 16 bits not a golden bullet, no size is a golden bullet because of various forms of code point combining. 2 bytes per character when we started preferring UCS-2 was a lot more expensive than 3-4 bytes per character when we stopped preferring it.
- kevin_thibedeau 3y ago
  wchar_t has the big problem that's width is platform dependent. On some it is 32-bits and can handle UCS-4. You just can't write portable code that uses that type.
  - kps 3y ago
    If a system supports Unicode, then wchar_t must be at least 21 bits, according to the C and C++ standards. Of course the 900lb gorilla has a 16-bit wchar_t, but the standards committees prefer to stick their fingers in their ears rather than deprecate it.
kps 3y ago
Did locales have any pre-existing implementation, or did X3J11 invent them?
- wahern 3y ago
  The OpenBSD man pages say that isspace, et al come from Version 7 Unix, which was released in 1979. The C89 rationale seems to suggest that they copied this pre-existing API. Rationle section 4.3 says[1], > Pains were taken to eliminate any ASCII dependencies from the definition of the character handling functions. One notable result of this policy was the elimination of the function isascii, both because of the name and because its function was hard to generalize. Rationale section 2.2.1 says[2], > The Committee ultimately came to remarkable unanimity on the subject of character set requirements. There was strong sentiment that C should not be tied to ASCII, despite its heritage and despite the precedent of Ada being defined in terms of ASCII. ... but it also makes clear that their goal wasn't a perfectly generalized locale framework, but something sufficient for writing a C compiler, i.e. to distinguish character classes relevant to C source code, > ... Rather, an implementation is required to provide a unique character code for each of the printable graphics used by C, and for each of the control codes representable by an escape sequence. (No particular graphic representation for any character is prescribed --- thus the common Japanese practice of using the glyph ¥ for the C character \ is perfectly legitimate.) Translation and execution environments may have different character sets, but each must meet this requirement in its own way. The goal is to ensure that a conforming implementation can translate a C translator written in C. [1] http://port70.net/~nsz/c/c89/rationale/d3.html#ASCII-4-3 http://port70.net/~nsz/c/c89/rationale/d3.html#ASCII-4-3 [2] http://port70.net/~nsz/c/c89/rationale/b.html#Ada-2-2-1 http://port70.net/~nsz/c/c89/rationale/b.html#Ada-2-2-1
  - kps 3y ago
    I know isspace() and the rest of ctype.h date to V7; I was wondering about locales specifically. I'm not aware of any pre-C89 implementation of anything similar to that, so I'm wondering whether X3J11 adopted it from some system I've never heard of, or created it themselves. C89 was supposed to ‘codify common existing practice’, but internationalization politics may have overridden that.
raxxorraxor 3y ago
I think some of the "bussiness languages" of MS often feature such translations. You suddenly have different quotes if you use another locale. I had to bugfix something here and I almost went crazy. Don't remember the language, it was just a rather simple formula in some arcane tool. Didn't want to believe it...
- account42 3y ago
  Haha it's not just their own business languages but also MS' support for standard formats: In locales where the comma is used as the decimal separator, Excel will by default use semicolons for "CSV" exports.
JohnFen 3y ago
Localization and support for multiple character sets turns out to be an incredibly hard problem that remains unsolved. Unicode is horrible mess to deal with and I hate it with a passion, but as terrible as it is, nobody has come up with a better solution. I think it's because there literally is no good solution to the problem.

by zokier 3y ago

Someone 3y ago

> In this case its extra-awkward to have an attempt of having unicode support on a function that takes a single char as an input Nitpick: it doesn’t take a char; it takes an int that must either be representable as unsigned char or be equal to EOF (https://en.cppreference.com/w/c/string/byte/isspace https://en.cppreference.com/w/c/string/byte/isspace) Given that description, I don’t think anybody attempted to have unicode support for isspace. IMO, the bug is to call isspace for bytes extracted from utf-8 data.

tialaramex 3y ago

There are a few of these "optimistic features", and presumably there will be more in today's languages as the state of the art moves on. Another existing example in C: The idea of "wide" characters and strings, which was intended to be maybe the right shape for UCS-2, but it turns out UCS-2 wasn't the future, UCS-4 (approximately UTF-32) is way too bloated to be reasonable, and so we're all going to speak UTF-8 which ironically C was already at least somewhat suitable for. An example from Java: The Java concurrent memory model. As shipped Java 1.0 assumed the way forward is you put locks on data structures, that way concurrent access "just works" correctly. It's amazing. It's also ludicrously expensive and you're paying for it even if you don't ever do concurrent access. Today our locks are much cheaper, but they aren't so cheap that you can afford to just lock all your data structures like it's no big deal. On a modern system taking and giving back the lock costs one atomic operation and maybe a fence - cheap but far from free. So modern Java data structures are mostly just unsafe to use concurrently because that's the trade everybody else took in other languages. A cursed C++ example: In the 1980s, unless you were a hash table expert, the hash table you probably knew how to write was a separate chaining hash table, even by 1989 when ANSI C was standardised, experts might grudgingly admit this was a reasonable choice - there was promising alternatives, but hey separate chaining works and it has well understood behaviour. Somehow though, C++ ended up standardising this bad old data structure as std::unordered_map not in 1998 (when it'd have seemed rather outmoded) but in 2011 (by which point it was beginning to smell bad). On the other hand it's also easy to make the mistake of assuming something is a variable when it's actually locked down by the time you're standardising. 'A' is 65. It was 65 in ASCII in the 1960s and it's still 65 in UTF-8 (as a result of grandfathering the entire ASCII 7-bit code) today. If you carefully wrote code so that 'A' needn't be 65 that was a waste of your time. Clearly in 1965 it was not safe to just assume 'A' is 65, it might change, and in 2023 it's crazy to make that adjustable, it won't change, but somewhere it shaded over and I'm sure some people got caught.

kps 3y ago
C still does not require letters to be in order or contiguous. (In EBCDIC they're not contiguous.)
- account42 3y ago
  I think today it's fair to say that if people want to have EBCDIC as the execution character set then its their problem.
Dylan16807 3y ago
> but it turns out UCS-2 wasn't the future, UCS-4 (approximately UTF-32) is way too bloated to be reasonable, and so we're all going to speak UTF-8 which ironically C was already at least somewhat suitable for. For UCS-4 I think it's less about the bloat and more about realizing that it's a hard problem. Not only was going to 16 bits not a golden bullet, no size is a golden bullet because of various forms of code point combining. 2 bytes per character when we started preferring UCS-2 was a lot more expensive than 3-4 bytes per character when we stopped preferring it.
kevin_thibedeau 3y ago
wchar_t has the big problem that's width is platform dependent. On some it is 32-bits and can handle UCS-4. You just can't write portable code that uses that type.
- kps 3y ago
  If a system supports Unicode, then wchar_t must be at least 21 bits, according to the C and C++ standards. Of course the 900lb gorilla has a 16-bit wchar_t, but the standards committees prefer to stick their fingers in their ears rather than deprecate it.

kps 3y ago

Did locales have any pre-existing implementation, or did X3J11 invent them?

wahern 3y ago
The OpenBSD man pages say that isspace, et al come from Version 7 Unix, which was released in 1979. The C89 rationale seems to suggest that they copied this pre-existing API. Rationle section 4.3 says[1], > Pains were taken to eliminate any ASCII dependencies from the definition of the character handling functions. One notable result of this policy was the elimination of the function isascii, both because of the name and because its function was hard to generalize. Rationale section 2.2.1 says[2], > The Committee ultimately came to remarkable unanimity on the subject of character set requirements. There was strong sentiment that C should not be tied to ASCII, despite its heritage and despite the precedent of Ada being defined in terms of ASCII. ... but it also makes clear that their goal wasn't a perfectly generalized locale framework, but something sufficient for writing a C compiler, i.e. to distinguish character classes relevant to C source code, > ... Rather, an implementation is required to provide a unique character code for each of the printable graphics used by C, and for each of the control codes representable by an escape sequence. (No particular graphic representation for any character is prescribed --- thus the common Japanese practice of using the glyph ¥ for the C character \ is perfectly legitimate.) Translation and execution environments may have different character sets, but each must meet this requirement in its own way. The goal is to ensure that a conforming implementation can translate a C translator written in C. [1] http://port70.net/~nsz/c/c89/rationale/d3.html#ASCII-4-3 http://port70.net/~nsz/c/c89/rationale/d3.html#ASCII-4-3 [2] http://port70.net/~nsz/c/c89/rationale/b.html#Ada-2-2-1 http://port70.net/~nsz/c/c89/rationale/b.html#Ada-2-2-1
- kps 3y ago
  I know isspace() and the rest of ctype.h date to V7; I was wondering about locales specifically. I'm not aware of any pre-C89 implementation of anything similar to that, so I'm wondering whether X3J11 adopted it from some system I've never heard of, or created it themselves. C89 was supposed to ‘codify common existing practice’, but internationalization politics may have overridden that.

raxxorraxor 3y ago

I think some of the "bussiness languages" of MS often feature such translations. You suddenly have different quotes if you use another locale. I had to bugfix something here and I almost went crazy. Don't remember the language, it was just a rather simple formula in some arcane tool. Didn't want to believe it...

account42 3y ago
Haha it's not just their own business languages but also MS' support for standard formats: In locales where the comma is used as the decimal separator, Excel will by default use semicolons for "CSV" exports.

JohnFen 3y ago

Localization and support for multiple character sets turns out to be an incredibly hard problem that remains unsolved. Unicode is horrible mess to deal with and I hate it with a passion, but as terrible as it is, nobody has come up with a better solution. I think it's because there literally is no good solution to the problem.