6 ms·
The C Standard Library Function Isspace() Depends on Locale
- kazinator 3y agoScanning a floating-point value with strtod depends on locale. If it's in some locale where the decimal point is a comma, it may stop recognizing the standard 123.456+EE notation. The fix is never to call setlocale(); calling setlocale is like asking "f___ my C program". ISO C localization was designed back in the 1980s, when nobody had any real experience with localizing. In a greenfield C program, it's best to do it all yourself from scratch and stay away from C localization, so you can depend on strtod and isspace to do what they are supposed to.
- bitwize 3y agos/greenfield C program/greenfield program/ s/from C localization.*$/from C./
- pavlov 3y agoIt's useful to remember that the 1980s approach to localization preceded the global Internet. The way most software worked was that you bought it from a local reseller. It came localized for your country (perhaps by the reseller or importer rather than the original authors of the software), and then you'd use it to conduct your local business. Data interchange wasn't that common. Desktop printers were hugely important because a hard copy was how you'd share anything. If you needed to get the information somewhere fast, you'd then fax it. Rarely when you did need to exchange files, you'd use floppies. Maybe you'd take your WordPerfect document to a professional print shop so they would do a layout using cutting-edge desktop publishing technology. So the notion that somebody in Germany might receive American files, or vice versa, wasn't really a primary concern. It was considered far more important that the Germans, and everybody else, would be able to work with their data with the number formatting that was preferred (and sometimes legally mandated).
- orf 3y agoCool but that was over 40 years ago. Who cares, and why hasn’t it been improved since then?
- IshKebab 3y agoI guess anyone that develops in C is happy living in the 80s amongst the footguns. Anyone who isn't has moved on to other languages where it has been improved.
- pvh 3y agoTo the former, curious people with an interest in how and why the world came to be as it is. To the latter, obviously "it" has improved, but ecosystem effects make certain changes very difficult and expensive to coordinate and what we see here is the scars from that process. Everything you see in the world grew out of things that came before, and was made by fallible people working with limited time, energy, and perspective. Honestly, I'm a bit surprised someone with a three letter handle wouldn't already recognize this. Surely you have been around here for a while.
- JdeBP 3y agoIt actually has been improved. See discssion of isspace_l() elsewhere on this page.
- nwellnhof 3y agoUnfortunately, not all locale-dependent functions have an *_l version, at least on some platforms. glibc doesn't have sprintf_l, for example, unlike BSD and MSVCRT.
- JdeBP 3y agoLuckily, sprintf() is irrelevant to the headlined case, that involves isspace_l(), which is in the BSD, GNU, and even musl C libraries.
- ridiculous_fish 3y agoHow would one implement locale-aware strtod() all by yourself from scratch?
- kazinator 3y agoUse the locale-unaware strtod. If the locale's decimal separator character isn't . (period) then filter the output of that to replace the period with that character.
- david2ndaccount 3y agoYep. Because of the terribleness of locales, you should always just roll your own to handle ascii, or use an explicitly unicode-aware function. Anything in libc that relies on locale is unusable because of this. Besides, isspace is such a trivial function that you don’t want to actually call an extern function (possibly even dynamically linked and thus having to hit the PLT) for it, you want something that is easily inlined.
- nemothekid 3y agompv's locale rant is another "blog post" about the frustration of locale. https://github.com/mpv-player/mpv/commit/1e70e82baa9193f6f027338b0fab0f5078971fbe https://github.com/mpv-player/mpv/commit/1e70e82baa9193f6f02...
- Dwedit 3y agoThis is the famous "shitfucked retarded legacy braindeath" post.
- dfox 3y agoThis touches on a issue that is a part of how we got to this situation. What is the encoding of zero and possibly slash delimited blobs of bytes (filenames, argv, envp…) that you get from and pass to the kernel? There are parts of userspace that don't care, parts that assume that LC_CTYPE is the correct answer (GNU tools are usually in one of these categories), userspace that assumes that all that is UTF-8 regardless of LC_CTYPE and more or less by sheer luck not fail completely when that is not true (Gnome) and then there is Python3 which tries to and fails at reconciling this mess with NT world where filenames have weird, but defined (and funilly enough, dependent on the locale the mkfs was ran with), UTF-16-like encoding.
- matheusmoreira 3y ago> What is the encoding of zero and possibly slash delimited blobs of bytes (filenames, argv, envp…) that you get from and pass to the kernel? Bytes. Every number other than 0 and 47 is allowed. They should probably be treated as opaque blobs of binary data.
- ben0x539 3y agoWell, sometimes the users ask to see them and they don't like numbers very much.
- account42 3y agoAssuming UTF-8 for display works pretty well, as long as you still treat it as bytes for the actual file operations and don't corrupt the data because you expect valid UTF-8. Windows has the same problem where in most places the kernel will allow all WCHAR sequences (minus a short list of reserved ones) and gives you no guarantee that that's actually UTF-16.
- zokier 3y agoC locales are one of those optimistic features that we have inherited that in retrospect ended up being misguided and more trouble than worth; any program that really needs to deal with localization probably will end up pulling something like ICU to deal with all sorts of cases, and on the other hand locales cause all sorts of weird issues for programs not really expecting to be localized. As a bonus locale support incurs heavy performance hit. In this case its extra-awkward to have an attempt of having unicode support on a function that takes a single char as an input; it can't actually handle arbitrary unicode codepoints anyways. I feel a common theme with these sort of things is the thinking that difficult problems can be made tractable by presenting a "simple" naive interface while fudging things behind the scenes. Those supposedly simple interfaces actually become complex to think about once you start asking difficult questions about correctness, error handling, and edge cases.
- Someone 3y ago> In this case its extra-awkward to have an attempt of having unicode support on a function that takes a single char as an input Nitpick: it doesn’t take a char; it takes an int that must either be representable as unsigned char or be equal to EOF (https://en.cppreference.com/w/c/string/byte/isspace https://en.cppreference.com/w/c/string/byte/isspace) Given that description, I don’t think anybody attempted to have unicode support for isspace. IMO, the bug is to call isspace for bytes extracted from utf-8 data.
- tialaramex 3y agoThere are a few of these "optimistic features", and presumably there will be more in today's languages as the state of the art moves on. Another existing example in C: The idea of "wide" characters and strings, which was intended to be maybe the right shape for UCS-2, but it turns out UCS-2 wasn't the future, UCS-4 (approximately UTF-32) is way too bloated to be reasonable, and so we're all going to speak UTF-8 which ironically C was already at least somewhat suitable for. An example from Java: The Java concurrent memory model. As shipped Java 1.0 assumed the way forward is you put locks on data structures, that way concurrent access "just works" correctly. It's amazing. It's also ludicrously expensive and you're paying for it even if you don't ever do concurrent access. Today our locks are much cheaper, but they aren't so cheap that you can afford to just lock all your data structures like it's no big deal. On a modern system taking and giving back the lock costs one atomic operation and maybe a fence - cheap but far from free. So modern Java data structures are mostly just unsafe to use concurrently because that's the trade everybody else took in other languages. A cursed C++ example: In the 1980s, unless you were a hash table expert, the hash table you probably knew how to write was a separate chaining hash table, even by 1989 when ANSI C was standardised, experts might grudgingly admit this was a reasonable choice - there was promising alternatives, but hey separate chaining works and it has well understood behaviour. Somehow though, C++ ended up standardising this bad old data structure as std::unordered_map not in 1998 (when it'd have seemed rather outmoded) but in 2011 (by which point it was beginning to smell bad). On the other hand it's also easy to make the mistake of assuming something is a variable when it's actually locked down by the time you're standardising. 'A' is 65. It was 65 in ASCII in the 1960s and it's still 65 in UTF-8 (as a result of grandfathering the entire ASCII 7-bit code) today. If you carefully wrote code so that 'A' needn't be 65 that was a waste of your time. Clearly in 1965 it was not safe to just assume 'A' is 65, it might change, and in 2023 it's crazy to make that adjustable, it won't change, but somewhere it shaded over and I'm sure some people got caught.
- wahern 3y agoThe fact that Unicode codepoints were being passed to isspace instead of iswspace indicates the relevant code was already fubar'd. > For example, isspace(0x01fe) is true. I can't figure out why this might be considered a whitespace character Because the only valid values (independent of locale) that can be passed to isspace are 0 to UCHAR_MAX and -1/EOF, where UCHAR_MAX refers to unsigned char (usually 255), not Unicode character. Most implementations I've seen (glibc, musl, OpenBSD) index the passed value into a locale-specific array of length UCHAR_MAX + 1, possibly masking the index and/or return values. But TIL macOS (and possibly FreeBSD and NetBSD at some point, if not currently) had vestigial support for passing higher values as part of a presumably long-abandoned approach to I18N. EDIT: FWIW, based on the glibc code (ctype/isctype.c), int __isctype (int ch, int mask) { return (((uint16_t *) _NL_CURRENT (LC_CTYPE, _NL_CTYPE_CLASS) + 128) [(int) (ch)] & mask); } where isspace(c) seems to be translated to __isctype(c, _ISspace) there's a good chance the array is being overflowed. Without looking further (glibc isn't the easiest code to grok), I'd guess the array size is probably 128 + UCHAR_MAX with the offset of 128 (instead of 1) to handle the common case, especially on systems where char is signed, of people passing in negative values, though that only works for a locale like ASCII where -1/EOF and 255/(unsigned char)-1 aren't ambiguous.
- dvh 3y agoSeems like instead of using boolean result better API should return tri-state: space, no space, bad request
- djoldman 3y agoFor implications in python, see for example: https://docs.python.org/3/library/re.html#re.LOCALE https://docs.python.org/3/library/re.html#re.LOCALE
- rwmj 3y agoSee also a classic glibc bug: "[0-9] matches ¼ ١ ２ 〣 and others, but not ９ (and other nines)" (https://news.ycombinator.com/item?id=17557243 https://news.ycombinator.com/item?id=17557243) Sadly they renamed the upstream bug report to something more sober (although didn't fix it).
- eMSF 3y agoWhile writing a fancy word counter I learnt that glibc iswspace (or the glibc locale data) actually does not consider non-breaking spaces as, well, spaces even when using a Unicode locale. This apparently conforms to ISO 30112. (For example MSVCRT does do so.) I happened to notice this via a result mismatch as GNU wc does count NBSPs as word separators. Even though it uses iswspace, it also additionally checks for a hard coded set of Unicode non-breaking spaces. (I have to say I'm a bit surprised at being at getting voted hidden here. I thought this was mostly related to the topic at hand. I would of course gladly be corrected if mistaken about the details.)
- nirvanis 3y agoSomewhat related tip: prepend LANG=C to many console commands such as grep to speed up many tools processing large files, as they will assume ASCII input (which is probably what you have in most cases)
- emmelaich 3y agoand set it for consistency of ordering (collation) between sort, join, tsort, look, etc.
- seanhunter 3y agoIf you care about speed you would probably be using ripgrep rather than grep anyway, but doesn’t `LANG=en_US.UTF-8` give a similar speed on modern systems without any compromise on consistency of sort ordering etc and support for extended characters?
- burntsushi 3y agoFor GNU grep in particular, no, using a UTF-8 locale can significantly slow it down: $ time LC_ALL=C grep -E '^\w{30}$' OpenSubtitles2018.raw.sample.en -c 3 real 0.808 user 0.744 sys 0.063 maxmem 10 MB faults 0 $ time LC_ALL=en_US.UTF-8 grep -E '^\w{30}$' OpenSubtitles2018.raw.sample.en -c 4 real 20.064 user 19.982 sys 0.077 maxmem 10 MB faults 0 Where as ripgrep is just Unicode aware by default, and still about as fast as the ASCII only variant of GNU grep above: $ time rg '^\w{30}$' OpenSubtitles2018.raw.sample.en -c 4 real 1.163 user 1.132 sys 0.030 maxmem 916 MB faults 0
- kps 3y agoFor grep, how much of the difference is due to '\w' having a different meaning between the two cases?
- burntsushi 3y agoThat's exactly the point. ripgrep uses the Unicode definition by default and so corresponds to what GNU grep is doing in the en_US.UTF-8 locale.
- deleted 3y ago[deleted]
- emmelaich 3y agoI think I've mentioned it before, but the isspace() and similar man pages used to warn that they made sense only if ascii. So the recommendation was to always do (isascii() && iswhatever()). With the advent of locales they seem to have just omitted this rather than put in a warning or hint.
- JdeBP 3y agoThe fix if one wants to stick to the standard library is to make use of locale_t posixctypelocale = newlocale(LC_CTYPE_MASK, "POSIX", NULL); saved somewhere early on and then b = isspace_l(c, posixctypelocale); and b = iswspace_l(wc, posixctypelocale); whenever one needs them. The irony is that systems based upon the BSD C library like MacOS and FreeBSD will have this. * https://pubs.opengroup.org/onlinepubs/9699919799/functions/newlocale.html https://pubs.opengroup.org/onlinepubs/9699919799/functions/n... * https://pubs.opengroup.org/onlinepubs/9699919799/functions/isspace.html https://pubs.opengroup.org/onlinepubs/9699919799/functions/i... * https://pubs.opengroup.org/onlinepubs/9699919799/functions/iswspace.html https://pubs.opengroup.org/onlinepubs/9699919799/functions/i... * https://man.freebsd.org/cgi/man.cgi?query=xlocale&sektion=3 https://man.freebsd.org/cgi/man.cgi?query=xlocale&sektion=3
- account42 3y agoThat's quite an absurdly inefficient way to check if a byte is one of 6 fixed values.
- JdeBP 3y agoIt's a simple array index and bitmask in most implementations, which is usually more efficient than what 6 short-circuited comparisons compiles into.
- account42 3y agoPlus a function call (likely with GOT indirection) plus another pointer dereference to get from the locale handle to the array. Also you now need some initialization code to create the locale and store the handle somewhere (unless you want to make this even more ridiculously inefficient and create it each time). Also, five of those whitespace characters have contiguous byte values so you don't even need 6 comparisons. The compiler doesn't have to keep the short circuting here.
- bawolff 3y ago> In this case, isspace() returns true for Unicode white-space values, which includes 0x85 = NEL = Next Line, and 0xA0 = NBSP = No-Break Space. Those aren't even unicode (utf-8) bytes for those characters. They are the iso-8859-1 bytes. (E.g. nbsp is U+00A0 which has a byte representation of 0xC2 0xA0)
- kps 3y agoThe argument to `isspace()` (in that implementation) is a Unicode code point, not an encoding byte.
- Sprocklem 3y agoRight, but the only reasonable way to localize `isspace()` is to have it based on code units / encoding bytes, since that is what it will be used for. When people want to test a unicode code point they instead call `iswspace()` (or a non-standard but more-sane version thereof).
- kps 3y agoMy unverified assumption is that MacOS's behaviour here descends from NeXTStep, which used UCS-2.
- smcameron 3y agoAnd if you just don't want to be bothered by locales, it's not enough to setlocale(LC_ALL, "C"); Other libraries may call setlocale() behind your back (e.g. GTK2). So, what you have to do is, you have to break setlocale's arms off: #define _GNU_SOURCE #include <stdio.h> #include <dlfcn.h> #include <locale.h> typedef char *(*setlocale_prototype)(int category, const char *locale); static setlocale_prototype real_setlocale = NULL; static char *the_only_locale = "C"; char *setlocale(__attribute__((unused)) int category, __attribute__((unused)) const char *locale) { char *msg; if (!real_setlocale) { *(void **) &real_setlocale = dlsym(RTLD_NEXT, "setlocale"); msg = dlerror(); if (msg) { fprintf(stderr, "Failed to override setlocale(): %s\n", msg); fflush(stderr); real_setlocale = NULL; return the_only_locale; } } /* C is the locale, and the locale shall be C */ return real_setlocale(LC_ALL, "C"); } Now it doesn't matter if GTK or whatever library calls setlocale() with whatever weird parameters, the locale shall be "C".
- 0xr0kk3r 3y agoAnother library can call setlocale and it will impact my code? I've never, ever written locale-specific code, so this is kinda surprise.
- teddyh 3y agoLocale is process-wide, so yes. But they shouldn’t; libraries should not call setlocale(). If it does, you should probably report it as a bug. It’s like if a library called chdir(); there are some things which libraries should not do, and setlocale() is one of them.
- JohnFen 3y agoI would be a bit more nuanced than that. I think it's OK if libraries change process state as long as they restore it as it was before the library call returns. And as long as this behavior is clearly documented.
- otikik 3y agoAt least its behavior is defined.
- account42 3y agoSo does whether words should be automatically capitalized in titles and in I say in the C locale they should not be.