8 ms·

Fun with Glibc and the Ctype.h Functions

70 points by picture 5y ago

josephcsible 5y ago
Here's what the C standard says about character handling functions: > In all cases the argument is an int, the value of which shall be representable as an unsigned char or shall equal the value of the macro EOF. If the argument has any other value, the behavior is undefined. So this is just a case of glibc being optimized in a way that's really unforgiving if you commit that particular UB.
- cryptonector 5y ago
  No, this is a case of glibc trying to support localization of ctype in spite of the fact that it can't be localized to anything other than English in UTF-8 locales, anything other than Latin scripts in ISO-8859-* locales, or English in C/POSIX or EBCDIC locales. And then on top of that trying to be fast. I'd give up on supporting localization for ctype. This makes me think, too, "never use ctype, just hardcode my own that assumes ASCII".
  - kazinator 5y ago
    I don't know about never, but try hard not to use anything whose behavior is influenced by a call to setlocale.
    - matheusmoreira 5y ago
      Is that even possible? This setlocale stuff is everywhere. Even basic functions like printf are affected. Reminds me of this incredible commit message: https://github.com/mpv-player/mpv/commit/1e70e82baa9193f6f027338b0fab0f5078971fbe https://github.com/mpv-player/mpv/commit/1e70e82baa9193f6f02... I found it easier to get rid of libc and write freestanding C instead. Linux system calls have none of these problems. This locale bullshit is nowhere to be found. No global state anywhere. No thread-local errno. No stupid stuff like EOF. Writing C became fun again.
      - kazinator 5y ago
        Although printf is affected, not every aspect of printf is affected. IIRC, one nasty area is floating-point numbers, when you're in the locale that doesn't have . as the decimal point, but your program's requirements are such that . is the decimal point. Like, oh, you're writing a programming language which specifies that. Like, you know, C does for floating-point constants.
        matheusmoreira 5y ago
        > floating-point numbers, when you're in the locale that doesn't have . as the decimal point I live in such a country. It's subtle but constant pain even for average users. Even the Windows calculator screws this up: it only accepts either . or , as input instead of both. > Like, oh, you're writing a programming language which specifies that. Yeah. The commit message I linked has one such example. > This is still less bad than that time when libquivi fucked up OpenGL rendering > calling a libquvi function would load some proxy abstraction library, which in turn loaded a KDE plugin > which in turn called setlocale() because Qt does this > made the mpv GLSL shader generation code emit "," instead of "." for numbers > and of course only for users who had that KDE plugin installed, and lived in a part of the world where "." is not used as decimal separator Just imagine debbuging this insanity.
        jcelerier 5y ago
        Yeah, as a french it bit me a lot of times. Now I just LC_ALL=C.UTF-8, french translations of english software are shitty most of the time anyways and in particular for dev tools.
        ygra 5y ago
        > Even the Windows calculator screws this up: it only accepts either . or , as input instead of both. It does, however, always accept the . key on the numeric keypad, so many users won't really notice the discrepancy.
      - gpderetta 5y ago
        If you had to takeaway just one thing from this glorious rant it would be: globals are evil.
  - emmelaich 5y ago
    The man pages used to warn that isalnum, ispunct etc were only defined if isascii was true. That seems to have disappeared - which supports your view.
guidovranken 5y ago
This also applies to C++ <locale> functions, like std::isspace. Another fun one: With FD_CLR, FD_ISSET, FD_SET you can corrupt memory by merely passing a socket descriptor that is not 0..1024. Pass a negative integer for some undefined behavior as well (shift by negative value occurs here [1]) [1] https://github.com/lattera/glibc/blob/895ef79e04a953cac1493863bcae29ad85657ee1/misc/sys/select.h#L56 https://github.com/lattera/glibc/blob/895ef79e04a953cac14938...
zx2c4 5y ago
Here are some branchless/constant-time versions of those functions that don't rely on locale: https://git.zx2c4.com/wireguard-tools/tree/src/ctype.h https://git.zx2c4.com/wireguard-tools/tree/src/ctype.h
- malkia 5y ago
  I like the suffix in 0x80001FU
gumby 5y ago
The rant behind her post (https://drewdevault.com/2020/09/25/A-story-of-two-libcs.html https://drewdevault.com/2020/09/25/A-story-of-two-libcs.html ), which has had some circulation, really shows its author’s limited perspective. glibc needs to solve two hard problems: be very fast and run on innumerable systems. Some of that conditional stuff is because all the world is not Linux or BSD; some of the macrology is there to make sure such handling is performed everywhere needed, and of course the preprocessor is the closest a language like C can get to preprocessing. I was in the code as glibc started to exist (we paid for a lot of it) and it looked like Musl: very straightforward.
- tyingq 5y ago
  I do get what you're saying, but musl also has to live in many different worlds. Using the example where glibc is trawling into endianness in the post you linked, for example. Musl runs on a bunch of different big and little endian router boxes and other unusual use cases. While I haven't tested, I'm guessing that their much simpler isalnum() works fine on all of them. Musl does have a lot less legacy to contend with, and musl is often much slower than glibc, so your point stands, of course.
  - masklinn 5y ago
    > While I haven't tested, I'm guessing that their much simpler isalnum() works fine on all of them. isalnum works fine of both, it only veers off when you get into UB which is UB. If you define “works fine” as “gives correct answers even in ub” then musl’s is completely broken since it only gives correct answers for english in ascii.
    - cryptonector 5y ago
      It can't give correct answers for anything other than English in UTF-8 locales. It can't give correct answers for any non-Latin scripts in any locales. The problem is ctype and POSIX. Given that, making ctype only work for ASCII (and maybe EBCDIC if you're really unlucky, which glibc is) is basically sufficient.
_kst_ 5y ago
IMHO the more interesting oddity about the functions declared in <ctype.h> is that they work with unsigned char, which means that they have undefined behavior if you pass a negative char value (other than EOF, which is typically -1). This means that if you have a char value (say, an element of a string), you need to cast it to unsigned char before passing it to any of the is*() functions.
anonymousiam 5y ago
Ran it on 32-bit ARM, 64-bit ARM, 32-bit x86, and 64-bit x86. All had different results, but all were the same until index 549, which is greater than the maximum value for unsigned char (255).
deleted 5y ago
[deleted]
floxy 5y ago
Looks like the array lookup isn't exclusive to glibc. Illumos: https://github.com/illumos/illumos-gate/blob/9ecd05bdc59e4a1091c51ce68cce2028d5ba6fd1/usr/src/lib/libc/port/locale/isdigit.c https://github.com/illumos/illumos-gate/blob/9ecd05bdc59e4a1... ...although there is a "sensible" version at: https://github.com/illumos/illumos-gate/blob/9ecd05bdc59e4a1091c51ce68cce2028d5ba6fd1/usr/src/uts/common/sys/ctype.h https://github.com/illumos/illumos-gate/blob/9ecd05bdc59e4a1... FreeBSD: You have to chase it through "__sbistype" to "__sbmaskrune". https://cgit.freebsd.org/src/tree/lib/libc/locale/isctype.c https://cgit.freebsd.org/src/tree/lib/libc/locale/isctype.c https://cgit.freebsd.org/src/tree/include/_ctype.h https://cgit.freebsd.org/src/tree/include/_ctype.h
- moonchild 5y ago
  Illumos also has ridiculously huge (but sanely-behaving, thankfully) strings functions, because somebody met jump tables once and thought they were cool. https://github.com/illumos/illumos-gate/blob/master/usr/src/lib/libc/amd64/gen/memset.s https://github.com/illumos/illumos-gate/blob/master/usr/src/... e.g.
- masklinn 5y ago
  A table-based implementation, aside from the convenience of being declarative, is also natural if not outright necessary for locales support.
deleted 5y ago
[deleted]
yesenadam 5y ago
What is supposed to be the problem? I can't see mentioned anywhere in the articles or comments what's actually meant to go wrong. Mine didn't crash (after 30 seconds or so) and I couldn't see a problem with the output.

70 points by picture 5y ago

Here's what the C standard says about character handling functions: > In all cases the argument is an int, the value of which shall be representable as an unsigned char or shall equal the value of the macro EOF. If the argument has any other value, the behavior is undefined. So this is just a case of glibc being optimized in a way that's really unforgiving if you commit that particular UB.

cryptonector 5y ago
No, this is a case of glibc trying to support localization of ctype in spite of the fact that it can't be localized to anything other than English in UTF-8 locales, anything other than Latin scripts in ISO-8859-* locales, or English in C/POSIX or EBCDIC locales. And then on top of that trying to be fast. I'd give up on supporting localization for ctype. This makes me think, too, "never use ctype, just hardcode my own that assumes ASCII".
- kazinator 5y ago
  I don't know about never, but try hard not to use anything whose behavior is influenced by a call to setlocale.
  - matheusmoreira 5y ago
    Is that even possible? This setlocale stuff is everywhere. Even basic functions like printf are affected. Reminds me of this incredible commit message: https://github.com/mpv-player/mpv/commit/1e70e82baa9193f6f027338b0fab0f5078971fbe https://github.com/mpv-player/mpv/commit/1e70e82baa9193f6f02... I found it easier to get rid of libc and write freestanding C instead. Linux system calls have none of these problems. This locale bullshit is nowhere to be found. No global state anywhere. No thread-local errno. No stupid stuff like EOF. Writing C became fun again.
    - kazinator 5y ago
      Although printf is affected, not every aspect of printf is affected. IIRC, one nasty area is floating-point numbers, when you're in the locale that doesn't have . as the decimal point, but your program's requirements are such that . is the decimal point. Like, oh, you're writing a programming language which specifies that. Like, you know, C does for floating-point constants.
      - matheusmoreira 5y ago
        > floating-point numbers, when you're in the locale that doesn't have . as the decimal point I live in such a country. It's subtle but constant pain even for average users. Even the Windows calculator screws this up: it only accepts either . or , as input instead of both. > Like, oh, you're writing a programming language which specifies that. Yeah. The commit message I linked has one such example. > This is still less bad than that time when libquivi fucked up OpenGL rendering > calling a libquvi function would load some proxy abstraction library, which in turn loaded a KDE plugin > which in turn called setlocale() because Qt does this > made the mpv GLSL shader generation code emit "," instead of "." for numbers > and of course only for users who had that KDE plugin installed, and lived in a part of the world where "." is not used as decimal separator Just imagine debbuging this insanity.
        jcelerier 5y ago
        Yeah, as a french it bit me a lot of times. Now I just LC_ALL=C.UTF-8, french translations of english software are shitty most of the time anyways and in particular for dev tools.
        ygra 5y ago
        > Even the Windows calculator screws this up: it only accepts either . or , as input instead of both. It does, however, always accept the . key on the numeric keypad, so many users won't really notice the discrepancy.
    - gpderetta 5y ago
      If you had to takeaway just one thing from this glorious rant it would be: globals are evil.
- emmelaich 5y ago
  The man pages used to warn that isalnum, ispunct etc were only defined if isascii was true. That seems to have disappeared - which supports your view.

guidovranken 5y ago

This also applies to C++ <locale> functions, like std::isspace. Another fun one: With FD_CLR, FD_ISSET, FD_SET you can corrupt memory by merely passing a socket descriptor that is not 0..1024. Pass a negative integer for some undefined behavior as well (shift by negative value occurs here [1]) [1] https://github.com/lattera/glibc/blob/895ef79e04a953cac1493863bcae29ad85657ee1/misc/sys/select.h#L56 https://github.com/lattera/glibc/blob/895ef79e04a953cac14938...

zx2c4 5y ago

Here are some branchless/constant-time versions of those functions that don't rely on locale: https://git.zx2c4.com/wireguard-tools/tree/src/ctype.h https://git.zx2c4.com/wireguard-tools/tree/src/ctype.h

malkia 5y ago
I like the suffix in 0x80001FU

gumby 5y ago

The rant behind her post (https://drewdevault.com/2020/09/25/A-story-of-two-libcs.html https://drewdevault.com/2020/09/25/A-story-of-two-libcs.html ), which has had some circulation, really shows its author’s limited perspective. glibc needs to solve two hard problems: be very fast and run on innumerable systems. Some of that conditional stuff is because all the world is not Linux or BSD; some of the macrology is there to make sure such handling is performed everywhere needed, and of course the preprocessor is the closest a language like C can get to preprocessing. I was in the code as glibc started to exist (we paid for a lot of it) and it looked like Musl: very straightforward.

tyingq 5y ago
I do get what you're saying, but musl also has to live in many different worlds. Using the example where glibc is trawling into endianness in the post you linked, for example. Musl runs on a bunch of different big and little endian router boxes and other unusual use cases. While I haven't tested, I'm guessing that their much simpler isalnum() works fine on all of them. Musl does have a lot less legacy to contend with, and musl is often much slower than glibc, so your point stands, of course.
- masklinn 5y ago
  > While I haven't tested, I'm guessing that their much simpler isalnum() works fine on all of them. isalnum works fine of both, it only veers off when you get into UB which is UB. If you define “works fine” as “gives correct answers even in ub” then musl’s is completely broken since it only gives correct answers for english in ascii.
  - cryptonector 5y ago
    It can't give correct answers for anything other than English in UTF-8 locales. It can't give correct answers for any non-Latin scripts in any locales. The problem is ctype and POSIX. Given that, making ctype only work for ASCII (and maybe EBCDIC if you're really unlucky, which glibc is) is basically sufficient.

_kst_ 5y ago

IMHO the more interesting oddity about the functions declared in <ctype.h> is that they work with unsigned char, which means that they have undefined behavior if you pass a negative char value (other than EOF, which is typically -1). This means that if you have a char value (say, an element of a string), you need to cast it to unsigned char before passing it to any of the is*() functions.

anonymousiam 5y ago

Ran it on 32-bit ARM, 64-bit ARM, 32-bit x86, and 64-bit x86. All had different results, but all were the same until index 549, which is greater than the maximum value for unsigned char (255).

deleted 5y ago

[deleted]

floxy 5y ago

Looks like the array lookup isn't exclusive to glibc. Illumos: https://github.com/illumos/illumos-gate/blob/9ecd05bdc59e4a1091c51ce68cce2028d5ba6fd1/usr/src/lib/libc/port/locale/isdigit.c https://github.com/illumos/illumos-gate/blob/9ecd05bdc59e4a1... ...although there is a "sensible" version at: https://github.com/illumos/illumos-gate/blob/9ecd05bdc59e4a1091c51ce68cce2028d5ba6fd1/usr/src/uts/common/sys/ctype.h https://github.com/illumos/illumos-gate/blob/9ecd05bdc59e4a1... FreeBSD: You have to chase it through "__sbistype" to "__sbmaskrune". https://cgit.freebsd.org/src/tree/lib/libc/locale/isctype.c https://cgit.freebsd.org/src/tree/lib/libc/locale/isctype.c https://cgit.freebsd.org/src/tree/include/_ctype.h https://cgit.freebsd.org/src/tree/include/_ctype.h

moonchild 5y ago
Illumos also has ridiculously huge (but sanely-behaving, thankfully) strings functions, because somebody met jump tables once and thought they were cool. https://github.com/illumos/illumos-gate/blob/master/usr/src/lib/libc/amd64/gen/memset.s https://github.com/illumos/illumos-gate/blob/master/usr/src/... e.g.
masklinn 5y ago
A table-based implementation, aside from the convenience of being declarative, is also natural if not outright necessary for locales support.

deleted 5y ago

[deleted]

yesenadam 5y ago

What is supposed to be the problem? I can't see mentioned anywhere in the articles or comments what's actually meant to go wrong. Mine didn't crash (after 30 seconds or so) and I couldn't see a problem with the output.