23 ms·

A Tale of Two Libcs

84 points by stargrave 6y ago

buster 6y ago
I am wondering what would happen, if someone swapped out the glibc implementation? Would this uncover the deeper technical reason for the implementation? Would suddenly bugs occur elsewhere? It probably just shows how old the glibc is and like most software entropy is increasing over time.
- rightbyte 6y ago
  I have not looked at the implementation but a lookup table would suggest EBCDIC support? Otherwise you would have to do like 7 range checks.
  - masklinn 6y ago
    The comment block mentions locales so I'd think non-C locales and extended-ascii (though non-ascii is also possible) e.g. in ISO-8859-9, 0xDD is İ (uppercase dotted i) and 0xFD is ı (lowercase dotless i) which you would expect to be an alphabetic letter in the turkic locale.
    - bonzini 6y ago
      Or à á é è í ì ò ó ù ú in pretty much anywhere in Europe other than the England (and also places like Greece, Serbia and Bulgaria which have a full alphabet of their own to handle in isalnum :)). Musl assumes your locale is either C or UTF-8, glibc doesn't. Sure almost everything is UTF-8 these days and has been in the last 15 years, but claiming that the glibc code makes no sense is utterly condescending and I expected ddevault to know better than that. I also wouldn't be surprised if those countries that have a non-Latin alphabet were still using single byte character encodings since UTF-8 would double the size of their text.
      - masklinn 6y ago
        > Or à á é è í ì ò ó ù ú in pretty much anywhere in Europe country other than the England (and also places like Greece, Serbia and Bulgaria which have a full alphabet of their own to handle in isalnum :)). That's more complicated because depending on the language these may or may not be considered separate letter e.g. the french alphabet has 26 letters: while required by the grammar, diacritics and ligatures are "extras". This means isalnum wouldn't necessarily return true for, say, ç in the french locale (I've no idea what it actually does). Turkish however does have 29 letters, and the dotted / dotless i is very much part of the alphabet (so are ç, ş, ğ, ö and ü). > Musl assumes your locale is either C or UTF-8 UTF-8 is an encoding, not a locale. And Musl just plain assumes the C locale, it clearly has no support whatsover for anything else. Which, for what that's worth, is a respectable choice: posix locales are really really bad, so not bothering with them is not necessarily a negative, but at the same time dinging glibc for supporting locales or old platform is not very honest.
        bonzini 6y ago
        > isalnum wouldn't necessarily return true for, say, ç in the french locale (I've no idea what it actually does). It returns true, otherwise "advance to the next word" or "grep -w" would be utterly broken. ("grep -w ça" would not match anything, and in fact it probably won't match anything under musl). Other European languages also treat characters with diacritics as separate letters, for example the Czech alphabet has 42 letters (one of which is "ch" which doesn't have its own Unicode character as far as I remember; if it did there would be more fun to be had with normalization and title case). > And Musl just plain assumes the C locale, it clearly has no support whatsover for anything else. Yeah what I meant is that if you are reading UTF-8 you won't be passing chars in the 128-255 range to the ctype functions. But musl doesn't implement Unicode wchar_t either.
        masklinn 6y ago
        > Yeah what I meant is that if you are reading UTF-8 you won't be passing chars in the 128-255 range to the ctype functions. But musl doesn't implement Unicode wchar_t either. The trap is that ctype functions take int, that's exactly the mistake TFAA made: they passed in an integer assuming isalnum would work with codepoints (which doesn't actually make sense since int is only required to be 16 bits).
rightbyte 6y ago
I guess the glibc complexity with a LUT is to support non-ascii character sets? It seems quite silly to compare a general solution to a hard coded special case.
- buster 6y ago
  I think one major point is the unreadable mess, though. Is there a reason for this?
  - rightbyte 6y ago
    Any big standard lib I laid my eyes on (gcc, clang, MS headers) seems to be a mess. I too wonder if the complexity is really necessary, but I guess so? Macros in my experience is ofent to fool the compiler to generate better code. Maybe that is a legacy reason?
    - ThreeFx 6y ago
      99% of the time the answer is speed. It's just way faster to evaluate and build all of this at compile time than at runtime. I understand both sides of the argument here - on one hand you have fast and unreadable code and on the other hand slower and more readable code. The problem here is 1. the leaky abstraction of characters and code points and 2. loss of type safety due to C's weak typing rules, which is a bit lost in the "hurr durr messy code."
    - DaiPlusPlus 6y ago
      Windows.h is still full of obsolete ancient macros that can’t be removed for compatibility. Things like references to near and far pointers (still!) and how every Win32 function with string params is a macro for the real A/W-suffixed functions, and all the A-suffixed functions now intentionally crash on post-98 systems.
      - tjalfi 6y ago
        > all the A-suffixed functions now intentionally crash on post-98 systems. Do you have a source for this claim? I find it hard to believe that Microsoft would drop support for ANSI applications.
pitterpatter 6y ago
Man, the condescension pouring off from the post. >So the fix is obvious at this point. Okay, fine, my bad. My code is wrong. I apparently cannot just hand a UCS-32 codepoint to isalnum and expect it to tell me if it’s between 0x30-0x39, 0x41-0x5A, or 0x61-0x7A. If that's what you want, just go ahead and write that.
- boris 6y ago
  And while at it, stop using all the "typical GNU bullshit". Let's see how far you will get.
  - kick 6y ago
    Drew...doesn't, actually, use much of it at all. I don't like the article, but you picked the one person who basically lives by what he rants about.
    - boris 6y ago
      Let's be specific, then, and exclude any software that is built with GNU make? First thing that goes is the Linux kernel (which can only be built with GNU make). As far as I know, Alpine still relies on the Linux kernel, so that's out.
      - deleted 6y ago
        [deleted]
  - pjmlp 6y ago
    Pretty far, ask Android, ChromeOS and any BSD user.
    - bonzini 6y ago
      Most BSD users I know (including most command line macOS users) do install GNU coreutils.
fancyfredbot 6y ago
TLDR; glibc is complicated, musl is not. libc supports thread specific locales, musl does not.
- masklinn 6y ago
  > libc supports locales, musl does not. The musl isalnum code is completely locale-unaware. Which you might argue is a good thing as POSIX locales are awful, but still.
minus7 6y ago
I was wondering whether there was a significant performance improvement of the table lookup glibc does over the naive implementation musl has. The answer is a pretty big yes. Glibc's isalnum is about 6-7 times faster than musl's. Benchmark code: https://paste.sr.ht/~minus/18b44cfe58789bc1fb69494130e859a1189d1772 https://paste.sr.ht/~minus/18b44cfe58789bc1fb69494130e859a11...
- saagarjha 6y ago
  Color me surprised, since locale stuff usually makes all these functions really slow…
  - JdeBP 6y ago
    Speaking as someone who wrote a Standard C library like this many years ago, I can confirm. All of the work is done inside setlocale(), switching (or constructing) tables. The character type functions devolve to a single table lookup. For those interested: The technique is used in P. J. Plauger's 1992 The Standard C Library and has been around a long time.
- ddevault 6y ago
  Here's the thing: isalnum has never been a bottleneck in my code. The sum amount of runtime that glibc's optimizations have saved my programs is almost certainly significantly less than the amount of time I had to spend untangling glibc's bullshit when it broke. Isn't "programmer time is more expensive than computer time" the lie that we've built this whole mess on anyway? And even if you want to use a LUT, other implementations have incorporated a LUT without this segfault issue, for instance plan 9: https://code.9front.org/hg/plan9front/file/00397858f3d8/sys/include/ctype.h https://code.9front.org/hg/plan9front/file/00397858f3d8/sys/... https://code.9front.org/hg/plan9front/file/00397858f3d8/sys/src/libc/port/ctype.c https://code.9front.org/hg/plan9front/file/00397858f3d8/sys/...
  - minus7 6y ago
    Absolutely, isalnum et al are probably never going to be a bottleneck. I was curious whether it even made a difference at all, after all, the naive implementation doesn't really do a lot. And I fully agree with you that this glibc code is awful. As evidenced by the Plan 9 implementation, table lookups can be done simple as well, and probably perform just the same.
pojntfx 6y ago
Switched to Alpine Linux (a distro built with musl libc) a while back for my dev and k3s cluster machines and never looked back since.
joosters 6y ago
I absolutely loathe and despise the arrogant 'your code must be bullshit' attitude that runs throughout this article. When you encounter other peoples' code and it looks a bit complicated, and not the 'obvious and simple' thing, your first thought should never be 'they wrote bullshit code', but instead 'what have I missed? What did I forget to consider for this function?' I don't know anything about the glibc authors, or their code style, but if you go through your life assuming incompetence if someone wrote a function differently to your initial thoughts, then you are an arrogant prick.
- iforgotpassword 6y ago
  I don't have a problem with strong language. Drew easily comes across as arrogant sometimes, and I don't agree with everything he says. But if there's something I'm sick of it's that stupid "well the spec says it's UB so let's just write code that FUCKS THE DEV as hard as possible." attitude in certain circles regarding a one letter programming language. The code in question is a classic array lookup with missing bounds check. If you write code like that and then blame people for using it wrong because the spec says so, you're just as arrogant, just in a different way.
- browserface 6y ago
  I actually loved it. I found it very entertaining. Tangent: surprises me how often comment people lose their shit when you utter the slightest word of criticism about some pet {language,code, tech}, but are absolutely fine, supportive and enabling when someone verbally eviscerates the living shit out of another human based on what they said, or their opinion. How is attacking a non human idea and no one in particular bad, but attacking a specific person viciously ok? I don't get that and I never did.
  - Munksgaard 6y ago
    Didn't a human being write the glibc implementation?
    - browserface 6y ago
deleted 6y ago
[deleted]

84 points by stargrave 6y ago

I am wondering what would happen, if someone swapped out the glibc implementation? Would this uncover the deeper technical reason for the implementation? Would suddenly bugs occur elsewhere? It probably just shows how old the glibc is and like most software entropy is increasing over time.

rightbyte 6y ago
I have not looked at the implementation but a lookup table would suggest EBCDIC support? Otherwise you would have to do like 7 range checks.
- masklinn 6y ago
  The comment block mentions locales so I'd think non-C locales and extended-ascii (though non-ascii is also possible) e.g. in ISO-8859-9, 0xDD is İ (uppercase dotted i) and 0xFD is ı (lowercase dotless i) which you would expect to be an alphabetic letter in the turkic locale.
  - bonzini 6y ago
    Or à á é è í ì ò ó ù ú in pretty much anywhere in Europe other than the England (and also places like Greece, Serbia and Bulgaria which have a full alphabet of their own to handle in isalnum :)). Musl assumes your locale is either C or UTF-8, glibc doesn't. Sure almost everything is UTF-8 these days and has been in the last 15 years, but claiming that the glibc code makes no sense is utterly condescending and I expected ddevault to know better than that. I also wouldn't be surprised if those countries that have a non-Latin alphabet were still using single byte character encodings since UTF-8 would double the size of their text.
    - masklinn 6y ago
      > Or à á é è í ì ò ó ù ú in pretty much anywhere in Europe country other than the England (and also places like Greece, Serbia and Bulgaria which have a full alphabet of their own to handle in isalnum :)). That's more complicated because depending on the language these may or may not be considered separate letter e.g. the french alphabet has 26 letters: while required by the grammar, diacritics and ligatures are "extras". This means isalnum wouldn't necessarily return true for, say, ç in the french locale (I've no idea what it actually does). Turkish however does have 29 letters, and the dotted / dotless i is very much part of the alphabet (so are ç, ş, ğ, ö and ü). > Musl assumes your locale is either C or UTF-8 UTF-8 is an encoding, not a locale. And Musl just plain assumes the C locale, it clearly has no support whatsover for anything else. Which, for what that's worth, is a respectable choice: posix locales are really really bad, so not bothering with them is not necessarily a negative, but at the same time dinging glibc for supporting locales or old platform is not very honest.
      - bonzini 6y ago
        > isalnum wouldn't necessarily return true for, say, ç in the french locale (I've no idea what it actually does). It returns true, otherwise "advance to the next word" or "grep -w" would be utterly broken. ("grep -w ça" would not match anything, and in fact it probably won't match anything under musl). Other European languages also treat characters with diacritics as separate letters, for example the Czech alphabet has 42 letters (one of which is "ch" which doesn't have its own Unicode character as far as I remember; if it did there would be more fun to be had with normalization and title case). > And Musl just plain assumes the C locale, it clearly has no support whatsover for anything else. Yeah what I meant is that if you are reading UTF-8 you won't be passing chars in the 128-255 range to the ctype functions. But musl doesn't implement Unicode wchar_t either.
        masklinn 6y ago
        > Yeah what I meant is that if you are reading UTF-8 you won't be passing chars in the 128-255 range to the ctype functions. But musl doesn't implement Unicode wchar_t either. The trap is that ctype functions take int, that's exactly the mistake TFAA made: they passed in an integer assuming isalnum would work with codepoints (which doesn't actually make sense since int is only required to be 16 bits).

rightbyte 6y ago

I guess the glibc complexity with a LUT is to support non-ascii character sets? It seems quite silly to compare a general solution to a hard coded special case.

buster 6y ago
I think one major point is the unreadable mess, though. Is there a reason for this?
- rightbyte 6y ago
  Any big standard lib I laid my eyes on (gcc, clang, MS headers) seems to be a mess. I too wonder if the complexity is really necessary, but I guess so? Macros in my experience is ofent to fool the compiler to generate better code. Maybe that is a legacy reason?
  - ThreeFx 6y ago
    99% of the time the answer is speed. It's just way faster to evaluate and build all of this at compile time than at runtime. I understand both sides of the argument here - on one hand you have fast and unreadable code and on the other hand slower and more readable code. The problem here is 1. the leaky abstraction of characters and code points and 2. loss of type safety due to C's weak typing rules, which is a bit lost in the "hurr durr messy code."
  - DaiPlusPlus 6y ago
    Windows.h is still full of obsolete ancient macros that can’t be removed for compatibility. Things like references to near and far pointers (still!) and how every Win32 function with string params is a macro for the real A/W-suffixed functions, and all the A-suffixed functions now intentionally crash on post-98 systems.
    - tjalfi 6y ago
      > all the A-suffixed functions now intentionally crash on post-98 systems. Do you have a source for this claim? I find it hard to believe that Microsoft would drop support for ANSI applications.

pitterpatter 6y ago

Man, the condescension pouring off from the post. >So the fix is obvious at this point. Okay, fine, my bad. My code is wrong. I apparently cannot just hand a UCS-32 codepoint to isalnum and expect it to tell me if it’s between 0x30-0x39, 0x41-0x5A, or 0x61-0x7A. If that's what you want, just go ahead and write that.

boris 6y ago
And while at it, stop using all the "typical GNU bullshit". Let's see how far you will get.
- kick 6y ago
  Drew...doesn't, actually, use much of it at all. I don't like the article, but you picked the one person who basically lives by what he rants about.
  - boris 6y ago
    Let's be specific, then, and exclude any software that is built with GNU make? First thing that goes is the Linux kernel (which can only be built with GNU make). As far as I know, Alpine still relies on the Linux kernel, so that's out.
    - deleted 6y ago
      [deleted]
- pjmlp 6y ago
  Pretty far, ask Android, ChromeOS and any BSD user.
  - bonzini 6y ago
    Most BSD users I know (including most command line macOS users) do install GNU coreutils.

fancyfredbot 6y ago

TLDR; glibc is complicated, musl is not. libc supports thread specific locales, musl does not.

masklinn 6y ago
> libc supports locales, musl does not. The musl isalnum code is completely locale-unaware. Which you might argue is a good thing as POSIX locales are awful, but still.

minus7 6y ago

I was wondering whether there was a significant performance improvement of the table lookup glibc does over the naive implementation musl has. The answer is a pretty big yes. Glibc's isalnum is about 6-7 times faster than musl's. Benchmark code: https://paste.sr.ht/~minus/18b44cfe58789bc1fb69494130e859a1189d1772 https://paste.sr.ht/~minus/18b44cfe58789bc1fb69494130e859a11...

saagarjha 6y ago
Color me surprised, since locale stuff usually makes all these functions really slow…
- JdeBP 6y ago
  Speaking as someone who wrote a Standard C library like this many years ago, I can confirm. All of the work is done inside setlocale(), switching (or constructing) tables. The character type functions devolve to a single table lookup. For those interested: The technique is used in P. J. Plauger's 1992 The Standard C Library and has been around a long time.
ddevault 6y ago
Here's the thing: isalnum has never been a bottleneck in my code. The sum amount of runtime that glibc's optimizations have saved my programs is almost certainly significantly less than the amount of time I had to spend untangling glibc's bullshit when it broke. Isn't "programmer time is more expensive than computer time" the lie that we've built this whole mess on anyway? And even if you want to use a LUT, other implementations have incorporated a LUT without this segfault issue, for instance plan 9: https://code.9front.org/hg/plan9front/file/00397858f3d8/sys/include/ctype.h https://code.9front.org/hg/plan9front/file/00397858f3d8/sys/... https://code.9front.org/hg/plan9front/file/00397858f3d8/sys/src/libc/port/ctype.c https://code.9front.org/hg/plan9front/file/00397858f3d8/sys/...
- minus7 6y ago
  Absolutely, isalnum et al are probably never going to be a bottleneck. I was curious whether it even made a difference at all, after all, the naive implementation doesn't really do a lot. And I fully agree with you that this glibc code is awful. As evidenced by the Plan 9 implementation, table lookups can be done simple as well, and probably perform just the same.

pojntfx 6y ago

Switched to Alpine Linux (a distro built with musl libc) a while back for my dev and k3s cluster machines and never looked back since.

joosters 6y ago

I absolutely loathe and despise the arrogant 'your code must be bullshit' attitude that runs throughout this article. When you encounter other peoples' code and it looks a bit complicated, and not the 'obvious and simple' thing, your first thought should never be 'they wrote bullshit code', but instead 'what have I missed? What did I forget to consider for this function?' I don't know anything about the glibc authors, or their code style, but if you go through your life assuming incompetence if someone wrote a function differently to your initial thoughts, then you are an arrogant prick.

iforgotpassword 6y ago
I don't have a problem with strong language. Drew easily comes across as arrogant sometimes, and I don't agree with everything he says. But if there's something I'm sick of it's that stupid "well the spec says it's UB so let's just write code that FUCKS THE DEV as hard as possible." attitude in certain circles regarding a one letter programming language. The code in question is a classic array lookup with missing bounds check. If you write code like that and then blame people for using it wrong because the spec says so, you're just as arrogant, just in a different way.
browserface 6y ago
I actually loved it. I found it very entertaining. Tangent: surprises me how often comment people lose their shit when you utter the slightest word of criticism about some pet {language,code, tech}, but are absolutely fine, supportive and enabling when someone verbally eviscerates the living shit out of another human based on what they said, or their opinion. How is attacking a non human idea and no one in particular bad, but attacking a specific person viciously ok? I don't get that and I never did.
- Munksgaard 6y ago
  Didn't a human being write the glibc implementation?
  - browserface 6y ago

deleted 6y ago

[deleted]