5 ms·
Because glibc and GNU set a terrible precedent. On GNU/Linux systems the shared binary interpreter / loader, GCC compiler, the C library and the system C/C++ AB
by okanat 1mo ago
Because glibc and GNU set a terrible precedent. On GNU/Linux systems the shared binary interpreter / loader, GCC compiler, the C library and the system C/C++ ABI all depend into each other. You cannot change any of them independently. All shared libraries depend on the specific glibc version to load them into memory to be able to use that specific glibc version as their C library and make calls like dlopen.
Shared libraries have always been broken in Linux. Unfortunately many things like GPU drivers, graphics libraries and NSS need shared libraries to dynamically load certain runtimes (because you don't want to load all possible GPU drivers in existence to your RAM). So an ecosystem has been developed on top of terrible ABI and architecture GNU/glibc provided.
- uecker 1mo agoIn what sense do binary interpreter / loader, GCC compiler, C library and system C/C++ ABI dependent on each other? I have certainly mixed different versions of all these components without problems so far.
- okanat 1mo agoWhen you compile GCC you need to provide a full glibc installation as your target. It is also a dependency of libstdc++. C++ global/static variable initialization depends on the specific version of glibc (they don't usually break compat, but they can and they did in the past) which also provides ld-linux.so that loads those global variable placeholders in the correct manner such that glibc and libstdc++ can initialize them correctly. This is just one example. Thread local variables and behavior of things like pthreads with signal, fork etc all depend on glibc.
- uecker 1mo agoI can't comment on the C++, I can imagine there plenty of issues, but for C I don't see this. You need some libc if you compile with gcc, but this generally does not introduce a hard version dependency on the specific version (there may be a minimum requirement if you compile against a new version that a symbol with a different ABI).
- pg83 1mo agoFor example, the itanium unwind ABI implementation lies between these three entities.
- duped 1mo agoThe interpreter/loader is glibc and a key part of bootstrapping an executable built against glibc is loading libc itself before continuing on to load the program. Versioning is a problem when distributing binaries linked against a newer glibc to distros that ship an older one. The C compiler doesn't really care as much.
- okanat 1mo ago> The C compiler doesn't really care as much. Until you define a thread local variable (C11) or use atomics (also C11) or define a global with an initial value. Then it happily generates code that depends on "whatever my target glibc + ld-linux.so needs".
- uecker 1mo agoIt depends on functions defined in a standardized ABI.
- okanat 1mo agoYou'd expect that but, no. That's why you cannot load glibc-linked binaries in a Musl distro. Edit: that's why the hacks like the original post is needed, as well. The ABI is strongly dependent on explicit libc implementation in current Linux systems. There is no libc independent ABI on Linux.
- uecker 1mo agoSorry, can you be more specific. I do not understand what the problem is. If Musl does not implement support for the ABI, this would be a musl problem?
- okanat 1mo agoThere is no libc independent ABI. ABI doesn't purely mean just calling conventions. When you compile libc, you also get a binary loader ld-linux.so with it. They are not two independent components of a system. Basically all .so files compiled with glibc require the ld-linux.so that's also generated by that glibc (or a later version, if they didn't break the binary compatibility). There are a lot of stuff that's executed by ld-linux.so and glibc that are not explicitly documented but they are absolutely necessary for your program to start and correctly initialize things like global variables or signal handling or loading other dynamic libraries. Some of that functionality sits in ld-linux.so and some of that in glibc. They have circular dependencies to each other. glibc expects ld-linux.so to put things in certain order but ld-linux.so also must load glibc first to have access to certain APIs. They are not part of System V ABI. They are not documented. Musl maybe can implement this but it is simply reverse engineering what glibc did and then playing a game of cat and mouse. There is no independent ABI standard.
- mananaysiempre 1mo agoI don’t imagine that you’re unaware of any of this, but: ld.so and libc.so are heavily interdependent in deliberately undocumented ways with Glibc and outright the same file with shared Musl. And while you might usually get away with using any old GCC with the right architecture and ABI (especially for C; cf the musl-gcc hack), technically it needs to be built to target a specific libc version (particularly via symbol versioning; I’ve long wanted to gather a set of patches to build an old Glibc and subsequently a cross-compiler using a new GCC so I could avoid PyPA’s manylinux monster or its moral equivalent for compatible dynamic binaries in simple cases). The C compiler of course is tied to the C ABI, and this wouldn’t be really worth mentioning except for the time where the GCC devs accidentally the whole SysV i386 ABI and pretended that the stack was always 16-byte aligned, why do you ask, except on RHEL. The C++ parts I can’t really comment on.
- uecker 1mo agoI am not really sure. For ld.so and libc.so I may believe this. The C ABI is very stable, and if you use a new symbol from a newer glibc, you certainly depend on it, but this can also be avoided. In any case, I do not see what is fundamentally misdesigned here. I can't quite image how it could work differently. If you upgrade something so that the e ABI changed you natually need to update other components. Static linking certainly seems a very poor replacement for this.
- account42 1mo ago> I’ve long wanted to gather a set of patches to build an old Glibc and subsequently a cross-compiler using a new GCC so I could avoid PyPA’s manylinux monster or its moral equivalent for compatible dynamic binaries in simple cases And that's the correct approach and also one that many have taken. We just need someone willing to maintain that as an easy mode SDK for everyone.
- pg83 1mo ago> And that's the correct approach. Who said that? This approach has many problems that have already been discussed here, not to mention the fact that it leaves Alpine and Bionic-based systems out in the cold. Let me remind you that Bionic is the most widespread libc in the Linux world, and Alpine is the most popular Docker layer. The world doesn't end with glibc. And it doesn't begin with it.
- asveikau 1mo ago> system C/C++ ABI C++ abi should not be included in this. It is independent from the other pieces and historically a source of incompatibility on its own. Saying "C/C++ abi" as if they are the same is looney tunes, the former is very simple and stable and the latter is very complex.
- okanat 1mo agoSee: https://news.ycombinator.com/item?id=49355262 https://news.ycombinator.com/item?id=49355262 How libstdc++ initializes global variables absolutely depends on glibc and ld-linux.so. That is part of C++ ABI.
- account42 1mo agoThat's libstdc++ depending on a C ABI (or a GNU ABI to be precise) but that doesn't make it a C++ ABI. You can also have global constructors in C with GCC.
- josefx 1mo ago> the former is very simple Somehow most of my portability issues seem to be caused by glibc, its symbol versioning and close ties to the dynamic loader. Minor versions aren't compatible, no two Linux distros ship the same version and you can't just provide your own without also patching in your own dynamic loader. At least as far as the defaults on Linux go I consider C the root of all evil.
- badsectoracula 1mo agoAFAIK glibc is backwards compatible as long as a) your program is running on a newer version (i.e. you're not trying to dynamically link against an older version) and b) you're not using hidden/undocumented symbols. In which case as long as you're using the documented public API and compile your program with the oldest version of glibc you want to support (some Ubuntu from 4-5 years ago should cover pretty much every current desktop) you should be fine. And with something like Docker this is trivial to do. Sure it is annoying that you cannot use your current distro (especially if you use some rolling distro) to make binaries for everyone, but it takes very little effort to work around that. The only issue i can think of is if you absolutely want to compile using the latest version of your compiler and you cannot build the compiler from source to work in the Docker (or whatever) contain to work against the older glibc.
- b5n 1mo agoWhile I don't disagree with some of the pain you describe, you conveniently gloss over the fact that gnu developed a system that worked, and then made it free to everyone to consult and use.
- okanat 1mo agoBSD also did it. They did it better. Maybe more modern but AOSP also did it but at a different level of binary: instead of ELF, using compiled Java bytecode archives.
- lmz 1mo agoThe root cause is alternative libc implementations. Are BSD syscalls considered stable? I remember Go moving to use libc on OpenBSD. Solaris also has the libc as the stable interface. Linux kernel is an outlier here guaranteeing stable syscalls but you wanting to use another libc is not Glibc's problem.
- duped 1mo ago> all shared libraries depend on the specific glibc version to load them Not really, though. glibc uses symbol versions that are forward but not backward compatible. If you got an error that said "this program was built for a newer version of <distro>" would you say the same thing? Note this is the same (if not worse) on MacOS, and on windows you used to distribute the CRT with your application just to deal with the same problem.
- okanat 1mo agoSee https://news.ycombinator.com/item?id=49355262 https://news.ycombinator.com/item?id=49355262 . Yes glibc has some backwards compat but you cannot load a binary compiled with a newer version of glibc using an older ld-linux.so. That's because the interdependency. Nor you can load binaries that depend on different libc.so files with glibc systems I cannot comment on macOS, I have never used it. However this is not a problem with Windows. You can ship a newer CRT or you can install it as a system component using Microsoft's MSI. The dependency is one way on Windows. CRT purely depends on Win32. Moreover the loader is completely independent and DLLs are loaded into their own unique scoped namespace unlike Linux that loads them in global symbol namespace. That's why you can mix and match DLLs compiled for different CRT versions.
- duped 1mo agoThat's what I'm saying though, glibc-linked binaries are forwards (but not backwards) compatible.
- okanat 1mo agoIt is not just compatibility. You cannot load them into the memory with your system dynamic loader. You need to also ship ld-linux.so with the new version of glibc you have, if you were to distribute your program independently. On Windows you don't need to ship a new binary loader. I can just ship Windows 10 UCRT DLL (which is the new libc of Windows) to Vista and my binaries will work. The binary loader isn't interlinked with the libc.
- adev_ 1mo ago> All shared libraries depend on the specific glibc version to load them into memory to be able to use that specific glibc version as their C library That's currently the real core of the problem. The loader (and libdl) need to be decoupled from the glibc itself under Linux. Without that, any attempt to ship static binaries (or any binary with a different Libc) will be a source of perpetual pain. nss plugins and its associated pain (sssd and avahi) are an other examples of that.
- inigyou 1mo agolibdl is the loader FWIW. It's all one big glibc codebase.
- adev_ 1mo agoWhat we name the loader is typically ld.so, more precisely /lib/ld-linux-x86-64.so.2 under x86_64. But I do agree with you, the glibc is all one giant codebase shipped in one giant blob, dispatch under several names. Even libdl.so is barely more than a wrapper and some glue around the libc.so itself.
- krupan 1mo agoIn what way are they "broken" when Linux runs fine on millions of boxes? Sure, it might be a pain for proprietary software, but if your app is open source it's not that hard to build it on whichever distro you want. If your app is popular enough the distro maintainer will build it for you
- matheusmoreira 1mo agoYeah, it's mind boggling how everything hard depends on glibc, even critical graphics systems. I've become obsessed with getting rid of it, especially after I realized that contributing to GNU itself was a dead end. Freestanding Linux programming turned out to be much more fun anyway. All libraries out there should adopt the SQLite design: programmers provide it with all the necessary functions. Instead of libraries hard depending on glibc, we get to inject the libc-ish subset it needs. Then we can use whatever we want under the hood. I'm working on porting SQLite to freestanding Linux system calls so it can run with zero dependencies. Wish I could say the same for software like mesa, I'd need a lot of help for this one...
- yxhuvud 1mo agoModularizing (and versioning each part independently) libc would go a long way. There is also a need to separate the stuff needed for system integration with what is necessary for users of the C language to actually do stuff.
- matheusmoreira 1mo ago> Modularizing Can't be done. The libc is legacy, it can't be changed without breaking everything. It's also mandatory on every operating system other than Linux. A change in paradigm is necessary. Freestanding C, not hosted C. This completely gets rid of the libc and is a surprisingly clean language. Linux only, because it's the only kernel with a stable binary interface. Every other OS forces a C runtime. I once worked on a liblinux project that embodied this... Stopped because Linux itself has a nolibc thing in the kernel tree and I didn't want to compete with it. Now I'm working on the Rust version. > what is necessary for users of the C language to actually do stuff Surprisingly little. I wrote an entire lisp interpreter in freestanding C with Linux system calls. It managed to survive for a rather long time without any memory allocation at all. The system layer is refreshingly tiny. It consists of a memory allocator and extremely basic functions like memmove and strlen. I successfully got rid of total nonsense like thread local errno, locales, implicit buffering, cached global state, possibly more. All that stuff is gone! Exactly one global survived: the stack canary generated by GCC and clang. Every other symbol in the ELF is controlled by me. Wasn't able to get rid of the NUL terminator. Linux itself needs it. To get rid of that little billion dollar mistake requires an entirely new kernel with zero UNIX/POSIX influence. I had to make my peace with that one. All my buffers maintain an extra NUL byte at the end.
- Root_Access 1mo agoMost everyone else is talking about the problem while you mapped the room. Solo is basically pg83's answer to the architecture you just described. If there is no libc independent ABI, build your own loader and shim the boundary. I can understand Linus's obsession with taste and the areas it was overlooked or traded.
- torginus 1mo agoWasn't Linux Standard Base supposed to fix this? I cannot imagine any reason why glibc would break at a rate that you can't keep the current version binary compatible for years. Or do the MS thing, and ship multiple versions like msvcrt
- okanat 1mo ago> Wasn't Linux Standard Base supposed to fix this? I cannot imagine any reason why glibc would break at a rate that you can't keep the current version binary compatible for years. Guess what happened with LSB? Some distros like Debian never adopted it. It also broke backwards compatibility at random times. It is now abandoned. > Or do the MS thing, and ship multiple versions like msvcrt MS can do that since the CRT is decoupled from the system DLL loader and Windows ABI is designed to isolated loaded DLLs into their own symbol namespaces. So your program can load DLLs that depend on a different CRT version. Glibc's design tightly couples the system loader with the specific libc. Moreover symbols in ELF files are loaded into a single global namespace. You cannot ship shared libraries that are compiled with a different version of Glibc or Musl libc.
- torginus 1mo agoBtw, absolutely insane that it's 2026, and Linux cannot do the most defining OS thing - that is, provide a standardized environment to run binaries against. All solutions to this problem are hacky, complex and controversal and highly fragmented, where this should be BASIC functionality
- okanat 1mo agoIt's not insane. Many people who were in the GNU organisation wanted the OS to be as hostile against binaries as possible to prevent people distributing proprietary software with GNU. That GNU became the userspace layer. It isn't that this is 2026 and it was a bug that hasn't been fixed since. The userspace of GNU is designed against what you consider as basic functionality.