8 ms·

They are a problem because gcc automatically links to the latest version of glibc. As to why they don't add an option to specify an older version? I don't know

They are a problem because gcc automatically links to the latest version of glibc. As to why they don't add an option to specify an older version? I don't know either and it is rather annoying to have to use docker images of older OSes to target older glibc versions. It's just one of many things that prevents linux from being as popular as windows for desktop users.

IshKebab 2y ago
I think the main reason they don't offer a `--make-my-binary-compatible-with-the-ancient-linux-versions-users-often-have` is that GCC/glibc is a GNU project and the are philosophically against distributing software as binaries. I don't think there's any technical reason why it couldn't be done. To be fair to them though, Mac has the same problem. I worked at a company where we had to keep old Mac machines to produce compatible binaries, and Apple makes it hard to even download old versions of MacOS and Xcode. I guess the difference is MacOS is easy to upgrade so you don't have to support versions from 13 years ago or whatever like you do with glibc.
- codexon 2y ago
  > I think the main reason they don't offer a `--make-my-binary-compatible-with-the-ancient-linux-versions-users-often-have` is that GCC/glibc is a GNU project and the are philosophically against distributing software as binaries. You don't have to statically compile glibc, gcc just needs an option to tell the compiler to target say, version 2.14 instead of the latest one. The newest glibc has all the older versions in it. That's why you can compile on say ubuntu 14 and have it run on ubuntu 24.
  - saurik 2y ago
    No like, the point is that the only reason you (and I: I do this all the time, including with my open source software... like: no judgment) want to target some old version of glibc is so you can distribute that binary to people without caring as much about what version of the OS they have; but that would be unnecessary if you just gave them the source code and have them compile their own copy for their system targeting the exact libraries they have.
    - codexon 2y ago
      Unfortunately most people don't want to bother compiling, myself included. I tried gentoo one time and it took 1 hour to compile 5 minutes worth of apt-get on ubuntu.
  - fweimer 2y ago
    Only the dynamically linked bits, the statically linked startup code and libc_nonshared.a are missing from newer versions. Most programs don't need them (who needs working ELF constructors in the main program?). The libc_nonshared.a bits can be reimplemented from scratch easily enough (but we should switch them over to header-only implementations eventually).
  - IshKebab 2y ago
    Yes exactly. That's what the flag would do. But it doesn't exist.
- fweimer 2y ago
  I used to think that binary compatibility benefits proprietary applications, but I'm not so sure anymore. From a commercial perspective, when we break binary compatibility (not that we want to), it's an opportunity for selling more stuff. Many distributions do periodic mass rebuilds anyway and do not need that much long-term ABI compatibility. Binary compatibility seems mostly for people who compile their own software, but have not automated that and therefore couldn't keep up with updates if there wasn't ABI compatibility.
  - IshKebab 2y ago
    I agree. It's annoying for closed source apps but they generally have the resources to deal with it anyway. E.g. with Questa I can just unzip it and run it. No trouble. It's disproportionately annoying for open source projects who don't want to waste their time dealing with this.
forrestthewoods 2y ago
> They are a problem because gcc automatically links to the latest version of glibc. As to why they don't add an option to specify an older version? Because glibc and ld/lld are badly designed. glibc is stuck in the 80s with awful and unnecessary automagic configure steps. ld/lld expect a full and complete shared library to exist when compiling even though it expects a different shared library to exist in the future. Zig solves the glibc linking issue. You can trivially target any old version for any supported target platform. The only thing you actually need are headers and a thin, implementation free lib that contains stub functions. Unfortunately glibc is not architected to make this trivial. But this is just because glibc is stuck with decades of historic cruft, not because it's actually a hard problem.
- einpoklum 2y ago
  > awful and unnecessary automagic configure steps Steps taken when? When building glibc? And - what steps? > ld/lld expect a full and complete shared library to exist when compiling ... But ld and lld are linkers... > Zig solves the glibc linking issue. But Zig is a language. Do you mean the Zig standard library? The Zig compiler? > The only thing you actually need are headers and a thin, implementation free lib that contains stub functions. Why do you need stub functions at all, if you're not actually using them?
  - deleted 2y ago
    [deleted]
  - ploxiln 2y ago
    The zig compiler can compile C and C++, using llvm, and it also packages various libc implementations, including glibc, musl, mingw, and msvc, and more for other OSes. Some people use it as a more convenient golang-like cross-compiler. And this whole combination is a smaller download and install than most other toolchains out there. It just took some inspiration and grunt work, to dissect the necessary (processed) headers and other bits of each libc ... hats of to the zig devs. Random blog post I just found about it: https://ruoyusun.com/2022/02/27/zig-cc.html https://ruoyusun.com/2022/02/27/zig-cc.html
bregma 2y ago
Hmm. The MSVCRT.DLL/MSVCRTD.DLL not being binary compatible between releases of Visual Studio is the same thing, except of course you can't even combine some modules compiled for debug with modules compiled without debug in the same executable. The Windows problem has always been so so much worse that pretty much all developers simple resorted to shipping the OS system runtime with every package and it's just expected nowadays. It's where the phrase "DLL hell" originated, after all. Not to say the ABI problem isn't real if you want to combine binary packages from different Linux-based OSes. Plenty of solutions for that have cropped up as well: containers, flatpacks, snaps, the list goes on.
- forrestthewoods 2y ago
  > The Windows problem has always been so so much worse Hard, hard disagree. The problems are somewhat comparable. But if any platform is more painful it's Linux. Although they're similar if you exclude glibc pain. At least in my personal experience of writing lots of code that needs to run on win/mac/linux/android. > pretty much all developers simple resorted to shipping the OS system runtime with every package Meanwhile Linux developers have resorted to shipping an entire OS via docker to run every program. Because managing Linux environment dependencies is so painful you have to package the whole system. Needing to docker to simply launch a program is so embarrassing. > except of course you can't even combine some modules compiled for debug with modules compiled without debug in the same executable That's not any different on Linux. That has more to do with C++.
  - graemep 2y ago
    > Meanwhile Linux developers have resorted to shipping an entire OS via docker to run every program. > Needing to docker to simply launch a program is so embarrassing. I have never needed docker "just to launch a program". Docker makes it easy to provide multiple containerised copies of an identical environment. Containers are a light alternative to VM images. I assume you find the existence of Windows containers just as embarrassing? https://learn.microsoft.com/en-us/virtualization/windowscontainers/about/ https://learn.microsoft.com/en-us/virtualization/windowscont...
Sesse__ 2y ago
Linking to the latest version of glibc is, in itself, not a problem -- glibc hasn't bumped its soname in ages, it is using symbol versioning instead. So you only get a problem if you use a symbol that doesn't exist in older glibc (i.e., some specific interface that you are using changed). As for using an older version of glibc, _linking_ isn't the problem -- swapping out the header files would be. You can probably install an old version of the header files somewhere else and just -I that directory, but I've never tried. libstdc++ would probably be harder, if you're in C++ land.
- fweimer 2y ago
  Recent libstdc++ has a _dl_find_object@GLIBC_2.35 dependency, so it's not exactly trivial anymore to link a C++ program against a older, side-installed glibc version because it won't have that symbol. It's possible to work around that (link against a stub that has _dl_find_object@GLIBC_2.35 as a compat symbol, so that libstdc++ isn't rejected), but linking statically is much more difficult because libstc++.a (actually, libgcc_eh.a) does not have the old code anymore that _dl_find_object replaces (once GCC is built against a glibc version that has _dl_find_object). This applies to other libraries as well because there are new(ish) math functions, strlcpy, posix_spawn extensions etc. that seem to be quite widely used already.
  - ynik 2y ago
    I find it's best to treat this as a case of "cross-compilation to old glibc version". That is, you don't want to just link against an old glibc version, you want a full cross-compilation toolchain where the compiler does not pick up headers from `/usr/include` by default (but instead has its own copy of the system headers), and where libstdc++ is also linked against that old glibc version. We used https://crosstool-ng.github.io/ https://crosstool-ng.github.io/ to create such a "cross-compiler". Now it doesn't matter which distribution our developers use, the same source code will always turn into the same binary (reproducible builds, yay!) and those binaries will work on older distributions than the developers are using. This allows us to ship dynamically linked linux executables; our linux customers can just unzip + run, same as our windows customers, no need to mess with docker containers. The downside is that we can't just use a library by `apt install`ing it, everything needs to be built with the cross compilation toolchain.
zX41ZdbW 2y ago
I've made a library named "glibc-compatibility": https://github.com/ClickHouse/ClickHouse/tree/master/base/glibc-compatibility https://github.com/ClickHouse/ClickHouse/tree/master/base/gl... When linking with this library before glibc, the resulting binary will not depend on the new symbol versions. It will run on glibc 2.4 and on systems as old as Ubuntu 8.04 and CentOS 5 even when built on the most modern system.
- thrtythreeforty 2y ago
  How does this work?
  - threecheese 2y ago
    In case parent didn’t see your q: https://www.lightofdawn.org/wiki/wiki.cgi/NewAppsOnOldGlibc https://www.lightofdawn.org/wiki/wiki.cgi/NewAppsOnOldGlibc
- adastra22 2y ago
  You are a hero.
- cataphract 2y ago
  An easier option is to build against musl, remove the DEPENDS with patchelf, and use that. There are a few incompatibilities with glibc (more on aarch64 than am64), but it's manageable (you might need to patch a few headers in musl and provide missing/incompatible functions). And you'll have a binary that works against musl and glibc both.
- kzrdude 2y ago
  Here is another interesting tool, this one for patching executables https://github.com/corsix/polyfill-glibc https://github.com/corsix/polyfill-glibc
  - superkuh 2y ago
    Thank you! This has solved multiple problems I've had for years.
LtWorf 2y ago
chroot has existed for many years.
sunshowers 2y ago
If you're building Rust, check out cargo zigbuild (yes, zigbuild) which lets you target old glibc.

IshKebab 2y ago

I think the main reason they don't offer a `--make-my-binary-compatible-with-the-ancient-linux-versions-users-often-have` is that GCC/glibc is a GNU project and the are philosophically against distributing software as binaries. I don't think there's any technical reason why it couldn't be done. To be fair to them though, Mac has the same problem. I worked at a company where we had to keep old Mac machines to produce compatible binaries, and Apple makes it hard to even download old versions of MacOS and Xcode. I guess the difference is MacOS is easy to upgrade so you don't have to support versions from 13 years ago or whatever like you do with glibc.

codexon 2y ago
> I think the main reason they don't offer a `--make-my-binary-compatible-with-the-ancient-linux-versions-users-often-have` is that GCC/glibc is a GNU project and the are philosophically against distributing software as binaries. You don't have to statically compile glibc, gcc just needs an option to tell the compiler to target say, version 2.14 instead of the latest one. The newest glibc has all the older versions in it. That's why you can compile on say ubuntu 14 and have it run on ubuntu 24.
- saurik 2y ago
  No like, the point is that the only reason you (and I: I do this all the time, including with my open source software... like: no judgment) want to target some old version of glibc is so you can distribute that binary to people without caring as much about what version of the OS they have; but that would be unnecessary if you just gave them the source code and have them compile their own copy for their system targeting the exact libraries they have.
  - codexon 2y ago
    Unfortunately most people don't want to bother compiling, myself included. I tried gentoo one time and it took 1 hour to compile 5 minutes worth of apt-get on ubuntu.
- fweimer 2y ago
  Only the dynamically linked bits, the statically linked startup code and libc_nonshared.a are missing from newer versions. Most programs don't need them (who needs working ELF constructors in the main program?). The libc_nonshared.a bits can be reimplemented from scratch easily enough (but we should switch them over to header-only implementations eventually).
- IshKebab 2y ago
  Yes exactly. That's what the flag would do. But it doesn't exist.
fweimer 2y ago
I used to think that binary compatibility benefits proprietary applications, but I'm not so sure anymore. From a commercial perspective, when we break binary compatibility (not that we want to), it's an opportunity for selling more stuff. Many distributions do periodic mass rebuilds anyway and do not need that much long-term ABI compatibility. Binary compatibility seems mostly for people who compile their own software, but have not automated that and therefore couldn't keep up with updates if there wasn't ABI compatibility.
- IshKebab 2y ago
  I agree. It's annoying for closed source apps but they generally have the resources to deal with it anyway. E.g. with Questa I can just unzip it and run it. No trouble. It's disproportionately annoying for open source projects who don't want to waste their time dealing with this.

forrestthewoods 2y ago

> They are a problem because gcc automatically links to the latest version of glibc. As to why they don't add an option to specify an older version? Because glibc and ld/lld are badly designed. glibc is stuck in the 80s with awful and unnecessary automagic configure steps. ld/lld expect a full and complete shared library to exist when compiling even though it expects a different shared library to exist in the future. Zig solves the glibc linking issue. You can trivially target any old version for any supported target platform. The only thing you actually need are headers and a thin, implementation free lib that contains stub functions. Unfortunately glibc is not architected to make this trivial. But this is just because glibc is stuck with decades of historic cruft, not because it's actually a hard problem.

einpoklum 2y ago
> awful and unnecessary automagic configure steps Steps taken when? When building glibc? And - what steps? > ld/lld expect a full and complete shared library to exist when compiling ... But ld and lld are linkers... > Zig solves the glibc linking issue. But Zig is a language. Do you mean the Zig standard library? The Zig compiler? > The only thing you actually need are headers and a thin, implementation free lib that contains stub functions. Why do you need stub functions at all, if you're not actually using them?
- deleted 2y ago
  [deleted]
- ploxiln 2y ago
  The zig compiler can compile C and C++, using llvm, and it also packages various libc implementations, including glibc, musl, mingw, and msvc, and more for other OSes. Some people use it as a more convenient golang-like cross-compiler. And this whole combination is a smaller download and install than most other toolchains out there. It just took some inspiration and grunt work, to dissect the necessary (processed) headers and other bits of each libc ... hats of to the zig devs. Random blog post I just found about it: https://ruoyusun.com/2022/02/27/zig-cc.html https://ruoyusun.com/2022/02/27/zig-cc.html

bregma 2y ago

Hmm. The MSVCRT.DLL/MSVCRTD.DLL not being binary compatible between releases of Visual Studio is the same thing, except of course you can't even combine some modules compiled for debug with modules compiled without debug in the same executable. The Windows problem has always been so so much worse that pretty much all developers simple resorted to shipping the OS system runtime with every package and it's just expected nowadays. It's where the phrase "DLL hell" originated, after all. Not to say the ABI problem isn't real if you want to combine binary packages from different Linux-based OSes. Plenty of solutions for that have cropped up as well: containers, flatpacks, snaps, the list goes on.

forrestthewoods 2y ago
> The Windows problem has always been so so much worse Hard, hard disagree. The problems are somewhat comparable. But if any platform is more painful it's Linux. Although they're similar if you exclude glibc pain. At least in my personal experience of writing lots of code that needs to run on win/mac/linux/android. > pretty much all developers simple resorted to shipping the OS system runtime with every package Meanwhile Linux developers have resorted to shipping an entire OS via docker to run every program. Because managing Linux environment dependencies is so painful you have to package the whole system. Needing to docker to simply launch a program is so embarrassing. > except of course you can't even combine some modules compiled for debug with modules compiled without debug in the same executable That's not any different on Linux. That has more to do with C++.
- graemep 2y ago
  > Meanwhile Linux developers have resorted to shipping an entire OS via docker to run every program. > Needing to docker to simply launch a program is so embarrassing. I have never needed docker "just to launch a program". Docker makes it easy to provide multiple containerised copies of an identical environment. Containers are a light alternative to VM images. I assume you find the existence of Windows containers just as embarrassing? https://learn.microsoft.com/en-us/virtualization/windowscontainers/about/ https://learn.microsoft.com/en-us/virtualization/windowscont...

Sesse__ 2y ago

Linking to the latest version of glibc is, in itself, not a problem -- glibc hasn't bumped its soname in ages, it is using symbol versioning instead. So you only get a problem if you use a symbol that doesn't exist in older glibc (i.e., some specific interface that you are using changed). As for using an older version of glibc, _linking_ isn't the problem -- swapping out the header files would be. You can probably install an old version of the header files somewhere else and just -I that directory, but I've never tried. libstdc++ would probably be harder, if you're in C++ land.

fweimer 2y ago
Recent libstdc++ has a _dl_find_object@GLIBC_2.35 dependency, so it's not exactly trivial anymore to link a C++ program against a older, side-installed glibc version because it won't have that symbol. It's possible to work around that (link against a stub that has _dl_find_object@GLIBC_2.35 as a compat symbol, so that libstdc++ isn't rejected), but linking statically is much more difficult because libstc++.a (actually, libgcc_eh.a) does not have the old code anymore that _dl_find_object replaces (once GCC is built against a glibc version that has _dl_find_object). This applies to other libraries as well because there are new(ish) math functions, strlcpy, posix_spawn extensions etc. that seem to be quite widely used already.
- ynik 2y ago
  I find it's best to treat this as a case of "cross-compilation to old glibc version". That is, you don't want to just link against an old glibc version, you want a full cross-compilation toolchain where the compiler does not pick up headers from `/usr/include` by default (but instead has its own copy of the system headers), and where libstdc++ is also linked against that old glibc version. We used https://crosstool-ng.github.io/ https://crosstool-ng.github.io/ to create such a "cross-compiler". Now it doesn't matter which distribution our developers use, the same source code will always turn into the same binary (reproducible builds, yay!) and those binaries will work on older distributions than the developers are using. This allows us to ship dynamically linked linux executables; our linux customers can just unzip + run, same as our windows customers, no need to mess with docker containers. The downside is that we can't just use a library by `apt install`ing it, everything needs to be built with the cross compilation toolchain.

zX41ZdbW 2y ago

I've made a library named "glibc-compatibility": https://github.com/ClickHouse/ClickHouse/tree/master/base/glibc-compatibility https://github.com/ClickHouse/ClickHouse/tree/master/base/gl... When linking with this library before glibc, the resulting binary will not depend on the new symbol versions. It will run on glibc 2.4 and on systems as old as Ubuntu 8.04 and CentOS 5 even when built on the most modern system.

thrtythreeforty 2y ago
How does this work?
- threecheese 2y ago
  In case parent didn’t see your q: https://www.lightofdawn.org/wiki/wiki.cgi/NewAppsOnOldGlibc https://www.lightofdawn.org/wiki/wiki.cgi/NewAppsOnOldGlibc
adastra22 2y ago
You are a hero.
cataphract 2y ago
An easier option is to build against musl, remove the DEPENDS with patchelf, and use that. There are a few incompatibilities with glibc (more on aarch64 than am64), but it's manageable (you might need to patch a few headers in musl and provide missing/incompatible functions). And you'll have a binary that works against musl and glibc both.
kzrdude 2y ago
Here is another interesting tool, this one for patching executables https://github.com/corsix/polyfill-glibc https://github.com/corsix/polyfill-glibc
- superkuh 2y ago
  Thank you! This has solved multiple problems I've had for years.

LtWorf 2y ago

chroot has existed for many years.

sunshowers 2y ago

If you're building Rust, check out cargo zigbuild (yes, zigbuild) which lets you target old glibc.