5 ms·

As far as I can see, it would be unwise to roll back 30 years of (Linux) systems building with dynamic linking in favor of static linking. It mostly works very

As far as I can see, it would be unwise to roll back 30 years of (Linux) systems building with dynamic linking in favor of static linking. It mostly works very well and does save some memory, disk, and has nice security properties. Both have significant pros and cons. I've been thinking (not a Linux expert by any means) the ideal solution would be to have better dependency management: I think a solution could be if say binaries themselves carried dependency information. That way you get the benefits of dynamic and static linking by just distributing binaries with embedded library requirements. Also, I think there should be a change of culture in library development to clearly mark compatibility breaks (I think something like semantic versioning works like that?). That way, your software could support any newer version up to a compatibility break -- which should be extremely rare. And if you must break compatibility there should be an effort to keep old versions available, secure and bug free (or at least the old versions should be flagged as insecure in some widely accessible database). Moreover, executing old/historical software should become significantly easier if library information was kept in the executable itself (you'd just have to find the old libraries, which could be kept available in repositories). I think something like that could finally enable portable Linux software? (Flatpak and AppImage notwithstanding)

josephg 3y ago
Yes, if someone actually did dependency management in Linux properly then I agree - dynamic linking would be fine. It works pretty well in Nixos as I understand it. But it’s called dependency hell for a reason. And the reason is almost no operating systems handle C dependencies well. There’s always weird, complex, distribution specific systems involving 18 different versions of every library. Do you want llvm18 or llvm18-dev or llvm-18-full-dev or something else entirely? Oh, you’re on gentoo? Better enable some USE flags. Redhat? It’s different again. If Linux dependency management worked well, there would be no need or appetite for docker. But it works badly. So people just use docker and flatpak and whatnot instead, while my hard drive gently weeps. I don’t know about you, but I’m happy to declare bankruptcy on this project. I’d take a 2mb statically linked binary over a 300mb Linux docker image any day of the week.
- StillBored 3y ago
  This isn't really an "operating system" problem. Particularly in the open-source world, there are a number of fairly core libraries that refuse to provide any kind of API compatibility. Then, when there are a couple dozen applications/etc that depend on that library, it's almost an impossible problem because each of those applications then needs to be updated in lockstep with the library version. There is nothing "clean" about how to handle this situation short of having loads of distro maintainers showing up in the upstream packages to fix them to support newer versions of the library. Of course, then all the distro's need to agree on what those versions are going to be... Hence containers, which don't fix the problem at all. Instead they just move the responsibility away from the distro, which should never really have been packaging applications to begin with.
  - palata 3y ago
    > away from the distro, which should never really have been packaging applications to begin with. I disagree here: the whole point of a "software distribution" is to "distribute" software. And it does so by packaging it. There is a ton of benefit in having distro/package maintainers, and we tend to forget it.
- palata 3y ago
  > If Linux dependency management worked well, there would be no need or appetite for docker. I kindly disagree here. Linux dependency management does work well. The problem is the bad libraries that don't do semver properly, and the users who still decide to use bad libraries. If people stopped using libraries that break ABI compatibility, then the authors of those libraries would have to do it properly, and it would work. The reason it doesn't work is really just malpractice.
  - zaphar 3y ago
    If Linux dependency management works well in theory but not in practice then it doesn't work. It works in nix because it can literally use multiple minor versions of a library when it needs to with no problem. Most distro's can't or won't do that. You can call it malpractice but it's not going to stop so in practice you need a way to deal with it.
kentonv 3y ago
Everything you describe already exists. Executables do list their dependencies, and we have well-defined conventions for indicating ABI breaks. It is entirely normal to have multiple major versions of a library installed for ABI compatibility reasons, and it is also entirely normal to expect that you can upgrade the dependencies out from under a binary as long as the library hasn't had an ABI break. The bigger dependency management problem is that every distro has their own package manager and package repository and it's tough for one application developer to build and test every kind of package. But if they just ship a binary, then it's up to the poor user to figure out what packages to install. Often the library you need may not even be available on some distros or the version may be too old.
- rwmj 3y ago
  That's why distros ask you to provide just the sources and we'll do the packaging work for you. The upstream developers shouldn't need to provide packages for every distro. (Of course you can help us downstream packagers by not having insane build requirements, using semantic versioning, not breaking stuff randomly etc).
  - kentonv 3y ago
    This is only realistic for established applications with large userbases. For new or very niche apps, distros are understandably not going to be very interested in doing this work. In that case the developer needs to find a way to distribute the app that they can reasonably maintain directly, and that's where containers or statically-linked binaries are really convenient.
    - rwmj 3y ago
      This isn't really true, Fedora, Debian and Arch have huge numbers of packages, many very niche. You might well need to make the distro aware that the new program exists, but there are established routes for doing that.
      - TimeBearingDown 3y ago
arghwhat 3y ago
Linus Torvalds disagrees: https://lore.kernel.org/lkml/CAHk-=whs8QZf3YnifdLv57+FhBi5_WeNTG1B-suOES=RcUSmQg@mail.gmail.com/ https://lore.kernel.org/lkml/CAHk-=whs8QZf3YnifdLv57+FhBi5_W...
- gnramires 3y ago
  I should have been more balanced or nuanced a bit: I also don't think static linking is to be forbidden or completely shunned. As Linus himself says, a combination of both may be ideal. For basic system libraries like GUI libraries the current approach works well. But you should be free to static link if you want, and if there are serious issues if you don't. Maybe dynamic linking should be focused on a smaller number of well curated libraries and the rest should be left to static. Library archeology seems like a potential serious problem years from now. I still think better listing dependencies (perhaps with the option to pin an exact version?) would be helpful, as well as better usage of something like semver. Someone mentioned binaries include paths to dependencies, but as far as I know, there is no tool to automatically try to resolve those dependencies or standard interface, maybe some more tooling in this area would help. Another nice point about how it current works is that I think it relieves work from programmers. The policy of "Don't worry about distribution (just tell us it exists)" from distros seems like one less headache for the creator (and you can provide static linked binaries too if you want). As most things in life, the ideal is somewhere in the middle...

josephg 3y ago

Yes, if someone actually did dependency management in Linux properly then I agree - dynamic linking would be fine. It works pretty well in Nixos as I understand it. But it’s called dependency hell for a reason. And the reason is almost no operating systems handle C dependencies well. There’s always weird, complex, distribution specific systems involving 18 different versions of every library. Do you want llvm18 or llvm18-dev or llvm-18-full-dev or something else entirely? Oh, you’re on gentoo? Better enable some USE flags. Redhat? It’s different again. If Linux dependency management worked well, there would be no need or appetite for docker. But it works badly. So people just use docker and flatpak and whatnot instead, while my hard drive gently weeps. I don’t know about you, but I’m happy to declare bankruptcy on this project. I’d take a 2mb statically linked binary over a 300mb Linux docker image any day of the week.

StillBored 3y ago
This isn't really an "operating system" problem. Particularly in the open-source world, there are a number of fairly core libraries that refuse to provide any kind of API compatibility. Then, when there are a couple dozen applications/etc that depend on that library, it's almost an impossible problem because each of those applications then needs to be updated in lockstep with the library version. There is nothing "clean" about how to handle this situation short of having loads of distro maintainers showing up in the upstream packages to fix them to support newer versions of the library. Of course, then all the distro's need to agree on what those versions are going to be... Hence containers, which don't fix the problem at all. Instead they just move the responsibility away from the distro, which should never really have been packaging applications to begin with.
- palata 3y ago
  > away from the distro, which should never really have been packaging applications to begin with. I disagree here: the whole point of a "software distribution" is to "distribute" software. And it does so by packaging it. There is a ton of benefit in having distro/package maintainers, and we tend to forget it.
palata 3y ago
> If Linux dependency management worked well, there would be no need or appetite for docker. I kindly disagree here. Linux dependency management does work well. The problem is the bad libraries that don't do semver properly, and the users who still decide to use bad libraries. If people stopped using libraries that break ABI compatibility, then the authors of those libraries would have to do it properly, and it would work. The reason it doesn't work is really just malpractice.
- zaphar 3y ago
  If Linux dependency management works well in theory but not in practice then it doesn't work. It works in nix because it can literally use multiple minor versions of a library when it needs to with no problem. Most distro's can't or won't do that. You can call it malpractice but it's not going to stop so in practice you need a way to deal with it.

kentonv 3y ago

Everything you describe already exists. Executables do list their dependencies, and we have well-defined conventions for indicating ABI breaks. It is entirely normal to have multiple major versions of a library installed for ABI compatibility reasons, and it is also entirely normal to expect that you can upgrade the dependencies out from under a binary as long as the library hasn't had an ABI break. The bigger dependency management problem is that every distro has their own package manager and package repository and it's tough for one application developer to build and test every kind of package. But if they just ship a binary, then it's up to the poor user to figure out what packages to install. Often the library you need may not even be available on some distros or the version may be too old.

rwmj 3y ago
That's why distros ask you to provide just the sources and we'll do the packaging work for you. The upstream developers shouldn't need to provide packages for every distro. (Of course you can help us downstream packagers by not having insane build requirements, using semantic versioning, not breaking stuff randomly etc).
- kentonv 3y ago
  This is only realistic for established applications with large userbases. For new or very niche apps, distros are understandably not going to be very interested in doing this work. In that case the developer needs to find a way to distribute the app that they can reasonably maintain directly, and that's where containers or statically-linked binaries are really convenient.
  - rwmj 3y ago
    This isn't really true, Fedora, Debian and Arch have huge numbers of packages, many very niche. You might well need to make the distro aware that the new program exists, but there are established routes for doing that.
    - TimeBearingDown 3y ago

arghwhat 3y ago

Linus Torvalds disagrees: https://lore.kernel.org/lkml/CAHk-=whs8QZf3YnifdLv57+FhBi5_WeNTG1B-suOES=RcUSmQg@mail.gmail.com/ https://lore.kernel.org/lkml/CAHk-=whs8QZf3YnifdLv57+FhBi5_W...

gnramires 3y ago
I should have been more balanced or nuanced a bit: I also don't think static linking is to be forbidden or completely shunned. As Linus himself says, a combination of both may be ideal. For basic system libraries like GUI libraries the current approach works well. But you should be free to static link if you want, and if there are serious issues if you don't. Maybe dynamic linking should be focused on a smaller number of well curated libraries and the rest should be left to static. Library archeology seems like a potential serious problem years from now. I still think better listing dependencies (perhaps with the option to pin an exact version?) would be helpful, as well as better usage of something like semver. Someone mentioned binaries include paths to dependencies, but as far as I know, there is no tool to automatically try to resolve those dependencies or standard interface, maybe some more tooling in this area would help. Another nice point about how it current works is that I think it relieves work from programmers. The policy of "Don't worry about distribution (just tell us it exists)" from distros seems like one less headache for the creator (and you can provide static linked binaries too if you want). As most things in life, the ideal is somewhere in the middle...