10 ms·

Doesn't linking everything statically imply that the base image -- and memory, at runtime -- will be bloated by many copies of libc and other common libraries?

Doesn't linking everything statically imply that the base image -- and memory, at runtime -- will be bloated by many copies of libc and other common libraries? I do like the simplicity of static linking but it sort of seems to go against the idea of avoiding "bloat".

zshrc 3y ago
musl is significantly smaller and "less bloat" than glibc, so even with a statically linked program, it still remains small in both system memory and storage.
- skywal_l 3y ago
  And using LTO[0] can also help. [0]https://gcc.gnu.org/wiki/LinkTimeOptimization https://gcc.gnu.org/wiki/LinkTimeOptimization
Shorel 3y ago
In a world where Docker and Kubernetes exist, where whole copies of operating systems are added to each running service... This seems a weird thing to complain about =)
- kentonv 3y ago
  I mean, if you ran every single executable on your desktop in a separate container I think you'd see problems. There are a pretty large number of programs running on most desktops, plus all the programs that get called by shell scripts, etc. Running a handful of containers representing major applications is more reasonable and the memory wastage may be worth it to avoid dependency conflicts.
  - drakenot 3y ago
    You've just described Qubes OS!
    - palata 3y ago
      Except that QubesOS uses VMs for their security benefits, which are greater than those of containers. Containers make a lot of sense to me on servers ("deploy a controlled environment"), but often on Desktop I feel like they are used as a solution to "I don't know how to handle dependencies" or "My dependencies are so unstable that it is impossible to install them system-wide", both of which should be solved by making slightly better software.
  - Gabrys1 3y ago
    Each electron app is like that
- lnxg33k1 3y ago
  Yeah but there I can still update vulnerable libraries independently, to be a statically linked system just means that if there is a bug in libpng then I have to recompile everything?
  - bzzzt 3y ago
    Yes, although it very much depends on how big 'everything' is if that's a problem.
  - greyw 3y ago
    In most cases relinking is enough.
  - Shorel 3y ago
    I was under the impression only Gentoo users recompile everything. In a statically linked system, your dependency manager will update more packages. And if your program is written in C/C++/Go/Rust, then yes, it will be recompiled.
    - lnxg33k1 3y ago
      I use Gentoo, so I am not against rebuild everything, but afaik unless you have static-libs USE flag for something, it's dynamically linked so relinking on rebuilding the dependency is enough, with static-libs the dependent package is also rebuilt
  - colonwqbang 3y ago
    Not recompile I guess, but you need to relink everything. Oasis seems to have a good way of doing that, with the whole system being built in a single tree by an efficient build tool (my recollection from last time it was posted). A dynamic executable needs to relink every time it's run, which also takes time.
  - nordsieck 3y ago
    > if there is a bug in libpng then I have to recompile everything? You say that as if it's such a burden. But it's really not. I'm somewhat sympathetic to the space argument, but a package manager/docker registry means that updating software is very easy. And it happens all the time for other reasons today anyhow.
- palata 3y ago
  > This seems a weird thing to complain about =) On the contrary, I find it relevant: I think that the modern way is wasting way, way too much.
  - Shorel 3y ago
    On that respect, we agree.
liampulles 3y ago
It would be bloated, but how big of a problem is that these days? A TB of storage is pretty cheap.
- mayli 3y ago
  Imagine base windows install requires 1TB of storage.
- cmovq 3y ago
  A TB of memory is not
thanatos519 3y ago
KSM could help with that: https://docs.kernel.org/admin-guide/mm/ksm.html https://docs.kernel.org/admin-guide/mm/ksm.html ... oh wait, the apps have to hint that it's possible. Nebbermind.
jezze 3y ago
A linker typically only includes the parts of the library it needs for each binary so some parts will definately have many copies of the same code when you statically link but it will not make complete copies. But I wouldnt consider this bloat. To me it is just a better seperation of concerns. To me bloat would be to have a system that has to keep track of all library dependencies instead, both from a packaging perspective but also in runtime. I think it depends where you are coming from. To me static linking is just cleaner. I dont care much for the extra memory it might use.
- jvanderbot 3y ago
  Dynamic linking served us when OS upgrades came infrequently, user software was almost never upgraded short of mailing out new disks, and vendors had long lead times to incorporate security fixes. In the days of fast networks, embedded OSs, emphemeral containers, and big hard drives, a portable static binary is way less complex and only somewhat less secure (unless you're regularly rebuilding your containers/execs in which case it's break even security wise or possibly more secure, simply because each exec may not include vulnerable code)
  - teaearlgraycold 3y ago
    Yeah I’d prefer we just use another gigabyte of storage than add so much complexity. Even with what is a modest SSD capacity today I have a hard time imagining how I’d fill my storage. I’m reminded of my old workstation from 8 years ago. It had a 500GB hard drive and a 32GB SSD for caching. I immediately reconfigured to just use the SSD for everything by default. It ended up being plenty.
  - gnramires 3y ago
    As far as I can see, it would be unwise to roll back 30 years of (Linux) systems building with dynamic linking in favor of static linking. It mostly works very well and does save some memory, disk, and has nice security properties. Both have significant pros and cons. I've been thinking (not a Linux expert by any means) the ideal solution would be to have better dependency management: I think a solution could be if say binaries themselves carried dependency information. That way you get the benefits of dynamic and static linking by just distributing binaries with embedded library requirements. Also, I think there should be a change of culture in library development to clearly mark compatibility breaks (I think something like semantic versioning works like that?). That way, your software could support any newer version up to a compatibility break -- which should be extremely rare. And if you must break compatibility there should be an effort to keep old versions available, secure and bug free (or at least the old versions should be flagged as insecure in some widely accessible database). Moreover, executing old/historical software should become significantly easier if library information was kept in the executable itself (you'd just have to find the old libraries, which could be kept available in repositories). I think something like that could finally enable portable Linux software? (Flatpak and AppImage notwithstanding)
bzzzt 3y ago
I know lots of compilers/linkers don't optimize for it but it should be possible to 'tree shake' libraries so only the parts that are used by an application are included. That would shake off a lot of the 'bloat'.
- volemo 3y ago
  Wait, it's not being done?
  - dieortin 3y ago
    It is, major compilers do that by default
  - Fronzie 3y ago
    As far as I know, even with LTO, it requires -ffunction-sections -fdata-sections in order to strip out unused functions.
Gabrys1 3y ago
I guess each of the copies of libc can be optimized away and only the functions the specific binary calls will be left (and the compiler should be allowed to optimize past the library boundary), so maybe this balances the issues a bit. Not that I really know anything about it, ask jart
Gazoche 3y ago
I'll take bloat over dependency hell every day of the week. Feels like every single app is a bundled web browser these days anyways.
- palata 3y ago
  > dependency hell Dependency hell comes from bad dependencies that don't do semver properly. Choose your deps carefully, and that's perfectly fine. > Feels like every single app is a bundled web browser these days anyways. Yep, that's apparently the best way to use the bad libraries people want to use and not give a damn about semver.
  - hn_go_brrrrr 3y ago
    Semver is nearly impossible to do "properly" because of https://xkcd.com/1172 https://xkcd.com/1172. With a sufficient number of users, all bug fixes are breaking changes. If the behavior can possibly be observed in any way, some user will be depending on it, deliberately or otherwise.
    - Ar-Curunir 3y ago
      Semver defines what is breaking and not-breaking. E.g., Rust semver says that "code should continue compiling with a minor version bump, but not necessarily for a major version bump"
      - steveklabnik 3y ago
        Yes. The very first line of the spec: > Software using Semantic Versioning MUST declare a public API. This API could be declared in the code itself or exist strictly in documentation. However it is done, it SHOULD be precise and comprehensive. If it's not in the API, it is not bound by the rules. Many ecosystems come up with various norms, like Rust has, to help guide people in this. But it's almost certainly not a semver violation to make the change described in the XKCD because "handle unknown unknowns" is not possible. That doesn't mean that we should throw out the entire idea of software assisted upgrades to dependencies.
jacquesm 3y ago
Not necessarily. Bloat is one reason why originally dynamic linking was rolled out but the bigger benefit (to manufacturers) was to be able to update libraries without updating the applications. This has been the source of much trouble (dependency hell) and statically linked binaries suffer none of these issues. It's not like every application uses all of every library and an efficient linker is able to see which parts of the library it needs to link and which parts it can safely leave out.
arghwhat 3y ago
Static linked binaries are a generally lot smaller than a dynamically linked library and its dependencies, especially with link-time optimizations and inlining. You wouldn't want have 100 tools statically link the entirety of chromium, but for normal C library sizes you don't get bloat. The preference for dynamic libraries in Linux distros is just so they can roll out patch updates in one place instead of rebuilding dependents.
- marwis 3y ago
  But dynamically linked library only needs to be loaded to RAM once whereas with static linking you'd be loading the same code many times (unless you compile everything to single binary like BusyBox). This also gets you better cache utilization. Also I think inlining would typically increase the total size of output rather than decrease it.
  - jhallenworld 3y ago
    So you only need to load duplicated code for each different statically linked program. If there are many processes running the same program, they will all share the same physical pages for the code. So for example, having 100s of "bash" instances running does not use that much memory. You can see this by running "pmap <pid> -XX" (the output is very wide- probably load it into an editor). Look at the shared vs. private pages. Also: There is another way to automatically share pages between different programs: de-duplication. This would require common libraries to be statically linked on page boundaries. The OS would quickly de-duplicate during loading by hashing the pages. VMs use this technique to increase effective memory when there are many guest OS running.
    - marwis 3y ago
      Yes but most processes are unique. The only bash process I have running is my interactive shell.
  - arghwhat 3y ago
    Static linking gives you better instruction cache utilization as you are executing local code linearly rather than going through indirection with more boilerplate. This indirection costs a few cycles too. Inlining external code reduces the size not only by saving the call, PLT and and stack dance, but also through specialization (removal of unused conditional, pruning of no longer referenced symbols) as the code is locally optimized. This further reduction in size further improves cache behavior and performance. Duplication can be an issue (not necessarily for performance, but for total binary size), but compilers have heuristics for that. Even just having the symbol local saves some space and call overhead though (no PLT). The case for the shared library having better caching implies multiple processes that are distinct executables (otherwise they share program memory regardless of linkage) trying to hammer it at once, sharing the continued caching, but such scenario is hurt by the call overhead and lower optimization opportunities, negating the result.
javierhonduco 3y ago
Once it’s loaded in memory, if Kernel Samepage Merging is enabled it might not be as bad, but would love to hear of somebody has any thoughts https://docs.kernel.org/admin-guide/mm/ksm.html https://docs.kernel.org/admin-guide/mm/ksm.html
- LegionMammal978 3y ago
  From the link: > KSM only merges anonymous (private) pages, never pagecache (file) pages. So it wouldn't be able to help with static libraries loaded from different executables. (At any rate, they'd have to be at the same alignment within the page, which is unlikely without some special linker configuration.)
  - javierhonduco 3y ago
    Had completely missed that line — great point!
1vuio0pswjnm7 3y ago
I have seen this sort of statement on HN before. I am guessing that the persons who propagate this idea have never actually experimented with replacing dynamically-linked programs having numerous dependencies with statically-compiled ones. It's a theory that makes sense in the abstract, but they have not actually tested it. Though it is not a goal of mine to save storage space by using static binaries, and I actually expect to lose space as a tradeoff, I have actually saved storage space in some cases by using static binaries. This comes from being able to remove libraries from /usr/lib. TBH, I am not exactly sure why this is the case. Perhaps in part because one might be storing large libraries containing significant numbers of functions that one's programs never use. For me using static binaries works well. Even "common" libraries can be removed in some cases by using a multi-call/crunched binary like busybox. This might not work for everyone. I think much depends on what selection of programs the computer owner prefers. (Namely, the dependencies required by those programs.)

zshrc 3y ago

musl is significantly smaller and "less bloat" than glibc, so even with a statically linked program, it still remains small in both system memory and storage.

skywal_l 3y ago
And using LTO[0] can also help. [0]https://gcc.gnu.org/wiki/LinkTimeOptimization https://gcc.gnu.org/wiki/LinkTimeOptimization

Shorel 3y ago

In a world where Docker and Kubernetes exist, where whole copies of operating systems are added to each running service... This seems a weird thing to complain about =)

kentonv 3y ago
I mean, if you ran every single executable on your desktop in a separate container I think you'd see problems. There are a pretty large number of programs running on most desktops, plus all the programs that get called by shell scripts, etc. Running a handful of containers representing major applications is more reasonable and the memory wastage may be worth it to avoid dependency conflicts.
- drakenot 3y ago
  You've just described Qubes OS!
  - palata 3y ago
    Except that QubesOS uses VMs for their security benefits, which are greater than those of containers. Containers make a lot of sense to me on servers ("deploy a controlled environment"), but often on Desktop I feel like they are used as a solution to "I don't know how to handle dependencies" or "My dependencies are so unstable that it is impossible to install them system-wide", both of which should be solved by making slightly better software.
- Gabrys1 3y ago
  Each electron app is like that
lnxg33k1 3y ago
Yeah but there I can still update vulnerable libraries independently, to be a statically linked system just means that if there is a bug in libpng then I have to recompile everything?
- bzzzt 3y ago
  Yes, although it very much depends on how big 'everything' is if that's a problem.
- greyw 3y ago
  In most cases relinking is enough.
- Shorel 3y ago
  I was under the impression only Gentoo users recompile everything. In a statically linked system, your dependency manager will update more packages. And if your program is written in C/C++/Go/Rust, then yes, it will be recompiled.
  - lnxg33k1 3y ago
    I use Gentoo, so I am not against rebuild everything, but afaik unless you have static-libs USE flag for something, it's dynamically linked so relinking on rebuilding the dependency is enough, with static-libs the dependent package is also rebuilt
- colonwqbang 3y ago
  Not recompile I guess, but you need to relink everything. Oasis seems to have a good way of doing that, with the whole system being built in a single tree by an efficient build tool (my recollection from last time it was posted). A dynamic executable needs to relink every time it's run, which also takes time.
- nordsieck 3y ago
  > if there is a bug in libpng then I have to recompile everything? You say that as if it's such a burden. But it's really not. I'm somewhat sympathetic to the space argument, but a package manager/docker registry means that updating software is very easy. And it happens all the time for other reasons today anyhow.
palata 3y ago
> This seems a weird thing to complain about =) On the contrary, I find it relevant: I think that the modern way is wasting way, way too much.
- Shorel 3y ago
  On that respect, we agree.

liampulles 3y ago

It would be bloated, but how big of a problem is that these days? A TB of storage is pretty cheap.

mayli 3y ago
Imagine base windows install requires 1TB of storage.
cmovq 3y ago
A TB of memory is not

thanatos519 3y ago

KSM could help with that: https://docs.kernel.org/admin-guide/mm/ksm.html https://docs.kernel.org/admin-guide/mm/ksm.html ... oh wait, the apps have to hint that it's possible. Nebbermind.

jezze 3y ago

A linker typically only includes the parts of the library it needs for each binary so some parts will definately have many copies of the same code when you statically link but it will not make complete copies. But I wouldnt consider this bloat. To me it is just a better seperation of concerns. To me bloat would be to have a system that has to keep track of all library dependencies instead, both from a packaging perspective but also in runtime. I think it depends where you are coming from. To me static linking is just cleaner. I dont care much for the extra memory it might use.

jvanderbot 3y ago
Dynamic linking served us when OS upgrades came infrequently, user software was almost never upgraded short of mailing out new disks, and vendors had long lead times to incorporate security fixes. In the days of fast networks, embedded OSs, emphemeral containers, and big hard drives, a portable static binary is way less complex and only somewhat less secure (unless you're regularly rebuilding your containers/execs in which case it's break even security wise or possibly more secure, simply because each exec may not include vulnerable code)
- teaearlgraycold 3y ago
  Yeah I’d prefer we just use another gigabyte of storage than add so much complexity. Even with what is a modest SSD capacity today I have a hard time imagining how I’d fill my storage. I’m reminded of my old workstation from 8 years ago. It had a 500GB hard drive and a 32GB SSD for caching. I immediately reconfigured to just use the SSD for everything by default. It ended up being plenty.
- gnramires 3y ago
  As far as I can see, it would be unwise to roll back 30 years of (Linux) systems building with dynamic linking in favor of static linking. It mostly works very well and does save some memory, disk, and has nice security properties. Both have significant pros and cons. I've been thinking (not a Linux expert by any means) the ideal solution would be to have better dependency management: I think a solution could be if say binaries themselves carried dependency information. That way you get the benefits of dynamic and static linking by just distributing binaries with embedded library requirements. Also, I think there should be a change of culture in library development to clearly mark compatibility breaks (I think something like semantic versioning works like that?). That way, your software could support any newer version up to a compatibility break -- which should be extremely rare. And if you must break compatibility there should be an effort to keep old versions available, secure and bug free (or at least the old versions should be flagged as insecure in some widely accessible database). Moreover, executing old/historical software should become significantly easier if library information was kept in the executable itself (you'd just have to find the old libraries, which could be kept available in repositories). I think something like that could finally enable portable Linux software? (Flatpak and AppImage notwithstanding)

bzzzt 3y ago

I know lots of compilers/linkers don't optimize for it but it should be possible to 'tree shake' libraries so only the parts that are used by an application are included. That would shake off a lot of the 'bloat'.

volemo 3y ago
Wait, it's not being done?
- dieortin 3y ago
  It is, major compilers do that by default
- Fronzie 3y ago
  As far as I know, even with LTO, it requires -ffunction-sections -fdata-sections in order to strip out unused functions.

Gabrys1 3y ago

I guess each of the copies of libc can be optimized away and only the functions the specific binary calls will be left (and the compiler should be allowed to optimize past the library boundary), so maybe this balances the issues a bit. Not that I really know anything about it, ask jart

Gazoche 3y ago

I'll take bloat over dependency hell every day of the week. Feels like every single app is a bundled web browser these days anyways.

palata 3y ago
> dependency hell Dependency hell comes from bad dependencies that don't do semver properly. Choose your deps carefully, and that's perfectly fine. > Feels like every single app is a bundled web browser these days anyways. Yep, that's apparently the best way to use the bad libraries people want to use and not give a damn about semver.
- hn_go_brrrrr 3y ago
  Semver is nearly impossible to do "properly" because of https://xkcd.com/1172 https://xkcd.com/1172. With a sufficient number of users, all bug fixes are breaking changes. If the behavior can possibly be observed in any way, some user will be depending on it, deliberately or otherwise.
  - Ar-Curunir 3y ago
    Semver defines what is breaking and not-breaking. E.g., Rust semver says that "code should continue compiling with a minor version bump, but not necessarily for a major version bump"
    - steveklabnik 3y ago
      Yes. The very first line of the spec: > Software using Semantic Versioning MUST declare a public API. This API could be declared in the code itself or exist strictly in documentation. However it is done, it SHOULD be precise and comprehensive. If it's not in the API, it is not bound by the rules. Many ecosystems come up with various norms, like Rust has, to help guide people in this. But it's almost certainly not a semver violation to make the change described in the XKCD because "handle unknown unknowns" is not possible. That doesn't mean that we should throw out the entire idea of software assisted upgrades to dependencies.

jacquesm 3y ago

Not necessarily. Bloat is one reason why originally dynamic linking was rolled out but the bigger benefit (to manufacturers) was to be able to update libraries without updating the applications. This has been the source of much trouble (dependency hell) and statically linked binaries suffer none of these issues. It's not like every application uses all of every library and an efficient linker is able to see which parts of the library it needs to link and which parts it can safely leave out.

arghwhat 3y ago

Static linked binaries are a generally lot smaller than a dynamically linked library and its dependencies, especially with link-time optimizations and inlining. You wouldn't want have 100 tools statically link the entirety of chromium, but for normal C library sizes you don't get bloat. The preference for dynamic libraries in Linux distros is just so they can roll out patch updates in one place instead of rebuilding dependents.

marwis 3y ago
But dynamically linked library only needs to be loaded to RAM once whereas with static linking you'd be loading the same code many times (unless you compile everything to single binary like BusyBox). This also gets you better cache utilization. Also I think inlining would typically increase the total size of output rather than decrease it.
- jhallenworld 3y ago
  So you only need to load duplicated code for each different statically linked program. If there are many processes running the same program, they will all share the same physical pages for the code. So for example, having 100s of "bash" instances running does not use that much memory. You can see this by running "pmap <pid> -XX" (the output is very wide- probably load it into an editor). Look at the shared vs. private pages. Also: There is another way to automatically share pages between different programs: de-duplication. This would require common libraries to be statically linked on page boundaries. The OS would quickly de-duplicate during loading by hashing the pages. VMs use this technique to increase effective memory when there are many guest OS running.
  - marwis 3y ago
    Yes but most processes are unique. The only bash process I have running is my interactive shell.
- arghwhat 3y ago
  Static linking gives you better instruction cache utilization as you are executing local code linearly rather than going through indirection with more boilerplate. This indirection costs a few cycles too. Inlining external code reduces the size not only by saving the call, PLT and and stack dance, but also through specialization (removal of unused conditional, pruning of no longer referenced symbols) as the code is locally optimized. This further reduction in size further improves cache behavior and performance. Duplication can be an issue (not necessarily for performance, but for total binary size), but compilers have heuristics for that. Even just having the symbol local saves some space and call overhead though (no PLT). The case for the shared library having better caching implies multiple processes that are distinct executables (otherwise they share program memory regardless of linkage) trying to hammer it at once, sharing the continued caching, but such scenario is hurt by the call overhead and lower optimization opportunities, negating the result.

javierhonduco 3y ago

Once it’s loaded in memory, if Kernel Samepage Merging is enabled it might not be as bad, but would love to hear of somebody has any thoughts https://docs.kernel.org/admin-guide/mm/ksm.html https://docs.kernel.org/admin-guide/mm/ksm.html

LegionMammal978 3y ago
From the link: > KSM only merges anonymous (private) pages, never pagecache (file) pages. So it wouldn't be able to help with static libraries loaded from different executables. (At any rate, they'd have to be at the same alignment within the page, which is unlikely without some special linker configuration.)
- javierhonduco 3y ago
  Had completely missed that line — great point!

1vuio0pswjnm7 3y ago

I have seen this sort of statement on HN before. I am guessing that the persons who propagate this idea have never actually experimented with replacing dynamically-linked programs having numerous dependencies with statically-compiled ones. It's a theory that makes sense in the abstract, but they have not actually tested it. Though it is not a goal of mine to save storage space by using static binaries, and I actually expect to lose space as a tradeoff, I have actually saved storage space in some cases by using static binaries. This comes from being able to remove libraries from /usr/lib. TBH, I am not exactly sure why this is the case. Perhaps in part because one might be storing large libraries containing significant numbers of functions that one's programs never use. For me using static binaries works well. Even "common" libraries can be removed in some cases by using a multi-call/crunched binary like busybox. This might not work for everyone. I think much depends on what selection of programs the computer owner prefers. (Namely, the dependencies required by those programs.)