29 ms·

Dynamic linking

408 points by scrollaway 6y ago

hknapp 6y ago
I appreciate the brevity of this article.
wahern 6y ago
Do any Linux/glibc or Linux/musl systems support static PIE binaries, yet? Without static PIE support you don't benefit from ASLR (at least not fully). This 2018 article seems like a good breakdown of the issues: https://www.leviathansecurity.com/blog/aslr-protection-for-statically-linked-executables https://www.leviathansecurity.com/blog/aslr-protection-for-s... OpenBSD has supported static PIE since 2015; not just supported, but all system static binaries (e.g. /bin and /sbin) are built as static PIEs, and I believe PIE is the default behavior when using `cc -static`. The reasons for the switch and the required toolchain changes are summarized in this presentation: https://www.openbsd.org/papers/asiabsdcon2015-pie-slides.pdf https://www.openbsd.org/papers/asiabsdcon2015-pie-slides.pdf Also, simply checking the "static PIE" box isn't the end of the story. There are different ways to accomplish it, and some are better than others in terms of ASLR, W^X, and other exploit mitigations. It's been a couple of years since I last looked into and had a hold on all the issues simultaneously[1], but the basic takeaway is that dynamic linking in system toolchains and system runtimes is far more mature than static linking. [1] static PIE issues are a nexus of exploit mitigation techniques, so if you want to deep dive into exploit mitigation or even just linking issues then chasing the static PIE rabbit is a good approach.
- monocasa 6y ago
  I imagine there's some way to, because the kernel is able to apply ASLR to the dynamic linker itself.
  - wahern 6y ago
    It's not a matter of possible, but whether the work has been done throughout the toolchain and runtime stacks. When OpenBSD implemented static PIE, -static and -pie, the literal options and the code generation aspects more generally, were still mutually exclusive in GCC and clang. GCC required patching to enable both static linking and PIE generation. In fact, my local GCC 9.3 man page still says that -static overrides -pie; to build static PIE seems to require the special option -static-pie, in addition to -fPIE/-fPIC when building the object code. But it's not enough for the compiler and compile time linker to support it. libc and libstdc++/libc++ also need support, both internally as well as in the built libraries. And I think libdl might need support if you want dlopen to work from a static PIE binary. Likewise for libpthread. Supporting static PIE requires the cooperation of many moving parts. And that's just to support it. Making it easy, let alone the default behavior, so that it doesn't require many carefully coordinated, obscure flags without the risk of any misstep silently disabling some exploit mitigation, is yet another story. It can be done. It should be done. But to what extent has it been done? I'm not sure, though some quick Googling suggests GCC and clang are mostly there, at least in terms of nominal support (which, again, is distinct from fully supporting all the same mitigation measures). glibc seems to have gotten some support (e.g. --enable-static-pie in glibc's build), though I'm not sure whether it's available in distros. And I would guess that musl libc support is pretty far along, and presumably more mature than glibc's given that Rich Felker started experimenting with static PIE several years ago, shortly after OpenBSD did their work. See https://www.openwall.com/lists/musl/2015/06/01/12 https://www.openwall.com/lists/musl/2015/06/01/12
    - monocasa 6y ago
      For sure. My comment was directed at Linux the kernel rather than Linux the ecosystem, but it's a totally valid point that for practical purposes that distinction can border on meaningless.
- mforney 6y ago
  Most Linux/musl systems support it. On Alpine, gcc is built with `--enable-default-pie` so all static libraries can be linked into a static PIE. On vanilla gcc (since version 8 when static PIE was upstreamed), `-static` means non-PIE static executable and there is a separate flag for `-static-pie` for static PIE. Alpine patches gcc so that `-static` and `-pie` are independent flags, so both `cc -static` and `cc -static-pie` will produce a static PIE.
- Ericson2314 6y ago
  Nixpkgs / NixOS does.
- saagarjha 6y ago
  I'm not sure I understand the first link you posted: if a binary is statically linked, why does it need a GOT? It's literally calling functions in its own binary…
  - wahern 6y ago
    But where in its own binary? To support ASLR of code (a ROP mitigation), especially fine-grain randomization, function call sites can't use static addressing, either absolute or relative, pointing directly to a function. To support ASLR of code you can either rewrite every function call site on load (kinda similar to DLLs on Windows), or use one or more tables that are updated at runtime (PLTs, GOT, etc). BSD and Linux environments use the PLT/GOT approach because 1) it's a better fit for ELF, 2) shares most of the preexisting instrumentation with dynamic linking, and 3) preserves the ability to share most of the mapped pages of static binaries (imagine if a static /bin/sh was effectively rewritten on every invocation). Trying to accomplish this safely and effectively is what RELRO is about. See https://www.redhat.com/en/blog/hardening-elf-binaries-using-relocation-read-only-relro https://www.redhat.com/en/blog/hardening-elf-binaries-using-... But RELRO poses some dilemmas regarding performance, and depending on your choices as well as the details of the implementation, RELRO might be ineffective or, worse, introduce exploits of its own. See https://www.usenix.org/system/files/conference/usenixsecurity15/sec15-paper-di-frederico.pdf https://www.usenix.org/system/files/conference/usenixsecurit... It's difficult to achieve a design that provides all the desirable exploit mitigations without sacrificing startup latency and other features. OpenBSD added a new syscall, kbind (https://man.openbsd.org/kbind https://man.openbsd.org/kbind), so that they can have lazy binding without being susceptible to the RELRO exploits mentioned in that Usenix paper. (Unfortunately, the 2018 leviathansecurity.com article fails to mention kbind, even though kbind was added to OpenBSD in 2015.) There are other approaches. I think the PaX Team has written quite alot about their preferred techniques. But the point is that static PIE, which is desirable because of ASLR and other reasons (e.g. unification of code generation techniques), touches upon varying and distant components of toolchains and runtimes.
    - saagarjha 6y ago
      I know how ASLR/ROP/PLT/GOT/RELRO work, but I'm still not understanding what's going on here. When you statically link a binary, you get…one file. And that gets loaded into memory together, and to make it PIE you have all the jumps be pc-relative. Like, a straight up jump instruction to a fixed offset. So where the the room for a table like this? (Unrelated, but since you brought it up: kbind is IMO not a very good mitigation. It seems that all you have to do to bypass it is leak a cookie and ROP to that one place in ld.so that is "blessed" and then you not only have the ability to scribble all over your read-only GOT but as far as I can tell you can overwrite any read-only memory, which means it opens up an extremely valuable exploit primitive…)
- cosmojg 6y ago
  > but the basic takeaway is that dynamic linking in system toolchains and system runtimes is far more mature than static linking. But that's a chicken-or-egg problem, right? The only way static linking tools will reach parity with dynamic ones in terms of maturity is if static linking replaces dynamic linking as the de facto method of choice.
  - pjmlp 6y ago
    Generally speaking static compilation has had 30 years of head start across multiple OS and hardware architectures, until dynamic compilation was even an option on mainstream computing, it was after all the only way, with dynamic only available in research platforms like what was going on at Xerox PARC. So this widespread lack of knowledge that static is actually more mature than dynamic is kind of ironic.
Vogtinator 6y ago
> On average, dynamically linked executables use only 4.6% of the symbols on offer from their dependencies. That's correct, but also very misleading and leads to the wrong conclusion. The dynamically linked library has references to itself, externally visible or not. It would be wrong to claim that Application.run(); only uses a single symbol of a library. > A good linker will remove unused symbols. With LTO or -f{function,data}-secions + --gc-sections any linker will do. Without those options no linker is allowed to. I believe that this the reason why static libraries are usually shipped as separate object files (.o) within ar archives (.a), as those were only linked in on demand.
- iforgotpassword 6y ago
  > I believe that this the reason why static libraries are usually shipped as separate object files (.o) within ar archives (.a), as those were only linked in on demand. Yep. One function per C/obj file for smallest static binary possible.
  - arcticbull 6y ago
    Can't you achieve the same result with -ffunction-sections?
    - loeg 6y ago
      Yes. And its twin, -fdata-sections.
    - iforgotpassword 6y ago
      Yes but that's modern fancy stuff. :-)
    - pjmlp 6y ago
      Yes, on the compilers that offer it.
- Someone 6y ago
  Also, how many symbols it uses is relevant for the time linking takes, but not for memory usage. Worse, I think it would have been easier to measure the size in bytes of binaries, or, with a bit more effort, resident memory size.
iveqy 6y ago
Suckless has a project to get a fully static compiled Linux environment. Unfortunately I don't know how far that have come
- __s 6y ago
  https://sta.li https://sta.li
- mforney 6y ago
  I don't think stali has seen any activity in several years. As far as I know, the only completely statically linked Linux distribution that is actively developed is my own project (inspired by stali), oasis: https://github.com/oasislinux/oasis https://github.com/oasislinux/oasis
  - loeg 6y ago
    Do you static-link Linux? :-)
    - mforney 6y ago
      I'm not sure what you mean here. Are you asking whether I build my kernel drivers as modules or built-in? Personally, I build my kernels without modules, but I've never heard of that technique being called "static-linking Linux".
      - loeg 6y ago
        Yes, exactly. Loadable modules are a form of dynamic linking. I've never heard it called "static-linking Linux," either, but I think static / dynamic linking is well understood, and I'm happy you understood the meaning from context.
- cryptonector 6y ago
  The thing that's missing is static link semantics that don't suck.
k__ 6y ago
What are the counter arguments?
- old-gregg 6y ago
  Run htop or similar, sort by "shared memory" column and see how much more memory you'd need per process if shared linking did not exist. I think the author's using a wrong method to make a point. Dynamic linking feels out of place for most long-running server-side apps (typical SaaS workload). One can argue that in a mostly CLI-environment there's also not much benefit. But even an empty Ubuntu desktop runs ~400 processes and dynamic linking makes perfect sense. libc alone would have to exist in hundreds reincarnations consuming hundred+ megabytes of RAM and I'm not even talking about much, much heavier GTK+ / cairo / freetype / etc libraries needed for GUI applications.
  - ddevault 6y ago
    That is an extremely misleading figure. Shared memory is page-aligned entire libraries dropped into RAM. Statically linking would, as the article shows, only use on average about 4% of the symbols available from the libraries, and the majority of this would not end up in RAM with your statically linked binary. And if you used a more selective approach, dynamically linking to no more than perhaps a dozen high-impact libraries and statically linking the rest, you'd get a lot of the benefits and few of the drawbacks. Put the cold hard numbers right in front of someone's face and still the cargo cult wins out.
    - loeg 6y ago
      > Put the cold hard numbers right in front of someone's face and still the cargo cult wins out. Come on. You did not actually measure the figure GP mentioned and which you are disputing. Your methodology and assumption — that 4% external symbol use translates into 4% size used — is a plausible guess, but you haven't supported it with data. Even if you had measured the figure you're accusing GP of ignoring, the tone of your remark is just aggressively condescending and inappropriate. Tone it down. To address your other claims: > Shared memory is page-aligned entire libraries dropped into RAM. There is a good reason to page- or superpage-align code generally; it burns some virtual memory but reduces TLB overhead and therefore misses / invalidations, which are very costly. You would want to do the same with executable code in a static-linked binary. > the majority of [the small fraction of static linked library used] would not end up in RAM with your statically linked binary. Huh? Why do you claim that? > And if you used a more selective approach, dynamically linking to no more than perhaps a dozen high-impact libraries and statically linking the rest, you'd get a lot of the benefits and few of the drawbacks. I think that claim is plausible! But it wasn't an option presented on your blog post, nor was it discussed by GP. Prior to that comment, discussion was only around 100% vs 0% dynamic linking.
devit 6y ago
A better way to do this analysis would be to build a Linux distribution with everything statically linked and compare to the normal version with dynamic linking, looking at disk space used, startup time, memory used, and time to launch specific applications both cold and hot.
- weinzierl 6y ago
  > A better way to do this analysis would be to build a Linux distribution with everything statically linked [..] Here you go: Stali https://dl.suckless.org/htmlout/sta.li/ https://dl.suckless.org/htmlout/sta.li/ "Stali distribution smashes assumptions about Linux" https://www.infoworld.com/article/3048737/stali-distribution-smashes-assumptions-about-linux.html https://www.infoworld.com/article/3048737/stali-distribution...
  - devit 6y ago
    For a useful comparison you need the static and dynamic distributions to be otherwise the same, i.e. you want to pick a mainstream distribution and build it from scratch with both static and dynamic linking and compare.
    - a1369209993 6y ago
      I don't know of any mainstream distribution that doesn't make full-static-from-scratch builds gratuitously painful and difficult. Personally I gave up after trying to blunt-force-trauma glibc into linking correctly, though, so someone with more internals knowledge might have better success at it.
      - yjftsjthsd-h 6y ago
        Depends on your ideas of "mainstream"; I expect nixos and gentoo are both happy to do such rebuilds for you. But, as you note, the real pain is that glibc really doesn't want you to do static builds... I wonder how gentoo and/or nixos support is for musl...
AshamedCaptain 6y ago
First we claim that dynamic linking does not provide any memory savings because the libc we used is small, and later on we use our lack of dynamic linking to justify having a small libc. Smart, very smart.
- kccqzy 6y ago
  That's what happens when you decouple components and make decisions for each component separately rather than holistically for the system as a whole.
eeereerews 6y ago
>Do your installed programs share dynamic libraries? >Findings: not really >Over half of your libraries are used by fewer than 0.1% of your executables. Findings: Yes, lots, but mostly the most common ones. Dynamically linking against something in the long tail is pretty pointless though.
- rumanator 6y ago
  > Dynamically linking against something in the long tail is pretty pointless though. I disagree. Dynamic linking, in the context of an OS which offers a curated list of packages in the form of an official package repository, means that a specialized third party is able to maintain a subcomponent of your system. This means you and me and countless others are able to reap the benefit of bugfixes and security fixes provided by a third-party without being actively engaged in the process. In the context of an OS where the DLL hell problem hasn't been addressed and all software packages are forced to ship all their libraries that are shared with no one at all, indeed its pretty pointless.
  - eeereerews 6y ago
    It can also cut the other way though. Bugs can be introduced, compatibility can be broken, users can not find the library in their package manager, or they may find too new of a version. The danger of this is smaller for popular libraries, but goes up as you move to the long tail.
    - wahern 6y ago
      But this also highlights the benefit of community packaging. Debian packagers often backport security fixes into older versions of libraries that are no longer maintained upstream. That's a big part of their job--not just to bang out a build and walk away, but to keep an eye on things. This is why it's important to only use distro-packaged libraries as much as you can, even when statically linking. Getting off the treadmill of integrating interface-breaking upstream changes is one of the biggest practical reasons people prefer static linking and directly adding upstream source repositories into their build. It's at least as important, IME, as being able to use newer versions of libraries unavailable in an LTS distro. It can work well for large organizations, such as Google with their monolithic build, because they can and often do substitute the army of open source packers with their own army of people to curate and backport upstream changes. For everybody else it's quite risky, and if containerization provides any measure we're definitely worse off given the staleness problems with even the most popular containers.[1] [1] I wouldn't be surprised if an open source project emerged to provide regularly rebuilt and possibly patched upstream containers, recapitulating the organizational evolution of the traditional FOSS distribution ecosystem.
pm215 6y ago
I think a more interesting analysis of "security vulnerability costs for static linking" would look not at just "how many bytes does the end user download" but "what are the overall costs to the distro to support a fully statically linked setup", looking at eg CPU costs of doing the rebuild or how much total-elapsed-time it would take to do a full rebuild of every affected package.
- mcguire 6y ago
  Not to mention that fixing a security vulnerability in, say, libm or libc becomes an amount of work equivalent to a distribution upgrade with all the associated risks. Or am I the only one who has occasional problems when replacing all the binaries on my system?
  - yjftsjthsd-h 6y ago
    That sounds like a separate issue; normally, having to do a full dist upgrade means that you changed versions of lots of core components (kernel, init, libc, gcc), which is indeed traumatic. If you replace every binary but the only change is bumping the statically-linked libc from x.y.0 to x.y.1, it should be just as boring as making the same change with a dynamically-linked libc.
cryptonector 6y ago
By far the most important reason for dynamic linking for C is semantics: static linking semantics are stuck in 1978 and suck (more on that below), while dynamic linking semantics make C a much better language. In particular, static linking for C has two serious problems: 1. symbol collisions -> accidental interposition (and crashes); 2. you have to flatten the dependency tree into a topological sort at the final link-edit. Both of these are related, and they are disastrous. They are also related to the lack of namespaces in C. Besides fixing these issues, the C dynamic linking universe also enables things like: - run-time code injection via LD_PRELOAD and intended interposition - run-time code loading/injection via dlopen(3) - audit (sotruss) - reflection - filters (which allow one to move parts of libraries contents to other libraries without forcing re-links and without forcing built systems to change to add new -lfoo arguments to link-edits) - use of dladdr(3) to find an object's install location, and then that to find related assets' install locations relative to the first, which then yields code that can be relocated at deploy time (sure, "don't do that" is a great answer, but if you statically-link then you think you can, and now you just can't have assets to load at run-time) - use of weak symbols to detect whether a process has some library loaded and others. C with those features is a far superior language -- a different language, really -- to C without them. (EDIT: A lot of the semantics of ELF could be brought to static linking. Static link archives could have a .o that has metadata like depedencies, "rpaths", exported/protected symbols, interposer symbols, etc. The link-editor would write and consume that metadata. However, it's 2020, and the static link ecosystem is stuck in 1980 because no one has bothered, and no one has bothered because dynamic linking is pretty awesome. Still, it could be done, and once in a while I think I ought to do it to help save people from themselves who want static linking.) > Do your installed programs share dynamic libraries? > Findings: not really > Over half of your libraries are used by fewer than 0.1% of your executables. The C library most certainly gets shared, as well as libm and such. The rest, it's true, not so much, but it does depend on what you're measuring. Are you measuring C++ apps? Yeah, C++ monomorphization leads to essentially static linking. Are you measuring Java apps with no significant JNI usage? You won't find much outside the libraries the JVM uses. > Is loading dynamically linked programs faster? > Findings: definitely not Dynamically-linked programs will load faster when their dependencies are already loaded in memory, and slower otherwise. The biggest win here is the C library. > Will security vulnerabilities in libraries that have been statically linked cause large or unmanagable updates? > Findings: not really Correct. But, being able to update libc or some such and not have to worry about updating consumers you might not even know about is a very nice feature.
setr 6y ago
I believe the original reasoning for Dynamic Linking wasn't performance gains, but security gains -- someone described the driving story to me as essentially a found vulnerability in a very common library required updating and re-compiling everything on every system, scarring sysadmins globally and permanently; the space saving and performance aspects came up as later "bonuses". I have little memory of the details of the story, and I'm not 100% sure it's true, but it's a much more satisfying and reasonable argument for dynamic linking than performance/space. Of course, the more modern solution would probably be a good package manager -- if its trivial to recompile things, and track what needs to be recompiled, then dynamic linking seems to gain little, but bring in a lot of its own headaches (as we know today)
- cryptonector 6y ago
  The original reasoning for shared libraries was reducing memory footprint -- back then memory was scarce. We're talking back in the days of SVR2, the mid-80s. The original reasoning for ELF was that static link semantics suck. ELF's semantics are far superior (https://news.ycombinator.com/item?id=23656173 https://news.ycombinator.com/item?id=23656173).
- bjourne 6y ago
  Stallman and other GNU folks have claimed that they stuck with dynamic linking to foster cooperation between hackers. They knew that it wasn't a win performance-wise but they wanted to encourage hackers to help each other. The idea was that if someone found a bug in a library he or she depended on they would be "forced" to send the patch upstream rather than just fixing the bug in their local copy of the library. Thus they made dynamic linking the default in gcc and kept static linking as something of an after though. I read it on a mailing list a long time ago so I don't have a source.
- tannhaeuser 6y ago
  Don't forget that the principles of the LGPL (originally the "library GPL") were/are pretty much tied to the technical concept of a shared library object that an end-user can change, etc; IANAL so I can't say anything about the legal soundness of this interpretation. Moreover, in a recent discussion about this topic on HN, someone said the introduction of shared libs into the Linux user space was mainly in support of porting X Windows to Linux (supposedly because of binary video drivers or to accomodate MIT-licensed code?), but I haven't found any supporting reference for that. glibc's maintainer Ulrich Drepper also has pretty strong opinions on static linking [1]; no matter what you think about this technically, or Ulrich personally, his paper "How to write shared libraries" [2] is considered reference material on the subject. Personally, I think that the over-use of shared libs in the Linux userland clearly serves no purpose if users flock to entire new layers of abstractions (eg Docker-like containers) to isolate their app delivery from the IMHO overengineered mechanisms in ELF and ld.so with their multiple RUNPATHs, configs, loader scripts, and versioned glibc symbols (on top of build-time libtool/autootols) that still doesn't seem to get to the point. While the LSB effort for more uniform Linux distros isn't dead, it doesn't seem to be taken seriously. Idk, but maybe the GNU folks also see the lack of binary compat for Linux apps as a desideratum, to frustrate any and all attempts to ship binary apps? [1]: https://web.archive.org/web/20100527213559/http://people.redhat.com/drepper/no_static_linking.html https://web.archive.org/web/20100527213559/http://people.red... [2]: https://akkadia.org/drepper/dsohowto.pdf https://akkadia.org/drepper/dsohowto.pdf
dwheeler 6y ago
Not convinced. First, this analysis was done on Arch Linux, a source-based distribution. Since you know at compile time what your environment is, I would expect the benefits to be smaller. And of course, this means you're willing to do a lot of recompiles. I'd like to see analysis done on more traditional (& common) binary distros. Second, the arguments seem a little cherry-picked. "Over half of your libraries are used by fewer than 0.1% of your executables." is cute. But modern systems have a lot of executables, so 0.1% > 1, so that still matters, and what about the other half. Finally, we're already having serious problems getting containers to upgrade when a security vulnerability is found. Requiring recompilation of all transitive users is not likely to win any update speed contests. If it's completely automated then it would work, but any rocks in the process will leave people endlessly vulnerable. See the Android ecosystem, etc.
- eeereerews 6y ago
  What do you mean by "source-based distribution"?
  - riquito 6y ago
    One without packages shipped as prebuilt binaries: Gentoo [1] would be a good example. You compile every binary yourself (sometime you can use prebuilt binaries for the biggest software, if you are so inclined and the distro allows it). [1] https://www.gentoo.org/ https://www.gentoo.org/
    - danShumway 6y ago
      My Arch install doesn't compile anything locally unless I'm installing from AUR. Maybe it's doing it behind the scenes or something, but I'm doubtful, because otherwise I think my upgrades would take a lot longer.
    - lordleft 6y ago
      Doesn't arch use AUR / pacman for binaries / package management? I wasn't aware that like Gentoo you had to compile most software
bryanlarsen 6y ago
To get these stats, Drew used 5 different scripts in 5 different languages. Awk, sh, C, go & python. Well, the C program isn't a script it's a test artifact. Drew must subscribe to the "best tool for the job" philosophy rather than the "use what you know" philosophy.
albertzeyer 6y ago
> Will security vulnerabilities in libraries that have been statically linked cause large or unmanagable updates? This is maybe not an issue for open source packages which are managed by your distribution package manager, assuming they update all the dependent packages once some library gets updated (which would lead to a lot more updates all the time). However, the maybe more critical issue is about other independently installed software, or maybe closed source software, where you will not automatically get an update once some library gets updated.
- cosmojg 6y ago
  > However, the maybe more critical issue is about other independently installed software, or maybe closed source software, where you will not automatically get an update once some library gets updated. If you care about security, you shouldn't be running closed source software anyway, at least not outside of a container. Statically linked binaries for open source software, containers for everything else, and you're good to go.
earthboundkid 6y ago
Say we have program X with dependency Y. X+Y is either dynamic or static. X can either have responsive maintainers or unresponsive maintainers. Y can either change to fix a bug or change to add a bug. (With Heartbleed, I remember our server was fine because we were on some ancient version of OpenSSL.) Here are the scenarios: - dynamic responsive remove bug: Positive/neutral. Team X would have done it anyway. - dynamic unresponsive remove bug: Positive. - dynamic responsive add bug: Negative. Team X will see the bug but only be able to passively warn users not to use Y version whatever. - dynamic unresponsive add bug: Negative. Users will be impacted and have to get Y to fix the error. - static responsive remove bug: Positive/neutral: Team X will incorporate the change from Y, although possibly somewhat slower (but safer). - static unresponsive remove bug: Negative. Users will have to fork X or goad them into incorportating the fix. - static responsive add bug: Positive. Users will not get the bad version of Y. - static unresponsive add bug: Positive. Users will not get the bad version of Y. Overall, dynamic is positive 1, neutral 1, negative 2, and static is positive 2, neutral 1, negative 1. Unless you can rule out Y adding bugs, static makes more sense. Dynamic is best if "unresponsive remove bug" is likely, but if X is unresponsive, maybe you should just leave X anyway.
- petters 6y ago
  Sure, but I think the most important reason people have in mind when they argue for dynamic linking is that they will receive upstream bugfixes. I don't think your eight cases are equally important. You are of course right, though, and writing it down like this can be useful.
  - IshKebab 6y ago
    Why would you not receive upstream bug fixes with statically linked programs? Assuming you are using an Apt-style package manager then the statically linked program would be rebuilt and updated too. If you are not using an apt-style package manager then the program must include all of its dependencies (except ones that are guaranteed to be present on the platform, which is none on Linux and a few on Mac/Windows), and you will receive bug fixes when that program is updated, whether or not it uses static linking. Static/dynamic linking does not affect how likely you are to get bug fixes in any way as far as I can tell.
pengaru 6y ago
Back in the 90s we'd statically link the most frequently executed programs on busy servers for a significant performance boost. Dynamic linking is not a performance feature, it's a decoupling feature.
- binarycrusader 6y ago
  It is a performance feature if you are memory constrained. Shared libraries are “shared” for a reason. On a server with high multi-tenancy the savings can be significant.
  - jeffbee 6y ago
    Not really. Dynlinking solves a memory resource problem we had in the 80s. These days the only people with the problem are the very smallest embedded systems. As for "high multi-tenancy" there's nobody out there with server occupancy as high as Google's (see the recent Borg scheduler traces for concrete data) and they statically link everything.
    - binarycrusader 6y ago
      Not really. Yes, really. Dynlinking solves a memory resource problem we had in the 80s. These days the only people with the problem are the very smallest embedded systems. Definitely not true. Every bit of memory that's not available that could be shared memory instead is a reduction in memory available for filesystem caches, etc. As for "high multi-tenancy" there's nobody out there with server occupancy as high as Google's (see the recent Borg scheduler traces for concrete data) and they statically link everything. Cloud vendor computing models are not generally the computing model of the rest of the world. Comparison to their environment is not relevant to the general populace. I worked for a "big iron" OEM vendor until late 2017 and the savings were definitely still significant then both for their customers and the vendor themselves. There are numerous benefits to shared linking, that doesn't mean it's always the appropriate solution, but is not correct to claim that there are no performance benefits. Especially on more memory-constrained consumer devices, the shared memory benefits of dynamic linking are still significantly beneficial.
sprash 6y ago
Static linking allows LTO with aggressive inlining and is therefore able to achieve far superior performance beyond just the startup time. Arguing that dynamic vs. static has better RAM utilization or not is pointless because nowadays we have plenty of RAM but single core performance is stagnating for almost a decade already. Moores law might give us more transistors but single thread performance is still more or less bound by the clock and transistor switching frequency. Sooner or later static linking will become the only way to move forward and the conveniences dynamic linking offers will not be worth the costs.
- cryptonector 6y ago
  > Arguing that dynamic vs. static has better RAM utilization or not is pointless because nowadays we have plenty of RAM ... Using more RAM means having lower cache hit ratios. If dynamic linking means using less RAM, you win. But it's not a clear-cut thing -- it will depend a lot on the surrounding ecosystem. In any case, for C, the problem with static linking is about semantics. Until those are fixed I'll be resolutely against static linking for C, and for everything else, well, do whatever performs best.
- floatboth 6y ago
  lol, you don't want aggressive inlining everywhere. Here's a good example. Firefox does even cross-language LTO for stuff that matters — all the DOM/CSS/etc components that are tightly bound together. But it happily allows you to dynamically link to a system libjpeg, and you won't lose any performance because the libxul<->libjpeg boundary is crossed.. about a couple times per downloaded jpeg. Heck, what would you even inline between the jpeg decoder and the browser? Absolutely nothing! The world at large is already using static linking where it matters.. e.g. with C++, header-only libraries and anything that uses templates results in a lot of static linking. Even if you link to libboost_something.so, you probably already have most of that something inlined into your code because templates :)
- ndesaulniers 6y ago
  static linking != LTO LTO requires source, or at least IR. Static linking involves object files; there's been a loss of fidelity at that point that prevents inlining.
butterisgood 6y ago
ldd won’t get you all the programs that dynamically load with dlopen... but these are still interesting results.
sitzkrieg 6y ago
these benchmarks dont seem to take into account page cache
gok 6y ago
Why stop there? Package every binary in its own container, too! Most programs only use a few system calls, so you're not really getting anything by sharing a single kernel for the entire system.
- Shorel 6y ago
  Seems to be the general trend. A virtualized kernel per service.
- kortex 6y ago
  That's literally what RancherOS does. ls is a container. Blew my mind the first time I saw it. But they take the "cattle not pets" approach pretty much to its logical extreme (I suspect that the company's name arises from this metaphor) It makes ops so liquid and convenient. I love it. To really push this idea (and microkernel) to its logical extreme, it would be cool to see a null-kernel. You have socket drivers, that's it. Everything is either a network call or some sort of IPC. You might still need something to handle paging, though, and obviously your "peripherals" like storage would need to be more conventional in nature.
  - pjmlp 6y ago
    It makes sense from security point of view, but it definitely is much more resource intensive. Imagine something like InteliJ or Eclipse, where every single IDE plugin is its own process doing IPC.
- yencabulator 6y ago
  All containers on a single host very much share the same kernel. You're confused about virtualization vs containers.
pengaru 6y ago
Static linking has been such a nuisance for the libSDL folks that they implemented dynamic loading of itself [0], controlled via an environment variable, as an escape hatch from executables w/libSDL linked statically. It's understandable that games, especially proprietary ones, distribute statically-linked binaries ensuring any third-party dependencies will be present and be a compatible version. But the value of that decision tends to diminish with time, as those external dependencies are frequently the pieces interfacing with the system/outside world, which keeps changing, leaving such dependencies behind to atrophy at best or become vulnerable/incompatible/broken at worst. I don't personally think it makes sense to approach this so dogmatically. Static linking makes sense in the right circumstances, so does dynamic linking. For general-purpose operating systems, it seems obvious to me that you'd want most higher-order userspace programs dynamically linked. I want my openssl updates to touch a single library and affect all installed ssl-using programs, for example. Having said that, I do wish the average linux distro still statically linked everything in /bin and /sbin. It was nice to still be able to administrate the system even when the dynamic libraries were hosed. At some point it was changed to just a single static binary; sln for static ln IIRC, assuming you'd be able to fix your dynamic libraries with some symlinks if they were broken, if you happened to have a shell running and could navigate using just builtins. It was already an impossible situation, but even that seems to be gone nowadays. It's a more nuanced issue, taking an "everything dynamically linked!" or "everything statically linked!" approach strikes me as just another form of ignorant extremism. [0] http://hg.libsdl.org/SDL/file/2fabbbee604c/src/dynapi/SDL_dynapi.c#l292 http://hg.libsdl.org/SDL/file/2fabbbee604c/src/dynapi/SDL_dy...
- jart 6y ago
  There's nothing wrong with taking a principled approach to builds. For example, it's not just a question of static or dynamic. Linking of vendored/static/pinned/etc. sources/artifacts/etc. in general comes with the benefit of hermeticity. That enables tools to work better, since it gives them perfect knowledge of the things that exist. It also entails a certain degree of responsibility for writing your own scripts that can generate automated alerts should issues arise in the upstream sources.
- cryptonector 6y ago
bjourne 6y ago
Either something is wrong with the testing script or my computer is way faster than I thought: ./test.sh | awk 'BEGIN { sum = 0 } { sum += $2-$1 } END { print sum / NR }' -698915
joosters 6y ago
Over half of your libraries are used by fewer than 0.1% of your executables That's a very misleading reference and graph. First of all, what did they expect to find? As you add more executables, of course the % usage of a library will decrease. e.g. say I have a networking library on my computer, and, in a perfect world, all my installed network tools link against it. But now I install Gnome, and my machine has hundreds more binaries. Not all of the binaries will do networking stuff, so the % usage of the networking library goes down. But that doesn't mean that the networking library is not being shared as well as it could be. A much better metric would be to count, for each shared library on a machine, the number of programs that link against it. If only one program uses a shared library, then that means the 'shared-ness' is not being used. If more than one program use it, then the library is being effectively shared. But the actual count, whether it is 200 users or 20, doesn't mean anything more. That's why comparing all libraries against libc's usage shows nothing useful.
- jjoonathan 6y ago
  Yeah. Loading the GUI widget toolkit once rather than 30 times is pretty nifty.
- cryptonector 6y ago
  Even better would be to measure sharing in the totality of programs you use directly and indirectly from boot.
saagarjha 6y ago
As always, static and dynamic linking both have their advantages and drawbacks. The usual arguments for dynamic linking around brought up in the article, and as others have mentioned here, the analysis is a bit lacking so the conclusions aren't generally true. Static linking has its own, fairly straightforwards benefits as well. It's no surprise that those who push one or the other usually do so because of their specific needs. Sometimes we even see some interesting hybrid solutions: one recent one is Apple introducing dyld shared caches on macOS, which (while being a pain to reverse engineer) are basically all the system (dynamic) libraries all statically linked together and presented dynamically, with some linker tricks to make it appear seamless. Likewise, a lot of statically linked binaries are only partially statically linked, still using things like libc or graphics libraries. The moral really is to try both and pick whichever one is the one that's better for your use case, and perhaps even consider a mix of both to give you the most flexibility in which tradeoffs you'd like to make.
- cryptonector 6y ago
  Specifically for C, static linking is a trap due to its poor semantics. Don't do it. Or fix the C link-editors, then static linking C will be fine.
  - sigjuice 6y ago
    What would be examples of poor static linking semantics? Thanks!
    - cryptonector 6y ago
      https://news.ycombinator.com/item?id=23656173 https://news.ycombinator.com/item?id=23656173
- comex 6y ago
  Nitpick: macOS has used dyld shared caches for over a decade. The recent change is just to remove the original copies of the libraries on disk.
andyjpb 6y ago
This is an interesting analysis. For my part (anecdata), in my /usr/bin (Debian) I have 2,956 files under 9MiB in size, 1 of 13MiB and one of 39MiB. Most of the files are (much) under 1.0MiB. On the other hand, I have a three statically linked binaries for CloudFoundry in my home directory. One for Linux, one for MacOS and one for Windows. They are each between 24MiB and 27MiB each.
dan-robertson 6y ago
I’m curious about the supposed memory advantages of dynamic linking: on average how many different executables share each page of memory? What about when memory is in high demand? How high does that average become? What is the probability that a page of a shared library is already in memory (cache or otherwise) when it needs to be loaded, and in particular the probability that it is there because another program loaded it). My guess is that apart from eg libc, the average is pretty low (ie 1 for the pages that aren’t free).
- saagarjha 6y ago
  I think this depends heavily on your platform and how often libraries are reused. On macOS, for example, most libraries aside from libc are loaded R/O or CoW quite literally hundreds of times, because every app shares AppKit and WebKit and Security and the dozens of other platform frameworks (and their private dependencies!) that are basically "free" to use and ship with the system and so have very high adoption. On more "coherent" Linux distributions I'm sure things like GTK, glib, OpenSSL, zlib are used by a lot of things too. Sure, there's going to be a lot of one-off dynamic libraries too, but there's a lot of duplication with the popular dependencies and then a long tail.
pwdisswordfish2 6y ago
With NetBSD, it is relatively easy to compile the entire system as static. Is it easy to do this with Arch Linux? (Maybe Void Linux?)
zelly 6y ago
Dynamic linking provides encapsulation and security benefits. An application with a statically linked OpenSSL can be vulnerable if a CVE comes out for that version of OpenSSL, whereas a dynamically linked OpenSSL could be patched immediately without recompilation (which may be impossible if the software is proprietary). The vendor of the shared library can update the implementation without requiring all downstream consumers to recompile. This should not be downplayed. Pure static linking makes sense when you are deploying code you control onto an environment you control. While I like ease of deploying Go, where binaries are one big blob that just works, it makes it closer to the Java style than the UNIX modular-tool-does-one-thing-well tradition.
- vlovich123 6y ago
  From distributions where you build everything from source (most cloud providers today, mobile phone operators, etc), such advantages are purely theoretical & never play out in practice. In such environments you're pushing out updates of the entire system on whatever your regular cadence is. Additionally, because the old library is still mapped & running you have to know to restart all the processes that have the old version linked in. Not an easy task. About the only place it kind of matters is Linux distros where you are deploying closed source binaries linked against system libraries. I'm not sure that's a significant use-case. The only performance benefit is memory. If you have a core infrastructure library that's widely shared (e.g. libopenssl) then it can have some benefit. In practice I much prefer the microservice model with a formal IPC API. Then the SW update is trivial to fix the exploit - just kill the 1 process providing the service.
  - zelly 6y ago
    Suppose 100 processes on the same box call the same function in glibc. If each process had its own redundant version of it mapped into memory, then that's 100x more cache misses compared to them all sharing the same page. I get that only a handful of libraries are actually used, but that's a very fat tail and the performance will be very bad if that handful of libraries are always statically linked. This may be less important if the program is in a container (isn't sharing anyway) or launched as an AWS Lambda throwaway. I dunno, maybe dynamic linking isn't relevant anymore now that most Linux programs are meant to run in containers on an expensive cloud with tons of memory. > In practice I much prefer the microservice model with a formal IPC API. This converts a simple and solved problem, sharing memory, to a client-server distributed systems problem. There is no need. It's all on the same box.
ncmncm 6y ago
Paging Laura Creighton! Your time is now.
- cryptonector 6y ago
  What should I search for to get a clue as to what you're referring to?
  - ncmncm 6y ago
    Her name, and USENET. She was the most vocal opponent of the move to dynamic-linked libraries, back when it happened. Her expectation was that programs that once worked would stop working as the libraries rotted.
Shorel 6y ago
I think the elephant in the room here is things like Flatpack and Snap, which are basically ugly bastardized versions of static linking.
- eeereerews 6y ago
  Ugly maybe, but they're do kinda combine the advantages of both. They have the portability and reproducibility advantages of static linking, while still letting you take them apart and change the dependencies like with dynamic linking if you really want to.
  - Shorel 6y ago
    And the disadvantages of both. These being the huge loading times (even for a calculator!) and the disk space required.
cycomanic 6y ago
The first graph looks like a power law. It's fascinating how many processes follow power laws. As a side note, you are almost always better off (also on this case) by plotting power laws on a log scale.
ridiculous_fish 6y ago
This is a crazy conversation! Dynamic linking allows the system to decide the UI. Static linking means that the UI cannot evolve. Imagine statically linking UIKit or Android's UI library!
- saagarjha 6y ago
  Yeah but then you have CFLinkedOnOrAfter to negate that ;)
- thu2111 6y ago
  UI libraries tend to evolve by introducing new APIs or modes, because changing UI implementations has a habit of breaking apps. For example, on Android even the switch to hw accelerated rendering was a mode app devs had to opt in to, that didn't even change the look!
mrich 6y ago
During development of a large system static linking is hell. The final link just takes too long if you are only working on one object file. Also during debugging you have to load all symbols in gdb, which does not scale. With dynamic linking you only load the debug info for the library you are working on.
- adrianN 6y ago
  GDB handles binaries in the hundreds of megabytes fairly well.
tsimionescu 6y ago
> The total download cost to upgrade all binaries on my system which were affected by CVEs in 2019 is 3.8 GiB. This is reduced to 1.0 GiB if you eliminate glibc. The upgrade problem has almost nothing to do with download size. The real problem is that you have > 100 binaries which depend on those libraries, and instead of having the library authors go and update the library, you need each team responsible for one or more binaries to go and take the new library and release a new version of their binary. And then, when you want to check if your system is safe from Heartbleed, instead of checking if you have libopenssl > 1.0.1g, you need to check if bin1 > 1.2.56 or > 0.6.89h, bin2 > 5.76.1, or > 4.6.215,... bin100 > 1.67.89. And of course, if one of them does NOT have a newer version compiled with the patched library, you need to fix it yourself, and maintain a patched version of the binary. Assuming that you even know that binary had been linked to the vulnerable library.
- danieldk 6y ago
  The real problem is that you have > 100 binaries which depend on those libraries, and instead of having the library authors go and update the library, you need each team responsible for one or more binaries to go and take the new library and release a new version of their binary. This is a solvable problem. Package managers such as Nix and Guix rebuild packages if any of their transitive dependencies have been changed. The difficult part are now language-specific ecosystems that use lock files to lock all their dependencies. Traditional C/C++ programs, either statically linked or dynamically linked, have a dependency graph such that e.g. each program that uses OpenSSL has the same package definition in their transitive dependencies. However, e.g. Rust programs may have different versions of the same crate locked. (There are solutions to that, but there is still work to be done.)
  - tsimionescu 6y ago
    I was talking about finding vulnerabilities on end-user systems and servers. I don't know about others, but I don't generally keep compiler tool chains for C, C++, Java, Go and maybe 1-2 others on my systems and on my servers in case I need to rebuild all of my binaries.
nabla9 6y ago
Would this distinction would go largely away if OS would have method to share read only memory pages or larger regions of memory between all processes? I suspect you would need to compute signatures for static libraries for binaries located /bin/, /lib, ... to make load times faster.
enriquto 6y ago
Static executables are a sweet spot between the craziness of dynamic dependencies (where we come from) and the idiocy of shipping a whole virtual machine for each program (where we seem to be going to). If we could just stop at that spot!
- AnIdiotOnTheNet 6y ago
  I kindof agree, but I think the real sweet spot is that system libraries[0] should be dynamically linked and everything else should be static. For Linux though, where the very concept of a standardized base system is loathed, static the sweet spot. [0] libraries that are likely to be used by a large number of applications like the GUI libraries, kernel interfaces, networking, encryption, etc.
perryizgr8 6y ago
The main problem I have with dynamic loading is that my app crashes without any sensible error message on runtime, if I misspelled a single symbol.

408 points by scrollaway 6y ago

I appreciate the brevity of this article.

Do any Linux/glibc or Linux/musl systems support static PIE binaries, yet? Without static PIE support you don't benefit from ASLR (at least not fully). This 2018 article seems like a good breakdown of the issues: https://www.leviathansecurity.com/blog/aslr-protection-for-statically-linked-executables https://www.leviathansecurity.com/blog/aslr-protection-for-s... OpenBSD has supported static PIE since 2015; not just supported, but all system static binaries (e.g. /bin and /sbin) are built as static PIEs, and I believe PIE is the default behavior when using `cc -static`. The reasons for the switch and the required toolchain changes are summarized in this presentation: https://www.openbsd.org/papers/asiabsdcon2015-pie-slides.pdf https://www.openbsd.org/papers/asiabsdcon2015-pie-slides.pdf Also, simply checking the "static PIE" box isn't the end of the story. There are different ways to accomplish it, and some are better than others in terms of ASLR, W^X, and other exploit mitigations. It's been a couple of years since I last looked into and had a hold on all the issues simultaneously[1], but the basic takeaway is that dynamic linking in system toolchains and system runtimes is far more mature than static linking. [1] static PIE issues are a nexus of exploit mitigation techniques, so if you want to deep dive into exploit mitigation or even just linking issues then chasing the static PIE rabbit is a good approach.

monocasa 6y ago
I imagine there's some way to, because the kernel is able to apply ASLR to the dynamic linker itself.
- wahern 6y ago
  It's not a matter of possible, but whether the work has been done throughout the toolchain and runtime stacks. When OpenBSD implemented static PIE, -static and -pie, the literal options and the code generation aspects more generally, were still mutually exclusive in GCC and clang. GCC required patching to enable both static linking and PIE generation. In fact, my local GCC 9.3 man page still says that -static overrides -pie; to build static PIE seems to require the special option -static-pie, in addition to -fPIE/-fPIC when building the object code. But it's not enough for the compiler and compile time linker to support it. libc and libstdc++/libc++ also need support, both internally as well as in the built libraries. And I think libdl might need support if you want dlopen to work from a static PIE binary. Likewise for libpthread. Supporting static PIE requires the cooperation of many moving parts. And that's just to support it. Making it easy, let alone the default behavior, so that it doesn't require many carefully coordinated, obscure flags without the risk of any misstep silently disabling some exploit mitigation, is yet another story. It can be done. It should be done. But to what extent has it been done? I'm not sure, though some quick Googling suggests GCC and clang are mostly there, at least in terms of nominal support (which, again, is distinct from fully supporting all the same mitigation measures). glibc seems to have gotten some support (e.g. --enable-static-pie in glibc's build), though I'm not sure whether it's available in distros. And I would guess that musl libc support is pretty far along, and presumably more mature than glibc's given that Rich Felker started experimenting with static PIE several years ago, shortly after OpenBSD did their work. See https://www.openwall.com/lists/musl/2015/06/01/12 https://www.openwall.com/lists/musl/2015/06/01/12
  - monocasa 6y ago
    For sure. My comment was directed at Linux the kernel rather than Linux the ecosystem, but it's a totally valid point that for practical purposes that distinction can border on meaningless.
mforney 6y ago
Most Linux/musl systems support it. On Alpine, gcc is built with `--enable-default-pie` so all static libraries can be linked into a static PIE. On vanilla gcc (since version 8 when static PIE was upstreamed), `-static` means non-PIE static executable and there is a separate flag for `-static-pie` for static PIE. Alpine patches gcc so that `-static` and `-pie` are independent flags, so both `cc -static` and `cc -static-pie` will produce a static PIE.
Ericson2314 6y ago
Nixpkgs / NixOS does.
saagarjha 6y ago
I'm not sure I understand the first link you posted: if a binary is statically linked, why does it need a GOT? It's literally calling functions in its own binary…
- wahern 6y ago
  But where in its own binary? To support ASLR of code (a ROP mitigation), especially fine-grain randomization, function call sites can't use static addressing, either absolute or relative, pointing directly to a function. To support ASLR of code you can either rewrite every function call site on load (kinda similar to DLLs on Windows), or use one or more tables that are updated at runtime (PLTs, GOT, etc). BSD and Linux environments use the PLT/GOT approach because 1) it's a better fit for ELF, 2) shares most of the preexisting instrumentation with dynamic linking, and 3) preserves the ability to share most of the mapped pages of static binaries (imagine if a static /bin/sh was effectively rewritten on every invocation). Trying to accomplish this safely and effectively is what RELRO is about. See https://www.redhat.com/en/blog/hardening-elf-binaries-using-relocation-read-only-relro https://www.redhat.com/en/blog/hardening-elf-binaries-using-... But RELRO poses some dilemmas regarding performance, and depending on your choices as well as the details of the implementation, RELRO might be ineffective or, worse, introduce exploits of its own. See https://www.usenix.org/system/files/conference/usenixsecurity15/sec15-paper-di-frederico.pdf https://www.usenix.org/system/files/conference/usenixsecurit... It's difficult to achieve a design that provides all the desirable exploit mitigations without sacrificing startup latency and other features. OpenBSD added a new syscall, kbind (https://man.openbsd.org/kbind https://man.openbsd.org/kbind), so that they can have lazy binding without being susceptible to the RELRO exploits mentioned in that Usenix paper. (Unfortunately, the 2018 leviathansecurity.com article fails to mention kbind, even though kbind was added to OpenBSD in 2015.) There are other approaches. I think the PaX Team has written quite alot about their preferred techniques. But the point is that static PIE, which is desirable because of ASLR and other reasons (e.g. unification of code generation techniques), touches upon varying and distant components of toolchains and runtimes.
  - saagarjha 6y ago
    I know how ASLR/ROP/PLT/GOT/RELRO work, but I'm still not understanding what's going on here. When you statically link a binary, you get…one file. And that gets loaded into memory together, and to make it PIE you have all the jumps be pc-relative. Like, a straight up jump instruction to a fixed offset. So where the the room for a table like this? (Unrelated, but since you brought it up: kbind is IMO not a very good mitigation. It seems that all you have to do to bypass it is leak a cookie and ROP to that one place in ld.so that is "blessed" and then you not only have the ability to scribble all over your read-only GOT but as far as I can tell you can overwrite any read-only memory, which means it opens up an extremely valuable exploit primitive…)
cosmojg 6y ago
> but the basic takeaway is that dynamic linking in system toolchains and system runtimes is far more mature than static linking. But that's a chicken-or-egg problem, right? The only way static linking tools will reach parity with dynamic ones in terms of maturity is if static linking replaces dynamic linking as the de facto method of choice.
- pjmlp 6y ago
  Generally speaking static compilation has had 30 years of head start across multiple OS and hardware architectures, until dynamic compilation was even an option on mainstream computing, it was after all the only way, with dynamic only available in research platforms like what was going on at Xerox PARC. So this widespread lack of knowledge that static is actually more mature than dynamic is kind of ironic.

Vogtinator 6y ago

> On average, dynamically linked executables use only 4.6% of the symbols on offer from their dependencies. That's correct, but also very misleading and leads to the wrong conclusion. The dynamically linked library has references to itself, externally visible or not. It would be wrong to claim that Application.run(); only uses a single symbol of a library. > A good linker will remove unused symbols. With LTO or -f{function,data}-secions + --gc-sections any linker will do. Without those options no linker is allowed to. I believe that this the reason why static libraries are usually shipped as separate object files (.o) within ar archives (.a), as those were only linked in on demand.

iforgotpassword 6y ago
> I believe that this the reason why static libraries are usually shipped as separate object files (.o) within ar archives (.a), as those were only linked in on demand. Yep. One function per C/obj file for smallest static binary possible.
- arcticbull 6y ago
  Can't you achieve the same result with -ffunction-sections?
  - loeg 6y ago
    Yes. And its twin, -fdata-sections.
  - iforgotpassword 6y ago
    Yes but that's modern fancy stuff. :-)
  - pjmlp 6y ago
    Yes, on the compilers that offer it.
Someone 6y ago
Also, how many symbols it uses is relevant for the time linking takes, but not for memory usage. Worse, I think it would have been easier to measure the size in bytes of binaries, or, with a bit more effort, resident memory size.

iveqy 6y ago

Suckless has a project to get a fully static compiled Linux environment. Unfortunately I don't know how far that have come

__s 6y ago
https://sta.li https://sta.li
mforney 6y ago
I don't think stali has seen any activity in several years. As far as I know, the only completely statically linked Linux distribution that is actively developed is my own project (inspired by stali), oasis: https://github.com/oasislinux/oasis https://github.com/oasislinux/oasis
- loeg 6y ago
  Do you static-link Linux? :-)
  - mforney 6y ago
    I'm not sure what you mean here. Are you asking whether I build my kernel drivers as modules or built-in? Personally, I build my kernels without modules, but I've never heard of that technique being called "static-linking Linux".
    - loeg 6y ago
      Yes, exactly. Loadable modules are a form of dynamic linking. I've never heard it called "static-linking Linux," either, but I think static / dynamic linking is well understood, and I'm happy you understood the meaning from context.
cryptonector 6y ago
The thing that's missing is static link semantics that don't suck.

k__ 6y ago

What are the counter arguments?

old-gregg 6y ago
Run htop or similar, sort by "shared memory" column and see how much more memory you'd need per process if shared linking did not exist. I think the author's using a wrong method to make a point. Dynamic linking feels out of place for most long-running server-side apps (typical SaaS workload). One can argue that in a mostly CLI-environment there's also not much benefit. But even an empty Ubuntu desktop runs ~400 processes and dynamic linking makes perfect sense. libc alone would have to exist in hundreds reincarnations consuming hundred+ megabytes of RAM and I'm not even talking about much, much heavier GTK+ / cairo / freetype / etc libraries needed for GUI applications.
- ddevault 6y ago
  That is an extremely misleading figure. Shared memory is page-aligned entire libraries dropped into RAM. Statically linking would, as the article shows, only use on average about 4% of the symbols available from the libraries, and the majority of this would not end up in RAM with your statically linked binary. And if you used a more selective approach, dynamically linking to no more than perhaps a dozen high-impact libraries and statically linking the rest, you'd get a lot of the benefits and few of the drawbacks. Put the cold hard numbers right in front of someone's face and still the cargo cult wins out.
  - loeg 6y ago
    > Put the cold hard numbers right in front of someone's face and still the cargo cult wins out. Come on. You did not actually measure the figure GP mentioned and which you are disputing. Your methodology and assumption — that 4% external symbol use translates into 4% size used — is a plausible guess, but you haven't supported it with data. Even if you had measured the figure you're accusing GP of ignoring, the tone of your remark is just aggressively condescending and inappropriate. Tone it down. To address your other claims: > Shared memory is page-aligned entire libraries dropped into RAM. There is a good reason to page- or superpage-align code generally; it burns some virtual memory but reduces TLB overhead and therefore misses / invalidations, which are very costly. You would want to do the same with executable code in a static-linked binary. > the majority of [the small fraction of static linked library used] would not end up in RAM with your statically linked binary. Huh? Why do you claim that? > And if you used a more selective approach, dynamically linking to no more than perhaps a dozen high-impact libraries and statically linking the rest, you'd get a lot of the benefits and few of the drawbacks. I think that claim is plausible! But it wasn't an option presented on your blog post, nor was it discussed by GP. Prior to that comment, discussion was only around 100% vs 0% dynamic linking.

devit 6y ago

A better way to do this analysis would be to build a Linux distribution with everything statically linked and compare to the normal version with dynamic linking, looking at disk space used, startup time, memory used, and time to launch specific applications both cold and hot.

weinzierl 6y ago
> A better way to do this analysis would be to build a Linux distribution with everything statically linked [..] Here you go: Stali https://dl.suckless.org/htmlout/sta.li/ https://dl.suckless.org/htmlout/sta.li/ "Stali distribution smashes assumptions about Linux" https://www.infoworld.com/article/3048737/stali-distribution-smashes-assumptions-about-linux.html https://www.infoworld.com/article/3048737/stali-distribution...
- devit 6y ago
  For a useful comparison you need the static and dynamic distributions to be otherwise the same, i.e. you want to pick a mainstream distribution and build it from scratch with both static and dynamic linking and compare.
  - a1369209993 6y ago
    I don't know of any mainstream distribution that doesn't make full-static-from-scratch builds gratuitously painful and difficult. Personally I gave up after trying to blunt-force-trauma glibc into linking correctly, though, so someone with more internals knowledge might have better success at it.
    - yjftsjthsd-h 6y ago
      Depends on your ideas of "mainstream"; I expect nixos and gentoo are both happy to do such rebuilds for you. But, as you note, the real pain is that glibc really doesn't want you to do static builds... I wonder how gentoo and/or nixos support is for musl...

AshamedCaptain 6y ago

First we claim that dynamic linking does not provide any memory savings because the libc we used is small, and later on we use our lack of dynamic linking to justify having a small libc. Smart, very smart.

kccqzy 6y ago
That's what happens when you decouple components and make decisions for each component separately rather than holistically for the system as a whole.

eeereerews 6y ago

>Do your installed programs share dynamic libraries? >Findings: not really >Over half of your libraries are used by fewer than 0.1% of your executables. Findings: Yes, lots, but mostly the most common ones. Dynamically linking against something in the long tail is pretty pointless though.

rumanator 6y ago
> Dynamically linking against something in the long tail is pretty pointless though. I disagree. Dynamic linking, in the context of an OS which offers a curated list of packages in the form of an official package repository, means that a specialized third party is able to maintain a subcomponent of your system. This means you and me and countless others are able to reap the benefit of bugfixes and security fixes provided by a third-party without being actively engaged in the process. In the context of an OS where the DLL hell problem hasn't been addressed and all software packages are forced to ship all their libraries that are shared with no one at all, indeed its pretty pointless.
- eeereerews 6y ago
  It can also cut the other way though. Bugs can be introduced, compatibility can be broken, users can not find the library in their package manager, or they may find too new of a version. The danger of this is smaller for popular libraries, but goes up as you move to the long tail.
  - wahern 6y ago
    But this also highlights the benefit of community packaging. Debian packagers often backport security fixes into older versions of libraries that are no longer maintained upstream. That's a big part of their job--not just to bang out a build and walk away, but to keep an eye on things. This is why it's important to only use distro-packaged libraries as much as you can, even when statically linking. Getting off the treadmill of integrating interface-breaking upstream changes is one of the biggest practical reasons people prefer static linking and directly adding upstream source repositories into their build. It's at least as important, IME, as being able to use newer versions of libraries unavailable in an LTS distro. It can work well for large organizations, such as Google with their monolithic build, because they can and often do substitute the army of open source packers with their own army of people to curate and backport upstream changes. For everybody else it's quite risky, and if containerization provides any measure we're definitely worse off given the staleness problems with even the most popular containers.[1] [1] I wouldn't be surprised if an open source project emerged to provide regularly rebuilt and possibly patched upstream containers, recapitulating the organizational evolution of the traditional FOSS distribution ecosystem.

pm215 6y ago

I think a more interesting analysis of "security vulnerability costs for static linking" would look not at just "how many bytes does the end user download" but "what are the overall costs to the distro to support a fully statically linked setup", looking at eg CPU costs of doing the rebuild or how much total-elapsed-time it would take to do a full rebuild of every affected package.

mcguire 6y ago
Not to mention that fixing a security vulnerability in, say, libm or libc becomes an amount of work equivalent to a distribution upgrade with all the associated risks. Or am I the only one who has occasional problems when replacing all the binaries on my system?
- yjftsjthsd-h 6y ago
  That sounds like a separate issue; normally, having to do a full dist upgrade means that you changed versions of lots of core components (kernel, init, libc, gcc), which is indeed traumatic. If you replace every binary but the only change is bumping the statically-linked libc from x.y.0 to x.y.1, it should be just as boring as making the same change with a dynamically-linked libc.

cryptonector 6y ago

By far the most important reason for dynamic linking for C is semantics: static linking semantics are stuck in 1978 and suck (more on that below), while dynamic linking semantics make C a much better language. In particular, static linking for C has two serious problems: 1. symbol collisions -> accidental interposition (and crashes); 2. you have to flatten the dependency tree into a topological sort at the final link-edit. Both of these are related, and they are disastrous. They are also related to the lack of namespaces in C. Besides fixing these issues, the C dynamic linking universe also enables things like: - run-time code injection via LD_PRELOAD and intended interposition - run-time code loading/injection via dlopen(3) - audit (sotruss) - reflection - filters (which allow one to move parts of libraries contents to other libraries without forcing re-links and without forcing built systems to change to add new -lfoo arguments to link-edits) - use of dladdr(3) to find an object's install location, and then that to find related assets' install locations relative to the first, which then yields code that can be relocated at deploy time (sure, "don't do that" is a great answer, but if you statically-link then you think you can, and now you just can't have assets to load at run-time) - use of weak symbols to detect whether a process has some library loaded and others. C with those features is a far superior language -- a different language, really -- to C without them. (EDIT: A lot of the semantics of ELF could be brought to static linking. Static link archives could have a .o that has metadata like depedencies, "rpaths", exported/protected symbols, interposer symbols, etc. The link-editor would write and consume that metadata. However, it's 2020, and the static link ecosystem is stuck in 1980 because no one has bothered, and no one has bothered because dynamic linking is pretty awesome. Still, it could be done, and once in a while I think I ought to do it to help save people from themselves who want static linking.) > Do your installed programs share dynamic libraries? > Findings: not really > Over half of your libraries are used by fewer than 0.1% of your executables. The C library most certainly gets shared, as well as libm and such. The rest, it's true, not so much, but it does depend on what you're measuring. Are you measuring C++ apps? Yeah, C++ monomorphization leads to essentially static linking. Are you measuring Java apps with no significant JNI usage? You won't find much outside the libraries the JVM uses. > Is loading dynamically linked programs faster? > Findings: definitely not Dynamically-linked programs will load faster when their dependencies are already loaded in memory, and slower otherwise. The biggest win here is the C library. > Will security vulnerabilities in libraries that have been statically linked cause large or unmanagable updates? > Findings: not really Correct. But, being able to update libc or some such and not have to worry about updating consumers you might not even know about is a very nice feature.

setr 6y ago

I believe the original reasoning for Dynamic Linking wasn't performance gains, but security gains -- someone described the driving story to me as essentially a found vulnerability in a very common library required updating and re-compiling everything on every system, scarring sysadmins globally and permanently; the space saving and performance aspects came up as later "bonuses". I have little memory of the details of the story, and I'm not 100% sure it's true, but it's a much more satisfying and reasonable argument for dynamic linking than performance/space. Of course, the more modern solution would probably be a good package manager -- if its trivial to recompile things, and track what needs to be recompiled, then dynamic linking seems to gain little, but bring in a lot of its own headaches (as we know today)

cryptonector 6y ago
The original reasoning for shared libraries was reducing memory footprint -- back then memory was scarce. We're talking back in the days of SVR2, the mid-80s. The original reasoning for ELF was that static link semantics suck. ELF's semantics are far superior (https://news.ycombinator.com/item?id=23656173 https://news.ycombinator.com/item?id=23656173).
bjourne 6y ago
Stallman and other GNU folks have claimed that they stuck with dynamic linking to foster cooperation between hackers. They knew that it wasn't a win performance-wise but they wanted to encourage hackers to help each other. The idea was that if someone found a bug in a library he or she depended on they would be "forced" to send the patch upstream rather than just fixing the bug in their local copy of the library. Thus they made dynamic linking the default in gcc and kept static linking as something of an after though. I read it on a mailing list a long time ago so I don't have a source.
tannhaeuser 6y ago
Don't forget that the principles of the LGPL (originally the "library GPL") were/are pretty much tied to the technical concept of a shared library object that an end-user can change, etc; IANAL so I can't say anything about the legal soundness of this interpretation. Moreover, in a recent discussion about this topic on HN, someone said the introduction of shared libs into the Linux user space was mainly in support of porting X Windows to Linux (supposedly because of binary video drivers or to accomodate MIT-licensed code?), but I haven't found any supporting reference for that. glibc's maintainer Ulrich Drepper also has pretty strong opinions on static linking [1]; no matter what you think about this technically, or Ulrich personally, his paper "How to write shared libraries" [2] is considered reference material on the subject. Personally, I think that the over-use of shared libs in the Linux userland clearly serves no purpose if users flock to entire new layers of abstractions (eg Docker-like containers) to isolate their app delivery from the IMHO overengineered mechanisms in ELF and ld.so with their multiple RUNPATHs, configs, loader scripts, and versioned glibc symbols (on top of build-time libtool/autootols) that still doesn't seem to get to the point. While the LSB effort for more uniform Linux distros isn't dead, it doesn't seem to be taken seriously. Idk, but maybe the GNU folks also see the lack of binary compat for Linux apps as a desideratum, to frustrate any and all attempts to ship binary apps? [1]: https://web.archive.org/web/20100527213559/http://people.redhat.com/drepper/no_static_linking.html https://web.archive.org/web/20100527213559/http://people.red... [2]: https://akkadia.org/drepper/dsohowto.pdf https://akkadia.org/drepper/dsohowto.pdf

dwheeler 6y ago

Not convinced. First, this analysis was done on Arch Linux, a source-based distribution. Since you know at compile time what your environment is, I would expect the benefits to be smaller. And of course, this means you're willing to do a lot of recompiles. I'd like to see analysis done on more traditional (& common) binary distros. Second, the arguments seem a little cherry-picked. "Over half of your libraries are used by fewer than 0.1% of your executables." is cute. But modern systems have a lot of executables, so 0.1% > 1, so that still matters, and what about the other half. Finally, we're already having serious problems getting containers to upgrade when a security vulnerability is found. Requiring recompilation of all transitive users is not likely to win any update speed contests. If it's completely automated then it would work, but any rocks in the process will leave people endlessly vulnerable. See the Android ecosystem, etc.

eeereerews 6y ago
What do you mean by "source-based distribution"?
- riquito 6y ago
  One without packages shipped as prebuilt binaries: Gentoo [1] would be a good example. You compile every binary yourself (sometime you can use prebuilt binaries for the biggest software, if you are so inclined and the distro allows it). [1] https://www.gentoo.org/ https://www.gentoo.org/
  - danShumway 6y ago
    My Arch install doesn't compile anything locally unless I'm installing from AUR. Maybe it's doing it behind the scenes or something, but I'm doubtful, because otherwise I think my upgrades would take a lot longer.
  - lordleft 6y ago
    Doesn't arch use AUR / pacman for binaries / package management? I wasn't aware that like Gentoo you had to compile most software

bryanlarsen 6y ago

To get these stats, Drew used 5 different scripts in 5 different languages. Awk, sh, C, go & python. Well, the C program isn't a script it's a test artifact. Drew must subscribe to the "best tool for the job" philosophy rather than the "use what you know" philosophy.

albertzeyer 6y ago

> Will security vulnerabilities in libraries that have been statically linked cause large or unmanagable updates? This is maybe not an issue for open source packages which are managed by your distribution package manager, assuming they update all the dependent packages once some library gets updated (which would lead to a lot more updates all the time). However, the maybe more critical issue is about other independently installed software, or maybe closed source software, where you will not automatically get an update once some library gets updated.

cosmojg 6y ago
> However, the maybe more critical issue is about other independently installed software, or maybe closed source software, where you will not automatically get an update once some library gets updated. If you care about security, you shouldn't be running closed source software anyway, at least not outside of a container. Statically linked binaries for open source software, containers for everything else, and you're good to go.

earthboundkid 6y ago

Say we have program X with dependency Y. X+Y is either dynamic or static. X can either have responsive maintainers or unresponsive maintainers. Y can either change to fix a bug or change to add a bug. (With Heartbleed, I remember our server was fine because we were on some ancient version of OpenSSL.) Here are the scenarios: - dynamic responsive remove bug: Positive/neutral. Team X would have done it anyway. - dynamic unresponsive remove bug: Positive. - dynamic responsive add bug: Negative. Team X will see the bug but only be able to passively warn users not to use Y version whatever. - dynamic unresponsive add bug: Negative. Users will be impacted and have to get Y to fix the error. - static responsive remove bug: Positive/neutral: Team X will incorporate the change from Y, although possibly somewhat slower (but safer). - static unresponsive remove bug: Negative. Users will have to fork X or goad them into incorportating the fix. - static responsive add bug: Positive. Users will not get the bad version of Y. - static unresponsive add bug: Positive. Users will not get the bad version of Y. Overall, dynamic is positive 1, neutral 1, negative 2, and static is positive 2, neutral 1, negative 1. Unless you can rule out Y adding bugs, static makes more sense. Dynamic is best if "unresponsive remove bug" is likely, but if X is unresponsive, maybe you should just leave X anyway.

petters 6y ago
Sure, but I think the most important reason people have in mind when they argue for dynamic linking is that they will receive upstream bugfixes. I don't think your eight cases are equally important. You are of course right, though, and writing it down like this can be useful.
- IshKebab 6y ago
  Why would you not receive upstream bug fixes with statically linked programs? Assuming you are using an Apt-style package manager then the statically linked program would be rebuilt and updated too. If you are not using an apt-style package manager then the program must include all of its dependencies (except ones that are guaranteed to be present on the platform, which is none on Linux and a few on Mac/Windows), and you will receive bug fixes when that program is updated, whether or not it uses static linking. Static/dynamic linking does not affect how likely you are to get bug fixes in any way as far as I can tell.

pengaru 6y ago

Back in the 90s we'd statically link the most frequently executed programs on busy servers for a significant performance boost. Dynamic linking is not a performance feature, it's a decoupling feature.

binarycrusader 6y ago
It is a performance feature if you are memory constrained. Shared libraries are “shared” for a reason. On a server with high multi-tenancy the savings can be significant.
- jeffbee 6y ago
  Not really. Dynlinking solves a memory resource problem we had in the 80s. These days the only people with the problem are the very smallest embedded systems. As for "high multi-tenancy" there's nobody out there with server occupancy as high as Google's (see the recent Borg scheduler traces for concrete data) and they statically link everything.
  - binarycrusader 6y ago
    Not really. Yes, really. Dynlinking solves a memory resource problem we had in the 80s. These days the only people with the problem are the very smallest embedded systems. Definitely not true. Every bit of memory that's not available that could be shared memory instead is a reduction in memory available for filesystem caches, etc. As for "high multi-tenancy" there's nobody out there with server occupancy as high as Google's (see the recent Borg scheduler traces for concrete data) and they statically link everything. Cloud vendor computing models are not generally the computing model of the rest of the world. Comparison to their environment is not relevant to the general populace. I worked for a "big iron" OEM vendor until late 2017 and the savings were definitely still significant then both for their customers and the vendor themselves. There are numerous benefits to shared linking, that doesn't mean it's always the appropriate solution, but is not correct to claim that there are no performance benefits. Especially on more memory-constrained consumer devices, the shared memory benefits of dynamic linking are still significantly beneficial.

sprash 6y ago

Static linking allows LTO with aggressive inlining and is therefore able to achieve far superior performance beyond just the startup time. Arguing that dynamic vs. static has better RAM utilization or not is pointless because nowadays we have plenty of RAM but single core performance is stagnating for almost a decade already. Moores law might give us more transistors but single thread performance is still more or less bound by the clock and transistor switching frequency. Sooner or later static linking will become the only way to move forward and the conveniences dynamic linking offers will not be worth the costs.

cryptonector 6y ago
> Arguing that dynamic vs. static has better RAM utilization or not is pointless because nowadays we have plenty of RAM ... Using more RAM means having lower cache hit ratios. If dynamic linking means using less RAM, you win. But it's not a clear-cut thing -- it will depend a lot on the surrounding ecosystem. In any case, for C, the problem with static linking is about semantics. Until those are fixed I'll be resolutely against static linking for C, and for everything else, well, do whatever performs best.
floatboth 6y ago
lol, you don't want aggressive inlining everywhere. Here's a good example. Firefox does even cross-language LTO for stuff that matters — all the DOM/CSS/etc components that are tightly bound together. But it happily allows you to dynamically link to a system libjpeg, and you won't lose any performance because the libxul<->libjpeg boundary is crossed.. about a couple times per downloaded jpeg. Heck, what would you even inline between the jpeg decoder and the browser? Absolutely nothing! The world at large is already using static linking where it matters.. e.g. with C++, header-only libraries and anything that uses templates results in a lot of static linking. Even if you link to libboost_something.so, you probably already have most of that something inlined into your code because templates :)
ndesaulniers 6y ago
static linking != LTO LTO requires source, or at least IR. Static linking involves object files; there's been a loss of fidelity at that point that prevents inlining.

butterisgood 6y ago

ldd won’t get you all the programs that dynamically load with dlopen... but these are still interesting results.

sitzkrieg 6y ago

these benchmarks dont seem to take into account page cache

gok 6y ago

Why stop there? Package every binary in its own container, too! Most programs only use a few system calls, so you're not really getting anything by sharing a single kernel for the entire system.

Shorel 6y ago
Seems to be the general trend. A virtualized kernel per service.
kortex 6y ago
That's literally what RancherOS does. ls is a container. Blew my mind the first time I saw it. But they take the "cattle not pets" approach pretty much to its logical extreme (I suspect that the company's name arises from this metaphor) It makes ops so liquid and convenient. I love it. To really push this idea (and microkernel) to its logical extreme, it would be cool to see a null-kernel. You have socket drivers, that's it. Everything is either a network call or some sort of IPC. You might still need something to handle paging, though, and obviously your "peripherals" like storage would need to be more conventional in nature.
- pjmlp 6y ago
  It makes sense from security point of view, but it definitely is much more resource intensive. Imagine something like InteliJ or Eclipse, where every single IDE plugin is its own process doing IPC.
yencabulator 6y ago
All containers on a single host very much share the same kernel. You're confused about virtualization vs containers.

pengaru 6y ago

Static linking has been such a nuisance for the libSDL folks that they implemented dynamic loading of itself [0], controlled via an environment variable, as an escape hatch from executables w/libSDL linked statically. It's understandable that games, especially proprietary ones, distribute statically-linked binaries ensuring any third-party dependencies will be present and be a compatible version. But the value of that decision tends to diminish with time, as those external dependencies are frequently the pieces interfacing with the system/outside world, which keeps changing, leaving such dependencies behind to atrophy at best or become vulnerable/incompatible/broken at worst. I don't personally think it makes sense to approach this so dogmatically. Static linking makes sense in the right circumstances, so does dynamic linking. For general-purpose operating systems, it seems obvious to me that you'd want most higher-order userspace programs dynamically linked. I want my openssl updates to touch a single library and affect all installed ssl-using programs, for example. Having said that, I do wish the average linux distro still statically linked everything in /bin and /sbin. It was nice to still be able to administrate the system even when the dynamic libraries were hosed. At some point it was changed to just a single static binary; sln for static ln IIRC, assuming you'd be able to fix your dynamic libraries with some symlinks if they were broken, if you happened to have a shell running and could navigate using just builtins. It was already an impossible situation, but even that seems to be gone nowadays. It's a more nuanced issue, taking an "everything dynamically linked!" or "everything statically linked!" approach strikes me as just another form of ignorant extremism. [0] http://hg.libsdl.org/SDL/file/2fabbbee604c/src/dynapi/SDL_dynapi.c#l292 http://hg.libsdl.org/SDL/file/2fabbbee604c/src/dynapi/SDL_dy...

jart 6y ago
There's nothing wrong with taking a principled approach to builds. For example, it's not just a question of static or dynamic. Linking of vendored/static/pinned/etc. sources/artifacts/etc. in general comes with the benefit of hermeticity. That enables tools to work better, since it gives them perfect knowledge of the things that exist. It also entails a certain degree of responsibility for writing your own scripts that can generate automated alerts should issues arise in the upstream sources.
cryptonector 6y ago

bjourne 6y ago

Either something is wrong with the testing script or my computer is way faster than I thought: ./test.sh | awk 'BEGIN { sum = 0 } { sum += $2-$1 } END { print sum / NR }' -698915

joosters 6y ago

Over half of your libraries are used by fewer than 0.1% of your executables That's a very misleading reference and graph. First of all, what did they expect to find? As you add more executables, of course the % usage of a library will decrease. e.g. say I have a networking library on my computer, and, in a perfect world, all my installed network tools link against it. But now I install Gnome, and my machine has hundreds more binaries. Not all of the binaries will do networking stuff, so the % usage of the networking library goes down. But that doesn't mean that the networking library is not being shared as well as it could be. A much better metric would be to count, for each shared library on a machine, the number of programs that link against it. If only one program uses a shared library, then that means the 'shared-ness' is not being used. If more than one program use it, then the library is being effectively shared. But the actual count, whether it is 200 users or 20, doesn't mean anything more. That's why comparing all libraries against libc's usage shows nothing useful.

jjoonathan 6y ago
Yeah. Loading the GUI widget toolkit once rather than 30 times is pretty nifty.
cryptonector 6y ago
Even better would be to measure sharing in the totality of programs you use directly and indirectly from boot.

saagarjha 6y ago

As always, static and dynamic linking both have their advantages and drawbacks. The usual arguments for dynamic linking around brought up in the article, and as others have mentioned here, the analysis is a bit lacking so the conclusions aren't generally true. Static linking has its own, fairly straightforwards benefits as well. It's no surprise that those who push one or the other usually do so because of their specific needs. Sometimes we even see some interesting hybrid solutions: one recent one is Apple introducing dyld shared caches on macOS, which (while being a pain to reverse engineer) are basically all the system (dynamic) libraries all statically linked together and presented dynamically, with some linker tricks to make it appear seamless. Likewise, a lot of statically linked binaries are only partially statically linked, still using things like libc or graphics libraries. The moral really is to try both and pick whichever one is the one that's better for your use case, and perhaps even consider a mix of both to give you the most flexibility in which tradeoffs you'd like to make.

cryptonector 6y ago
Specifically for C, static linking is a trap due to its poor semantics. Don't do it. Or fix the C link-editors, then static linking C will be fine.
- sigjuice 6y ago
  What would be examples of poor static linking semantics? Thanks!
  - cryptonector 6y ago
    https://news.ycombinator.com/item?id=23656173 https://news.ycombinator.com/item?id=23656173
comex 6y ago
Nitpick: macOS has used dyld shared caches for over a decade. The recent change is just to remove the original copies of the libraries on disk.

andyjpb 6y ago

This is an interesting analysis. For my part (anecdata), in my /usr/bin (Debian) I have 2,956 files under 9MiB in size, 1 of 13MiB and one of 39MiB. Most of the files are (much) under 1.0MiB. On the other hand, I have a three statically linked binaries for CloudFoundry in my home directory. One for Linux, one for MacOS and one for Windows. They are each between 24MiB and 27MiB each.

dan-robertson 6y ago

I’m curious about the supposed memory advantages of dynamic linking: on average how many different executables share each page of memory? What about when memory is in high demand? How high does that average become? What is the probability that a page of a shared library is already in memory (cache or otherwise) when it needs to be loaded, and in particular the probability that it is there because another program loaded it). My guess is that apart from eg libc, the average is pretty low (ie 1 for the pages that aren’t free).

saagarjha 6y ago
I think this depends heavily on your platform and how often libraries are reused. On macOS, for example, most libraries aside from libc are loaded R/O or CoW quite literally hundreds of times, because every app shares AppKit and WebKit and Security and the dozens of other platform frameworks (and their private dependencies!) that are basically "free" to use and ship with the system and so have very high adoption. On more "coherent" Linux distributions I'm sure things like GTK, glib, OpenSSL, zlib are used by a lot of things too. Sure, there's going to be a lot of one-off dynamic libraries too, but there's a lot of duplication with the popular dependencies and then a long tail.

pwdisswordfish2 6y ago

With NetBSD, it is relatively easy to compile the entire system as static. Is it easy to do this with Arch Linux? (Maybe Void Linux?)

zelly 6y ago

Dynamic linking provides encapsulation and security benefits. An application with a statically linked OpenSSL can be vulnerable if a CVE comes out for that version of OpenSSL, whereas a dynamically linked OpenSSL could be patched immediately without recompilation (which may be impossible if the software is proprietary). The vendor of the shared library can update the implementation without requiring all downstream consumers to recompile. This should not be downplayed. Pure static linking makes sense when you are deploying code you control onto an environment you control. While I like ease of deploying Go, where binaries are one big blob that just works, it makes it closer to the Java style than the UNIX modular-tool-does-one-thing-well tradition.

vlovich123 6y ago
From distributions where you build everything from source (most cloud providers today, mobile phone operators, etc), such advantages are purely theoretical & never play out in practice. In such environments you're pushing out updates of the entire system on whatever your regular cadence is. Additionally, because the old library is still mapped & running you have to know to restart all the processes that have the old version linked in. Not an easy task. About the only place it kind of matters is Linux distros where you are deploying closed source binaries linked against system libraries. I'm not sure that's a significant use-case. The only performance benefit is memory. If you have a core infrastructure library that's widely shared (e.g. libopenssl) then it can have some benefit. In practice I much prefer the microservice model with a formal IPC API. Then the SW update is trivial to fix the exploit - just kill the 1 process providing the service.
- zelly 6y ago
  Suppose 100 processes on the same box call the same function in glibc. If each process had its own redundant version of it mapped into memory, then that's 100x more cache misses compared to them all sharing the same page. I get that only a handful of libraries are actually used, but that's a very fat tail and the performance will be very bad if that handful of libraries are always statically linked. This may be less important if the program is in a container (isn't sharing anyway) or launched as an AWS Lambda throwaway. I dunno, maybe dynamic linking isn't relevant anymore now that most Linux programs are meant to run in containers on an expensive cloud with tons of memory. > In practice I much prefer the microservice model with a formal IPC API. This converts a simple and solved problem, sharing memory, to a client-server distributed systems problem. There is no need. It's all on the same box.

ncmncm 6y ago

Paging Laura Creighton! Your time is now.

cryptonector 6y ago
What should I search for to get a clue as to what you're referring to?
- ncmncm 6y ago
  Her name, and USENET. She was the most vocal opponent of the move to dynamic-linked libraries, back when it happened. Her expectation was that programs that once worked would stop working as the libraries rotted.

Shorel 6y ago

I think the elephant in the room here is things like Flatpack and Snap, which are basically ugly bastardized versions of static linking.

eeereerews 6y ago
Ugly maybe, but they're do kinda combine the advantages of both. They have the portability and reproducibility advantages of static linking, while still letting you take them apart and change the dependencies like with dynamic linking if you really want to.
- Shorel 6y ago
  And the disadvantages of both. These being the huge loading times (even for a calculator!) and the disk space required.

cycomanic 6y ago

The first graph looks like a power law. It's fascinating how many processes follow power laws. As a side note, you are almost always better off (also on this case) by plotting power laws on a log scale.

ridiculous_fish 6y ago

This is a crazy conversation! Dynamic linking allows the system to decide the UI. Static linking means that the UI cannot evolve. Imagine statically linking UIKit or Android's UI library!

saagarjha 6y ago
Yeah but then you have CFLinkedOnOrAfter to negate that ;)
thu2111 6y ago
UI libraries tend to evolve by introducing new APIs or modes, because changing UI implementations has a habit of breaking apps. For example, on Android even the switch to hw accelerated rendering was a mode app devs had to opt in to, that didn't even change the look!

mrich 6y ago

During development of a large system static linking is hell. The final link just takes too long if you are only working on one object file. Also during debugging you have to load all symbols in gdb, which does not scale. With dynamic linking you only load the debug info for the library you are working on.

adrianN 6y ago
GDB handles binaries in the hundreds of megabytes fairly well.

tsimionescu 6y ago

> The total download cost to upgrade all binaries on my system which were affected by CVEs in 2019 is 3.8 GiB. This is reduced to 1.0 GiB if you eliminate glibc. The upgrade problem has almost nothing to do with download size. The real problem is that you have > 100 binaries which depend on those libraries, and instead of having the library authors go and update the library, you need each team responsible for one or more binaries to go and take the new library and release a new version of their binary. And then, when you want to check if your system is safe from Heartbleed, instead of checking if you have libopenssl > 1.0.1g, you need to check if bin1 > 1.2.56 or > 0.6.89h, bin2 > 5.76.1, or > 4.6.215,... bin100 > 1.67.89. And of course, if one of them does NOT have a newer version compiled with the patched library, you need to fix it yourself, and maintain a patched version of the binary. Assuming that you even know that binary had been linked to the vulnerable library.

danieldk 6y ago
The real problem is that you have > 100 binaries which depend on those libraries, and instead of having the library authors go and update the library, you need each team responsible for one or more binaries to go and take the new library and release a new version of their binary. This is a solvable problem. Package managers such as Nix and Guix rebuild packages if any of their transitive dependencies have been changed. The difficult part are now language-specific ecosystems that use lock files to lock all their dependencies. Traditional C/C++ programs, either statically linked or dynamically linked, have a dependency graph such that e.g. each program that uses OpenSSL has the same package definition in their transitive dependencies. However, e.g. Rust programs may have different versions of the same crate locked. (There are solutions to that, but there is still work to be done.)
- tsimionescu 6y ago
  I was talking about finding vulnerabilities on end-user systems and servers. I don't know about others, but I don't generally keep compiler tool chains for C, C++, Java, Go and maybe 1-2 others on my systems and on my servers in case I need to rebuild all of my binaries.

nabla9 6y ago

Would this distinction would go largely away if OS would have method to share read only memory pages or larger regions of memory between all processes? I suspect you would need to compute signatures for static libraries for binaries located /bin/, /lib, ... to make load times faster.

enriquto 6y ago

Static executables are a sweet spot between the craziness of dynamic dependencies (where we come from) and the idiocy of shipping a whole virtual machine for each program (where we seem to be going to). If we could just stop at that spot!

AnIdiotOnTheNet 6y ago
I kindof agree, but I think the real sweet spot is that system libraries[0] should be dynamically linked and everything else should be static. For Linux though, where the very concept of a standardized base system is loathed, static the sweet spot. [0] libraries that are likely to be used by a large number of applications like the GUI libraries, kernel interfaces, networking, encryption, etc.

perryizgr8 6y ago

The main problem I have with dynamic loading is that my app crashes without any sensible error message on runtime, if I misspelled a single symbol.