22 ms·

“Static Linking Considered Harmful” Considered Harmful

sharikous 5y ago
Correct me if I am wrong but OpenBSD introduced the constraint that syscalls must come from a specific section of memory, that assigned to their libc. In a case like that static linking is not only unstable, as in Windows or macOS, but also impossible. And since that has security benefits we will probably see it being migrated to other platforms in the near future.
dhsysusbsjsi 5y ago
Personal end user perspective: troubleshooting dylib errors is the most common cause of wanting to throw the computer out the window and quit programming.
- cratermoon 5y ago
  See also: DLL Hell in Windows and, more generally Dependency Hell. Dynamic linking errors are a subset of general dependency problems.
  - spacechild1 5y ago
    As others have already mentioned elsewhere, DLL hell hasn't really been a thing anymore on Windows for years. I personally can remember it, but that's more than 15 years ago.
    - cratermoon 5y ago
      It's true, but only because every app essentially isolates itself from the system and carries around its own dependency bundle. I guess that's kind of what Docker containers are, so it's hard to be critical.
- eqvinox 5y ago
  "end user perspective" + "quit programming"? Do you troubleshoot dylib errors on your regular end user software and just happen to be a programmer unrelated to that? (If you're troubleshooting dylib errors on things you're developing, that's not an end user perspective?)
  - dizzy3gg 5y ago
    End user of a linker?
    - rewma 5y ago
      > End user of a linker? I'd argue that if you're not packaging your software or testing your software's dependencies, either you're doing something extremely exotic that lies far outside anyone's happy path or "dylib error" should not even be a keyword in your vocabulary.
      - alisonkisk 5y ago
        The windows version, DLL Hell, is well known to non programmers.
      - pjc50 5y ago
        Roughly the same thing happens on Windows and is called "DLL hell", occasionally witnessed by end users.
        rewma 5y ago
        DLL Hell ceased to be a practical concern over a decade ago, particularly given that Windows provides tight control over its dynamic linking search order. https://docs.microsoft.com/en-us/windows/win32/dlls/dynamic-link-library-search-order https://docs.microsoft.com/en-us/windows/win32/dlls/dynamic-... DLL Hell is not a linking problem, it's a packaging problem.
  - gbear605 5y ago
    There’s a middle level of “troubleshooting dylib errors on open source software I’m installing for personal use because I’m a masochist who isn’t happy with closed source software”
KronisLV 5y ago
Why are we still developing software like we used to 40 years ago? Why not just import external dependencies into your project at a function/class level rather than at a package one: if you only use FooPackage.BarClass and FooPackage.bazMethod(), you should be able to choose to make your project depend on just those two things, ideally in a completely transparent way by your IDE. Then having to manage the full scope of packages and their versions becomes less relevant, because the IDE can check whether a new version has been released in the package manager of choice, check whether those two things have changed and if they have, then demand that the developer have a look at the code changes to refactor code if necessary, otherwise not requiring any action on their part. Furthermore, if you ever need to migrate to a different package or even rewrite the functionality itself, you could just look at the few signatures that you use, rather than having to remove the package entirely and see what breaks in your IDE. Why can't we have itemized lists of everything that's called and everything that we depend on, as opposed to the abstraction of entire packages? Better yet, why even depend on binary blobs or hidden code (that's ignored by your IDE) in the first place? Why not just download the source for every package that you use and upon updates be able to review the code changes to the package much like a regular Git diff? Of course, this would probably require getting rid of reflection and other forms of dynamic code, which i fully support, since those have never been good for much in the first place and only destroy any hopes of determinism and full control flow analysis. As for the possible counterargument of this being hard to do: it wouldn't really be with more granular packages. Instead of trying to shove an ecosystem inside of a single package, why not split it into 10-20 bits of reusable code instead? Smaller packages, which would be easier to review and manage. Context: i dislike how Spring and Spring Boot in Java force a huge ecosystem of fragile dependencies upon you, with their reflection which ensures that your apps will break at runtime, for example, when moving from Spring Boot 1.5 to 2.0. Furthermore, in the JS world, the node_modules folders are probably 10-1000x too large for what's actually necessary to display a webpage with some interactive behaviour. Disclaimer: i have no delusions about any of the above being technically feasible right now. Perhaps "Why?" would be a good question to ask. In my eyes, perhaps the industry is too set in its current ways and as long as we don't have languages and entire ecosystems that approach established practices from a wildly different angle, no one can actually contest the decades of already built packages and tools. On the bright side, WireGuard kind of displaced OpenVPN somewhat and there are numerous benefits to it being smaller, which is a good example of getting rid of bloated software. Furthermore, Nix may or may not do that to alternative approaches to package management, but its usage still remains really low. In my eyes, the only long term solution is to be able to tell exactly what's necessary for your code to build and work, and to test every dependency update that comes out against this automated process. Static dependencies but at the speed of updates of dynamic dependencies. Then you'd just have to wait for a day until the devs fix the newest regressions of a new dependency release before getting a new statically linked app, as opposed to using dynamic linking and finding that some dependency breaks your app as it happens in prod.
ChrisSD 5y ago
> However, there is one big elephant in the room: right now, there are a lot of platforms that do not fully support static linking because you have to dynamically link libc or other system interfaces. The only one I know of that supports static linking is Linux. > The ironic thing is that operating systems only supported a stable syscall ABI, rather than requiring programs to dynamically link system libraries, they would have less problems with ABI breaks. > With a statically-linked binary, you can copy it to another machine with a Linux that supports all of the needed syscalls, and it will just run. I think this is a misunderstanding. With Linux there is a stable interface at the kernel boundary. Other OSes have their stable interfaces in user space. The difference is mostly irrelevant in this context. In one case you dynamically call into the kernel, in another case you dynamically link a user space library. Either way the code you're calling isn't static but the interface is.
- amelius 5y ago
  I'm distributing my code as a bootable USB drive.
- CyberShadow 5y ago
  There is a distinction between a fully-static binary and a non-fully-static one, which is what I think the article refers to. A fully static binary will not need to call the dynamic linker at all, and tools such as "file" or "ldd" will identify such binaries differently. A fully static binary will also run even in the complete absence of any userspace support for its architecture (assuming the hardware and kernel do support it) - e.g. a static AArch64 binary will run on a 32-bit ARM distribution, if the CPU supports AArch64.
  - ChrisSD 5y ago
    True, however... > assuming the hardware and kernel do support it Assuming kernel support is doing a lot of heavy lifting there. To put it another way, why is a syscall better than any other stable ABI? If both kernel and stable library are distributed together then why should it be considered different?
Wowfunhappy 5y ago
Are there any Linux distros that actually distribute (most) software as (mostly) static binaries? Right now, the only way to do it seems to be compiling everything myself...
- lrem 5y ago
  Why would you want that?
  - blablabla123 5y ago
    I think programs in /bin used to be usually statically linked. When updating the glibc for instance, this makes the change far more doable. In general dealing with static binaries is much less pain, especially when they haven't been compiled locally/by the distributor for this particular version of the distribution.
    - drivebycomment 5y ago
      /sbin used to be statically linked on unix. The entire point of /sbin is the independence from almost everything else so that you can rely on them even when the system boots up in a weird state.
  - Wowfunhappy 5y ago
    Simplicity. If everything is statically linked, then updating package A can never break package B. As I see it, the entire world of Docker containers exists primarily to solve the problem that static linking already solved. Something like Nix would work as well, and I think I would love Nix if I knew how to use it. But I don't, and I don't really have the time to learn.
    - lrem 5y ago
      I don't think that makes a lot of sense in a distro. If package A gets an update, it means a sufficiently bad bug was discovered in it. If it happens to be a library, you want to make sure you update every package B that uses it. Which, conveniently, the distro solves for you too.
3np 5y ago
""Static Linking Considered Harmful" Considered Harmful" Considered Harmful: This is the blog version of Re: Re: Fwd: Re: in email and has to stop here. See also https://meyerweb.com/eric/comment/chech.html https://meyerweb.com/eric/comment/chech.html
- Brian_K_White 5y ago
  Incorrect. Ok fine, explaination: "Re:" are not art. "Re:" are not added deliberately and for intentional artistic effect, and are not references (despite the literal) or homages.
- ghoward 5y ago
  Author here. See my reply to https://meyerweb.com/eric/comment/chech.html https://meyerweb.com/eric/comment/chech.html at https://news.ycombinator.com/item?id=28737279 https://news.ycombinator.com/item?id=28737279 .
  - 3np 5y ago
    I was mostly being facetious and wrote that in jest - with a title like that you should be prepared for some OT banter ;)
davidwf 5y ago
This is very situational, but I have recently been part of a project that does a lot of C server-side development and we have found that static linking our non-glibc dependencies has really improved our developer experience. Using ceedling's dependency plugin[1] and producing a single "statically" linked library has made our C development much closer to using a language with a more modern package manager. Don't get me wrong, if I was trying to distribute binaries to machines I didn't control I'd definitely be willing to invest in the Linux packaging "fun", but for a server-side application it's been a good choice for our team overall. [1] https://github.com/ThrowTheSwitch/Ceedling/tree/master/plugins/dependencies https://github.com/ThrowTheSwitch/Ceedling/tree/master/plugi...
- PaulDavisThe1st 5y ago
  Yes and no. In Ardour, which has a substantial dependency tree (80+ libraries), if we statically link, the edit/compile/debug cycle becomes incredibly bogged down by static linkage. It takes a long time (even with lld) to complete the link step. If we use shared libraries/dynamic linkage, the edit/compile/debug cycle is nice and fast, but application startup is a bit slower (not by as much as static linkage is slower, however). For users, it would be better if the application was statically linked, at least in terms of startup cost. But because developers do the edit/compile/debug cycle much, much more often than users start the application, we opt for dynamic linkage.
  - xyzzyz 5y ago
    You can dynamically link during development, and then ship statically linked binary (with LTO etc). That’s what Chrome does, for example.
    - PaulDavisThe1st 5y ago
      It's not that simple for us (we have tried this).
ris 5y ago
I eagerly await some concrete implementations of these new ideas. In the meantime I'm waiting for the first big vulnerability in a widely used golang or rust library to see if downstream projects pinning it also release CVEs as they should (which would probably DDoS the CVE system from the resulting avalanche) or they quietly bump it and move on (in which case their users won't get the nudge to upgrade). This is where someone says "well you should just always be running the latest version of everything", which is of course infeasible on a real life system with thousands of installed packages. And it's not that I don't completely sympathize or understand the advantages of static linking from a developer's point of view - you don't have to sell it to me there.
- DaiPlusPlus 5y ago
  The design of the Go build and packages system kinda renders that point moot because the expectation is that provided you follow Go's guidelines then a Go application should always be easily recompilable, and all external dependencies should always be distributed in source-form, not as binaries (e.g. https://github.com/golang/go/issues/2775 https://github.com/golang/go/issues/2775 ).
  - Macha 5y ago
    Right, but if every application did distribute as source and could just be recompiled, there would be fewer complaints about the difficulties distributing Linux binaries
  - Zababa 5y ago
    > provided you follow Go's guidelines then a Go application should always be easily recompilable, and all external dependencies should always be distributed in source-form, not as binaries So it's a human solution, and not a technical one, which means that it can and will fail.
    - DaiPlusPlus 5y ago
      That's like saying condoms are ineffective because you included people too lazy to put them on in the first place in your calculations. Provided you (and your org) use Go as Google intends then that isn't an issue.
flohofwoe 5y ago
Unfortunately not even Linux allows full static linking (as far as I'm aware at least) as soon as libraries like OpenGL are involved. Dynamic linking with core system libraries which are guaranteed to exist(!) is fine though, for everything else dynamic linking mostly has downsides. Also: it's only Linux where this is an issue, because it's nearly impossible to build small self-contained executables which run on another Linux machine, thanks to glibc's versioning mess (at least there's MUSL as an alternative to create portable command line tools).
- Macha 5y ago
  Didn't Go have to throw in the towel on not dynamically linking libc on macOS as there's no guarantee of syscall stability at all?
  - flohofwoe 5y ago
    Don't know, I only know that I can build a macOS command line tool on one machine, and run it on a different machine without problems, while on Linux it's very easy to run into the glibc version compatibility problem.
    - onei 5y ago
      I always understood the glibc versioning problem to be solved by just building on the oldest common ancestor. If you have rhel5 through 8, you build on rhel5 and move on. If you depend on more than glibc... Well rhel5 is less fun.
      - ptsneves 5y ago
        Indeed, the only difference is that you might be using less optimal system calls, but there is no reason to be blocked and not target ancient kernels. I linked to 2.6 this year and everything ran fine.
      - ghoward 5y ago
        Author here. Unfortunately, building on the oldest common ancestor will still not save you from ABI breaks. An example is [1]. If the size of `intmax_t` changes between that oldest ancestor and the machines you are running on, get ready for pain. [1]: https://thephd.dev/intmax_t-hell-c++-c https://thephd.dev/intmax_t-hell-c++-c
deleted 5y ago
[deleted]
deleted 5y ago
[deleted]
0xbadcafebee 5y ago
I wish I had three hours to waste on why almost all of these points are either innacurate or willfully misleading. But I don't, so I'll just point out the single biggest reason why static linking is harmful. Encryption libraries. If OpenSSL, or LibreSSL, or Go's encryption modules, or any gigantic encryption library has a vulnerability, you basically have to recompile, distribute, and then have your users download, every single god damn networking program. But it's worse than that. Because everyone wants to ship a statically compiled Go app, nobody packages for the distros. So now every user needs to go track down where they downloaded each of their static Go apps, safely download the update, verify its checksum or signature via a 3rd party, and upgrade the app. That is a logistical nightmare. Not only are apps going to remain vulnerable for way, way longer, more users will be compromised by phishing and other attacks because there's no package manager to safely automate all these updates (distros sign and verify packages for you). I've actually written a Shell quasi-package manager/installer just for statically compiled apps (cliv) so in theory that problem could be somewhat solved... But nobody even knows about that program, so the manual update problem still stands.
- AshamedCaptain 5y ago
  If you ever statically link OpenSSL, your binary now has a builtin expiration date. You are doing your users a disservice.
  - marcosdumay 5y ago
    To be fair, if you dynamically link to OpenSSL, your binary has a builtin expiration date too, and it's very likely the same date.
    - nly 5y ago
      OpenSSL ABI has been fairly stable within a given series. Something compiled against 1.1.0a (Sep 2016) has a fair chance of running on a system that has 1.1.0l (Aug 2021). 5 years of binary compatibility is nothing to be sniffed at for a C library. https://abi-laboratory.pro/index.php?view=timeline&l=openssl https://abi-laboratory.pro/index.php?view=timeline&l=openssl
rascul 5y ago
According to Linus, "Shared libraries are not a good thing in general". https://lore.kernel.org/lkml/CAHk-=whs8QZf3YnifdLv57+FhBi5_WeNTG1B-suOES=RcUSmQg@mail.gmail.com/ https://lore.kernel.org/lkml/CAHk-=whs8QZf3YnifdLv57+FhBi5_W...
- ghoward 5y ago
  Author here. I quote that email twice in my post. I decided not to quote that part of the email just in case it might inflame the debate. But yes, he did say that.
PaulDavisThe1st 5y ago
> I don’t think Ulrich Drepper could have foreseen all of the problems with DLL Hell (though there were signs on Windows), Drepper's article is from 2006. "DLL Hell" was beyond fully understood at that point, certainly by Drepper and large numbers of other well informed programmers.
- ghoward 5y ago
  Author here. In everything in the post, I attempted to give Drepper the benefit of the doubt, though there are cases where I don't want to since he sounds (to me) paternalistic in the way Apple and Microsoft do. So perhaps he understood, but I didn't want to assume that, especially since it might inflame the discussion, which is exactly what I was trying to avoid doing (again).
  - PaulDavisThe1st 5y ago
    DLL Hell was well understood in the mid-90s. I think you need to give him more benefit of the doubt on this point.
    - ghoward 5y ago
      I'm having a hard time understanding because if DLL Hell was well understood by 2006, it seems I should have given him less benefit of the doubt, but you say I should give him more. Apologies.
    - torstenvl 5y ago
      If someone glosses over an issue, it's either innocent (didn't know about it, didn't understand it) or deceptive (intentionally ignored it). Giving Drepper the benefit of the doubt here means assuming he didn't know or understand the problem he failed to adequately address.
      - PaulDavisThe1st 5y ago
        Here is most of the relevant text from TFA: > Ulrich Drepper claims, rightly, that you only need to update a shared library to apply security fixes to all executables that rely on that library. > This is a good thing! It is also a good vision. > Unfortunately, it is only a vision because it’s not the whole story. > As of yet, there is no way to automatically determine if the ABI or API of a shared library changed between updates. This is called DLL Hell. The final two words are a link to the wikipedia page on "DLL Hell". The final sentence of the intro section there states: > DLL Hell is the Windows ecosystem-specific form of the general concept dependency hell. That is, TFA defines "DLL Hell" using a wikipedia page that explicitly states that it is a Windows-specific version of a more general problem. Drepper's article from 2006 fully tackles the general problem, albeit without (as the TFA puts it): "a way to automatically determine if the ABI or API of a shared library changed between updates." Drepper's primary suggestion there is a naming/versioning scheme which describes specifically whether the ABI/API has changed in ways that matter for shared linkage. It's not automatic - that part is true. But is is a solution, widely used in Linux and *nix more broadly. Did Drepper address the Windows specific parts of "DLL Hell". He did not, but that's because "DLL Hell" is Windows specific (as other comments here fully clarify), and he was not writing about how to fix the full scope of that particular nightmare. He likely understood "DLL Hell" as well as anyone, nevertheless.
rwmj 5y ago
I read this interesting article recently about all the security problems that static linking, bundling and pinning cause for Linux distributions: https://blogs.gentoo.org/mgorny/2021/02/19/the-modern-packagers-security-nightmare/ https://blogs.gentoo.org/mgorny/2021/02/19/the-modern-packag...
- Aqueous 5y ago
  But should distributing and ensuring the security of software that users install on their own systems even be the responsibility of operating system maintainers?
  - eqvinox 5y ago
    Yes. If I installed software through my OS package manager, where else would I get security updates from?!
    - Aqueous 5y ago
      Like on every other major OS the publisher of those applications. And I mean they should be distributing the applications as well. Not the OS publishers.
  - rwmj 5y ago
    Isn't it obvious? Or do you suggest that end users should be responsible for checking and patching every package on their system? Given that Red Hat has a large department just to do that for RHEL, I can't imagine where end users would begin.
    - Aqueous 5y ago
      No, I’m suggesting that the authors of the applications are responsible for patching and distributing their own application like on every other major OS.
- ghoward 5y ago
  Author here, and I am also a Gentoo user on my daily driver. I read that article when it came out, and it felt disingenuous. mgorny helps maintain Gentoo, which of all distributions, does rebuild software as necessary. It would be a simple change to portage to rebuild all dependencies of a library when that library is updated, regardless of if the user uses static linking or not. In fact, there is a project to have a statically-linked musl-based Gentoo. So it's possible and (I would argue) would be an easy change. mgorny would not have to worry about much because he could, in essence, treat all libraries as dynamic libraries. The only pain to be felt would be by users who would see their build times increase (to rebuild dependents). As a user, I would happily take that tradeoff. I already build my browser, which takes 5-8 hours; I'll happily rebuild dependents of updated libraries.
api 5y ago
My personal view is that only standard core system libraries should be dynamically linked. Anything that is not totally ubiquitous should be static. That seems to be the Go and Rust default approach, or close to it. Link libc and such, but build in more obscure things. The idea of dynamically linking everything doesn’t scale. It asks too much of distribution maintainers and makes them the choke point for every little upgrade.
- PaulDavisThe1st 5y ago
  So, you link against a GUI toolkit that can be used to display images. Images in many different formats, including some that haven't been used in more than a decade and are unlikely to ever be encountered. Do you want the GUI toolkit linkage to automatically include every object module for every possible image file format in your static image, or do you want the GUI toolkit to use runtime dynamic linkage (explicit or implicit) to import only the modules required for the image formats actually encountered?
  - TillE 5y ago
    If I'm distributing binaries as a commercial product, I definitely want the former.
    - PaulDavisThe1st 5y ago
      There are many "commercial products" that run on Linux whose installation includes various *.so files that will be dynamically linked at run time. Are you saying this is a bad idea?
    - TeMPOraL 5y ago
      What about formats that don't exist yet? What about giving you the ability to supply your own encoder/decoder for some niche format you're using?
- josefx 5y ago
  > That seems to be the Go and Rust default approach I think Go tried to go fully static but ran into problems as most mainstream OSes except for Linux do not provide a stable system call interface, so applications have to link at least one dynamic library. The Linux system call interface is stable, but expects a C style stack layout and that also bit the Go devs. multiple times. So you can probably go fully static if you are only targeting Linux.
tinus_hn 5y ago
Considered harmful essays considered harmful: https://meyerweb.com/eric/comment/chech.html https://meyerweb.com/eric/comment/chech.html
- colejohnson66 5y ago
  I was wondering when something like this would be written. We’ve come full circle.
- ghoward 5y ago
  Author here. I read that essay a while ago and reread it before publishing my post. I don't think the author makes a good case. They make bare assertions without examples and without evidence, so it's unconvincing. For example, they say that a "Considered Harmful" essay: > often serves to inflame whatever debate is in progress, and thus makes it that much harder for a solution to be found through any means. But this could be true of any essay. More to the point, I think that the tone of the essay or post matters more. Case in point, my "Dynamic Linking Needs to Die" post probably inflames the debate more than this new "Considered Harmful" post. Maybe "Considered Harmful" essays have a reputation for inflaming the debate, and there's an argument to be made for that. But I personally don't think my post is one that follows that reputation. Also, because I was writing a post about a pre-existing "Considered Harmful" essay, it was natural to use it, especially since I needed to make it clear that my previous post is harmful too.
  - jasode 5y ago
    >Maybe "Considered Harmful" essays have a reputation for inflaming the debate, and there's an argument to be made for that. But I personally don't think my post is one that follows that reputation. Neither your post title nor your content bothered me but just as fyi... some people are very irritated by "snowclones". My previous comment with various examples: https://news.ycombinator.com/item?id=19071532 https://news.ycombinator.com/item?id=19071532
    - ghoward 5y ago
      Yeah, I suspected people might be. But since I was answering a "Considered Harmful" essay, it was too easy. Eh, oh well.
gnufx 5y ago
There are examples of the sort of reason I want dynamic linking under https://news.ycombinator.com/item?id=28734875 https://news.ycombinator.com/item?id=28734875 and you also really want it for profiling/tracing, and possibly debugging, e.g. in HPC with the MPI PMPI mechanism. libabigail is one tool for checking that a library's ELF versioning hasn't lied about ABI compatibility when you're building packages, for instance. Unfortunately people typically don't follow Drepper's advice (as far I remember the article) on building shared libraries, particularly using symbol versioning a la glibc.
- ghoward 5y ago
  Author here. Yeah, there are reasons to use dynamic linking, but I'm still not sure why dynamic linking gives you better profiling and tracing. > Unfortunately people typically don't follow Drepper's advice (as far I remember the article) on building shared libraries, particularly using symbol versioning a la glibc. I mention this in the post, but I quoted another person, Ian Lance Taylor, who wrote the gold linker, as saying that symbol combining, a portion of symbol versioning, is an unsolved problem. Even if developers don't follow Drepper's advice, I wouldn't put the blame on them because of that fact; I would argue that it's more accurate to say they can't really follow his advice.
  - dchapp 5y ago
    > Yeah, there are reasons to use dynamic linking, but I'm still not sure why dynamic linking gives you better profiling and tracing. It's not so much that the tracing becomes better, but that it becomes feasible at all. Two specific situations come to mind, both MPI-adjacent. (1) Running a PMPI-based tool on code you can't recompile yourself (e.g., you need a Q clearance to see the source, but not to actually execute it--weird I know, but not that uncommon in the DOE labs.); and (2) running multiple PMPI-based tools simultaneously which are composed at runtime via PnMPI.
    - ghoward 5y ago
      I addressed these arguments. I think that if you don't have access to the source, it's a security problem. That includes not having access to the source because of security clearances. As far as I am concerned, your superiors in the DOE are adversaries to you and your machine. Other people will have different viewpoints on that, and that's fine.
ghoward 5y ago
Author here. I don't know why my own submission of this [1] didn't catch and this one did. Oh well. XD I'm open to answering questions, and I'll be hanging in the thread. [1]: https://news.ycombinator.com/item?id=28728801 https://news.ycombinator.com/item?id=28728801
- comex 5y ago
  I am extremely skeptical of this calculation by Drew DeVault: > On average, dynamically linked executables use only 4.6% of the symbols on offer from their dependencies. A good linker will remove unused symbols. Percent of exported symbols used is a terrible proxy for percent of library functionality used. As an example, an SDL hello world program [1] uses 3 functions out of 750 exported by libSDL2. Does that mean it only uses 3/750 = 0.4% of libSDL2's functionality? Of course not; when I tried compiling it against a static libSDL2 instead of dynamic, the executable's binary size increased by a full 70% of the size of libSDL2.so. [2] Now, on one hand I admit that's an extreme example, as a real SDL program would use a higher number of functions. And SDL itself is not very amenable to dead code stripping; a good chunk of the hello-world program is for things like sound or input that the it doesn't actually use. Still, even the hello-world program legitimately needs a bunch of functionality, including support for both X11 and Wayland, and setting up hardware accelerated surfaces using OpenGL. And a real program would need more. Also, I'm not even counting the long list of shared libraries linked by SDL (ranging from libX11 to libFLAC); a SDL-based program will use 0% of their symbols (since it doesn't them directly), but may need a substantial portion of their functionality. More importantly, this general pattern, where exported APIs are just the tip of the iceberg, applies to many libraries. I tried a similar experiment with zpipe, the official example program for zlib – a much smaller and shallower library. It ended up using 23% of symbols but 53% of the implementation. On the other end of the spectrum, the problem would be far worse with a real GUI library, where even, say, a simple text input box truly requires massive amounts of functionality including display, layout, Unicode font rendering, menus, internationalization, and keyboard and mouse input. Plus, of course, a variety of input method editors if you want to support languages like Japanese. In a statically linked world you would probably want to implement most of that as IPC to shared daemons. Such a design could work; there isn't much that really needs the performance benefits of being in-process. But no major GUI libraries are written that way, because in a world with shared libraries there's no need. [1] https://gist.github.com/comex/c64b5a7e409d5d48ad6ebf37ec068b03 https://gist.github.com/comex/c64b5a7e409d5d48ad6ebf37ec068b... [2] Test harness also at [1]. This is with LTO; without LTO the percentage is much higher. I also had to disable SDL's "dynapi" feature that makes static linking secretly actually dynamically link. Which they put in for good reasons, albeit mostly applicable only to proprietary software where the user can't just recompile/relink.
cesarb 5y ago
The first section, on security fixes ("[...] you only need to update a shared library to apply security fixes to all executables that rely on that library"), starts talking about security fixes but then suddenly switches to talking about ABI and API breaks. But that sleight-of-hand hides the fact that many (perhaps even most) security fixes do not break the ABI or API; they are completely contained to the implementation (one obvious exception would be if the security issue was caused by bad API design, but even then often there are ways to fix it without breaking the ABI).
- rewma 5y ago
  > But that sleight-of-hand hides the fact that many (perhaps even most) security fixes do not break the ABI or API; they are completely contained to the implementation (one obvious exception would be if the security issue was caused by bad API design, but even then often there are ways to fix it without breaking the ABI). Right you are. I was also perplexed when I read that non sequitur. The author's reference to DLL Hell also suggests there's some confusion in his analysis of the underlying problem, given that DLL Hell is very specific to windows and at best is only orthogonally related to ABI. The author's references to API changes make even less sense, and definitely cast doubt over his insight into the issue.
  - pjmlp 5y ago
    And was kind of fixed with application manifests, on XP.
  - salawat 5y ago
    Differentiating DLL and SO hell is getting a bit beyond pedantic as they are implementations of the same fundamental abstraction. Any substantial difference in merely one of implementation details.
    - rewma 5y ago
      > Differentiating DLL and SO hell is getting a bit beyond pedantic (...) It really isn't. Unlike linking problems, where the problem is focused on how you need to fight your dependencies to be able to resolve symbols, DLL Hell has been for over a decade a dependency resolution problem that is solved at the packaging level. More importantly, on Windows, where sharing DLLs is not a thing, you can simply drop a DLL in the app dir and be done with it. In fact, it's customary for windows apps to just bundle all their dependencies.
lgg 5y ago
Every time I read one of these essays extolling the virtues of static linking it makes me pretty sad. Engineering is about trade offs, and sometimes it does make sense to statically link, but the fact that those trade offs are so lopsided on Linux has very little to do with dynamic linking vs static linking as generic concepts, and more to do with the fact that the design of ld.so is straight out of the 1990s and almost nothing has be done to either exploit the benefits dynamic linking brings, nor to mitigate the issues it causes. On Darwin derived systems (macOS, iOS, tvOS, watchOS) we have invested heavily in dynamic linking over the last two decades. That includes features we use to improve binary compatibility (things like two level namespaces (aka Direct Binding in ELF) and umbrella frameworks), middleware distribution (through techniques like bundling resources with their dylibs into frameworks), and mitigate the security issues (through technologies like __DATA_CONST). Meanwhile, we also substantially reduced the cost of dynamic linking through things like the dyld shared cache (which makes dynamic linking of most system frameworks almost free, and in practice it often reduces their startup cost to below the startup cost of statically linking them once you include the cost of rebasing for PIE), and mitigate much of the rest of the cost of dynamic linking through things like pre-calculated launch closures. It does not hurt that my team owns both the static and dynamic linkers. We have a tight design loop so that when we come up with ideas for how to change libraries to make the dynamic linker load them faster we have the ability to rapidly deploy those changes through the ecosystem. As I explained in here[1] dynamic linking on macOS is way faster than on Linux because we do it completely differently, and because of that it is used way more pervasively on our systems (a typical command line binary loads 80-150 dylibs, a GUI app around 300-800 depending on which of our OS you are talking about). IOW, by mitigating the costs we drove up the adoption which amplifies the benefits. And it is not like we have squeezed all the performance out of the dynamic linker that we can, not by a long shot. We have more ideas than we have time to pursue, as well as additional improvements to static linking. If you have any interest in either static or dynamic linking we're looking for people to work on both[2]. [1]: https://www.realworldtech.com/forum/?threadid=197081&curpostid=197486 https://www.realworldtech.com/forum/?threadid=197081&curpost... [2]: https://jobs.apple.com/en-us/details/200235669/systems-engineer-dyld https://jobs.apple.com/en-us/details/200235669/systems-engin... (edit: fixed link formatting)
- ghoward 5y ago
osigurdson 5y ago
I really dislike the tired re-use of the term "considered harmful". A better title may be "I don't like dynamic linking - here's why".
Jach 5y ago
I'd like to see solutions like SDL2's become more widespread: https://old.reddit.com/r/linux_gaming/comments/1upn39/sdl2_adds_dynamic_api_magic_to_allow_updating_it/ https://old.reddit.com/r/linux_gaming/comments/1upn39/sdl2_a... At runtime, the first call into SDL sets up a jump table for all SDL functions afterwards. When SDL is statically linked, it uses those static functions by default, but it's possible to specify an environmental variable pointing at an SDL dynamic lib, and the jump table will use those functions instead.
forrestthewoods 5y ago
I care far more about about bundled vs unbundled than dynamic vs static. I am passionately in-favor of bundled. Unbundled distribution is what has led to software being so outrageously impossible to execute that the only sane path to distribution is to create a bundled docker image. https://xkcd.com/1987/ https://xkcd.com/1987/
themulticaster 5y ago
> Second, ldd, the dynamic linker, can be manipulated into executing arbitrary code. ldd is not the dynamic linker, it's only a tool to debug the process of dynamic linking. The dynamic linker is ld-linux.so (exact name depends on glibc version, architecture, etc.) Also, I think the linked article [1] about the security of ldd is somewhat useless. The ldd(1) manpage [2] is very explicit about the security of ldd and tells you not to run ldd on untrusted executables: > [...] Thus, you should never employ ldd on an untrusted executable, since this may result in the execution of arbitrary code. It's a little amusing how the linked blog post explains how to create a malicious executable that runs arbitrary code when inspected with ldd, noting that "I researched this subject thoroughly and found that it's almost completely undocumented. I have no idea how this could have gone unnoticed for such a long time." and concluding with "Never run ldd on unknown executables!" - all while the manpage literally mentions that precise known limitation in the third paragraph! To be fair, you could argue that this limitation is not widely known and people should be made aware of the risks of ldd, but on the other hand you can hardly criticize ldd when its manpage is that explicit about the issue. [1] https://catonmat.net/ldd-arbitrary-code-execution https://catonmat.net/ldd-arbitrary-code-execution [2] https://www.man7.org/linux/man-pages/man1/ldd.1.html https://www.man7.org/linux/man-pages/man1/ldd.1.html
- ghoward 5y ago
  Author here. Thank you for pointing out my mistake. I'll fix it.
  - Hello71 5y ago
    ld isn't the dynamic linker either, it's the object linker. the name of the dynamic linker is ld.so or ld-linux.so. this is also documented in the ldd man page.
    - eqvinox 5y ago
      Fun fact: despite the library path and name, ld-linux.so is executable. Try running /lib64/ld-linux-x86-64.so.2 On your favorite Linux box. (Adjust path if needed for different architecture / distro.)
mst 5y ago
> Operating systems can statically map libraries. I have "fond" memories of running perebaseall on cygwin that this sentence reminded me of.
lamontcg 5y ago
I sorta think that static linking needs to die, but dynamic linking also needs to be a lot better. More of the choices that get made during autoconf needs to be made lazily at link time. That would make executables and dynamic libraries more like containers or lego blocks that could be rearranged without compilation. Which would fix a lot of issues that people hate about package managers and bloated distros along the way.
arduinomancer 5y ago
It seems a pain point of static linking is that if a library changes you need to rebuild everyone Maybe a stupid idea but has there ever been thought of distributing applications with source code included? That way you could have a system that automatically rebuilds the application if a static library changes. Meaning when a user sees “updating” it’s actually rebuilding the app underneath.

Correct me if I am wrong but OpenBSD introduced the constraint that syscalls must come from a specific section of memory, that assigned to their libc. In a case like that static linking is not only unstable, as in Windows or macOS, but also impossible. And since that has security benefits we will probably see it being migrated to other platforms in the near future.

dhsysusbsjsi 5y ago

Personal end user perspective: troubleshooting dylib errors is the most common cause of wanting to throw the computer out the window and quit programming.

cratermoon 5y ago
See also: DLL Hell in Windows and, more generally Dependency Hell. Dynamic linking errors are a subset of general dependency problems.
- spacechild1 5y ago
  As others have already mentioned elsewhere, DLL hell hasn't really been a thing anymore on Windows for years. I personally can remember it, but that's more than 15 years ago.
  - cratermoon 5y ago
    It's true, but only because every app essentially isolates itself from the system and carries around its own dependency bundle. I guess that's kind of what Docker containers are, so it's hard to be critical.
eqvinox 5y ago
"end user perspective" + "quit programming"? Do you troubleshoot dylib errors on your regular end user software and just happen to be a programmer unrelated to that? (If you're troubleshooting dylib errors on things you're developing, that's not an end user perspective?)
- dizzy3gg 5y ago
  End user of a linker?
  - rewma 5y ago
    > End user of a linker? I'd argue that if you're not packaging your software or testing your software's dependencies, either you're doing something extremely exotic that lies far outside anyone's happy path or "dylib error" should not even be a keyword in your vocabulary.
    - alisonkisk 5y ago
      The windows version, DLL Hell, is well known to non programmers.
    - pjc50 5y ago
      Roughly the same thing happens on Windows and is called "DLL hell", occasionally witnessed by end users.
      - rewma 5y ago
        DLL Hell ceased to be a practical concern over a decade ago, particularly given that Windows provides tight control over its dynamic linking search order. https://docs.microsoft.com/en-us/windows/win32/dlls/dynamic-link-library-search-order https://docs.microsoft.com/en-us/windows/win32/dlls/dynamic-... DLL Hell is not a linking problem, it's a packaging problem.
- gbear605 5y ago
  There’s a middle level of “troubleshooting dylib errors on open source software I’m installing for personal use because I’m a masochist who isn’t happy with closed source software”

KronisLV 5y ago

Why are we still developing software like we used to 40 years ago? Why not just import external dependencies into your project at a function/class level rather than at a package one: if you only use FooPackage.BarClass and FooPackage.bazMethod(), you should be able to choose to make your project depend on just those two things, ideally in a completely transparent way by your IDE. Then having to manage the full scope of packages and their versions becomes less relevant, because the IDE can check whether a new version has been released in the package manager of choice, check whether those two things have changed and if they have, then demand that the developer have a look at the code changes to refactor code if necessary, otherwise not requiring any action on their part. Furthermore, if you ever need to migrate to a different package or even rewrite the functionality itself, you could just look at the few signatures that you use, rather than having to remove the package entirely and see what breaks in your IDE. Why can't we have itemized lists of everything that's called and everything that we depend on, as opposed to the abstraction of entire packages? Better yet, why even depend on binary blobs or hidden code (that's ignored by your IDE) in the first place? Why not just download the source for every package that you use and upon updates be able to review the code changes to the package much like a regular Git diff? Of course, this would probably require getting rid of reflection and other forms of dynamic code, which i fully support, since those have never been good for much in the first place and only destroy any hopes of determinism and full control flow analysis. As for the possible counterargument of this being hard to do: it wouldn't really be with more granular packages. Instead of trying to shove an ecosystem inside of a single package, why not split it into 10-20 bits of reusable code instead? Smaller packages, which would be easier to review and manage. Context: i dislike how Spring and Spring Boot in Java force a huge ecosystem of fragile dependencies upon you, with their reflection which ensures that your apps will break at runtime, for example, when moving from Spring Boot 1.5 to 2.0. Furthermore, in the JS world, the node_modules folders are probably 10-1000x too large for what's actually necessary to display a webpage with some interactive behaviour. Disclaimer: i have no delusions about any of the above being technically feasible right now. Perhaps "Why?" would be a good question to ask. In my eyes, perhaps the industry is too set in its current ways and as long as we don't have languages and entire ecosystems that approach established practices from a wildly different angle, no one can actually contest the decades of already built packages and tools. On the bright side, WireGuard kind of displaced OpenVPN somewhat and there are numerous benefits to it being smaller, which is a good example of getting rid of bloated software. Furthermore, Nix may or may not do that to alternative approaches to package management, but its usage still remains really low. In my eyes, the only long term solution is to be able to tell exactly what's necessary for your code to build and work, and to test every dependency update that comes out against this automated process. Static dependencies but at the speed of updates of dynamic dependencies. Then you'd just have to wait for a day until the devs fix the newest regressions of a new dependency release before getting a new statically linked app, as opposed to using dynamic linking and finding that some dependency breaks your app as it happens in prod.

ChrisSD 5y ago

> However, there is one big elephant in the room: right now, there are a lot of platforms that do not fully support static linking because you have to dynamically link libc or other system interfaces. The only one I know of that supports static linking is Linux. > The ironic thing is that operating systems only supported a stable syscall ABI, rather than requiring programs to dynamically link system libraries, they would have less problems with ABI breaks. > With a statically-linked binary, you can copy it to another machine with a Linux that supports all of the needed syscalls, and it will just run. I think this is a misunderstanding. With Linux there is a stable interface at the kernel boundary. Other OSes have their stable interfaces in user space. The difference is mostly irrelevant in this context. In one case you dynamically call into the kernel, in another case you dynamically link a user space library. Either way the code you're calling isn't static but the interface is.

amelius 5y ago
I'm distributing my code as a bootable USB drive.
CyberShadow 5y ago
There is a distinction between a fully-static binary and a non-fully-static one, which is what I think the article refers to. A fully static binary will not need to call the dynamic linker at all, and tools such as "file" or "ldd" will identify such binaries differently. A fully static binary will also run even in the complete absence of any userspace support for its architecture (assuming the hardware and kernel do support it) - e.g. a static AArch64 binary will run on a 32-bit ARM distribution, if the CPU supports AArch64.
- ChrisSD 5y ago
  True, however... > assuming the hardware and kernel do support it Assuming kernel support is doing a lot of heavy lifting there. To put it another way, why is a syscall better than any other stable ABI? If both kernel and stable library are distributed together then why should it be considered different?

Wowfunhappy 5y ago

Are there any Linux distros that actually distribute (most) software as (mostly) static binaries? Right now, the only way to do it seems to be compiling everything myself...

lrem 5y ago
Why would you want that?
- blablabla123 5y ago
  I think programs in /bin used to be usually statically linked. When updating the glibc for instance, this makes the change far more doable. In general dealing with static binaries is much less pain, especially when they haven't been compiled locally/by the distributor for this particular version of the distribution.
  - drivebycomment 5y ago
    /sbin used to be statically linked on unix. The entire point of /sbin is the independence from almost everything else so that you can rely on them even when the system boots up in a weird state.
- Wowfunhappy 5y ago
  Simplicity. If everything is statically linked, then updating package A can never break package B. As I see it, the entire world of Docker containers exists primarily to solve the problem that static linking already solved. Something like Nix would work as well, and I think I would love Nix if I knew how to use it. But I don't, and I don't really have the time to learn.
  - lrem 5y ago
    I don't think that makes a lot of sense in a distro. If package A gets an update, it means a sufficiently bad bug was discovered in it. If it happens to be a library, you want to make sure you update every package B that uses it. Which, conveniently, the distro solves for you too.

3np 5y ago

""Static Linking Considered Harmful" Considered Harmful" Considered Harmful: This is the blog version of Re: Re: Fwd: Re: in email and has to stop here. See also https://meyerweb.com/eric/comment/chech.html https://meyerweb.com/eric/comment/chech.html

Brian_K_White 5y ago
Incorrect. Ok fine, explaination: "Re:" are not art. "Re:" are not added deliberately and for intentional artistic effect, and are not references (despite the literal) or homages.
ghoward 5y ago
Author here. See my reply to https://meyerweb.com/eric/comment/chech.html https://meyerweb.com/eric/comment/chech.html at https://news.ycombinator.com/item?id=28737279 https://news.ycombinator.com/item?id=28737279 .
- 3np 5y ago
  I was mostly being facetious and wrote that in jest - with a title like that you should be prepared for some OT banter ;)

davidwf 5y ago

This is very situational, but I have recently been part of a project that does a lot of C server-side development and we have found that static linking our non-glibc dependencies has really improved our developer experience. Using ceedling's dependency plugin[1] and producing a single "statically" linked library has made our C development much closer to using a language with a more modern package manager. Don't get me wrong, if I was trying to distribute binaries to machines I didn't control I'd definitely be willing to invest in the Linux packaging "fun", but for a server-side application it's been a good choice for our team overall. [1] https://github.com/ThrowTheSwitch/Ceedling/tree/master/plugins/dependencies https://github.com/ThrowTheSwitch/Ceedling/tree/master/plugi...

PaulDavisThe1st 5y ago
Yes and no. In Ardour, which has a substantial dependency tree (80+ libraries), if we statically link, the edit/compile/debug cycle becomes incredibly bogged down by static linkage. It takes a long time (even with lld) to complete the link step. If we use shared libraries/dynamic linkage, the edit/compile/debug cycle is nice and fast, but application startup is a bit slower (not by as much as static linkage is slower, however). For users, it would be better if the application was statically linked, at least in terms of startup cost. But because developers do the edit/compile/debug cycle much, much more often than users start the application, we opt for dynamic linkage.
- xyzzyz 5y ago
  You can dynamically link during development, and then ship statically linked binary (with LTO etc). That’s what Chrome does, for example.
  - PaulDavisThe1st 5y ago
    It's not that simple for us (we have tried this).

ris 5y ago

I eagerly await some concrete implementations of these new ideas. In the meantime I'm waiting for the first big vulnerability in a widely used golang or rust library to see if downstream projects pinning it also release CVEs as they should (which would probably DDoS the CVE system from the resulting avalanche) or they quietly bump it and move on (in which case their users won't get the nudge to upgrade). This is where someone says "well you should just always be running the latest version of everything", which is of course infeasible on a real life system with thousands of installed packages. And it's not that I don't completely sympathize or understand the advantages of static linking from a developer's point of view - you don't have to sell it to me there.

DaiPlusPlus 5y ago
The design of the Go build and packages system kinda renders that point moot because the expectation is that provided you follow Go's guidelines then a Go application should always be easily recompilable, and all external dependencies should always be distributed in source-form, not as binaries (e.g. https://github.com/golang/go/issues/2775 https://github.com/golang/go/issues/2775 ).
- Macha 5y ago
  Right, but if every application did distribute as source and could just be recompiled, there would be fewer complaints about the difficulties distributing Linux binaries
- Zababa 5y ago
  > provided you follow Go's guidelines then a Go application should always be easily recompilable, and all external dependencies should always be distributed in source-form, not as binaries So it's a human solution, and not a technical one, which means that it can and will fail.
  - DaiPlusPlus 5y ago
    That's like saying condoms are ineffective because you included people too lazy to put them on in the first place in your calculations. Provided you (and your org) use Go as Google intends then that isn't an issue.

flohofwoe 5y ago

Unfortunately not even Linux allows full static linking (as far as I'm aware at least) as soon as libraries like OpenGL are involved. Dynamic linking with core system libraries which are guaranteed to exist(!) is fine though, for everything else dynamic linking mostly has downsides. Also: it's only Linux where this is an issue, because it's nearly impossible to build small self-contained executables which run on another Linux machine, thanks to glibc's versioning mess (at least there's MUSL as an alternative to create portable command line tools).

Macha 5y ago
Didn't Go have to throw in the towel on not dynamically linking libc on macOS as there's no guarantee of syscall stability at all?
- flohofwoe 5y ago
  Don't know, I only know that I can build a macOS command line tool on one machine, and run it on a different machine without problems, while on Linux it's very easy to run into the glibc version compatibility problem.
  - onei 5y ago
    I always understood the glibc versioning problem to be solved by just building on the oldest common ancestor. If you have rhel5 through 8, you build on rhel5 and move on. If you depend on more than glibc... Well rhel5 is less fun.
    - ptsneves 5y ago
      Indeed, the only difference is that you might be using less optimal system calls, but there is no reason to be blocked and not target ancient kernels. I linked to 2.6 this year and everything ran fine.
    - ghoward 5y ago
      Author here. Unfortunately, building on the oldest common ancestor will still not save you from ABI breaks. An example is [1]. If the size of `intmax_t` changes between that oldest ancestor and the machines you are running on, get ready for pain. [1]: https://thephd.dev/intmax_t-hell-c++-c https://thephd.dev/intmax_t-hell-c++-c

deleted 5y ago

[deleted]

deleted 5y ago

[deleted]

0xbadcafebee 5y ago

I wish I had three hours to waste on why almost all of these points are either innacurate or willfully misleading. But I don't, so I'll just point out the single biggest reason why static linking is harmful. Encryption libraries. If OpenSSL, or LibreSSL, or Go's encryption modules, or any gigantic encryption library has a vulnerability, you basically have to recompile, distribute, and then have your users download, every single god damn networking program. But it's worse than that. Because everyone wants to ship a statically compiled Go app, nobody packages for the distros. So now every user needs to go track down where they downloaded each of their static Go apps, safely download the update, verify its checksum or signature via a 3rd party, and upgrade the app. That is a logistical nightmare. Not only are apps going to remain vulnerable for way, way longer, more users will be compromised by phishing and other attacks because there's no package manager to safely automate all these updates (distros sign and verify packages for you). I've actually written a Shell quasi-package manager/installer just for statically compiled apps (cliv) so in theory that problem could be somewhat solved... But nobody even knows about that program, so the manual update problem still stands.

AshamedCaptain 5y ago
If you ever statically link OpenSSL, your binary now has a builtin expiration date. You are doing your users a disservice.
- marcosdumay 5y ago
  To be fair, if you dynamically link to OpenSSL, your binary has a builtin expiration date too, and it's very likely the same date.
  - nly 5y ago
    OpenSSL ABI has been fairly stable within a given series. Something compiled against 1.1.0a (Sep 2016) has a fair chance of running on a system that has 1.1.0l (Aug 2021). 5 years of binary compatibility is nothing to be sniffed at for a C library. https://abi-laboratory.pro/index.php?view=timeline&l=openssl https://abi-laboratory.pro/index.php?view=timeline&l=openssl

rascul 5y ago

According to Linus, "Shared libraries are not a good thing in general". https://lore.kernel.org/lkml/CAHk-=whs8QZf3YnifdLv57+FhBi5_WeNTG1B-suOES=RcUSmQg@mail.gmail.com/ https://lore.kernel.org/lkml/CAHk-=whs8QZf3YnifdLv57+FhBi5_W...

ghoward 5y ago
Author here. I quote that email twice in my post. I decided not to quote that part of the email just in case it might inflame the debate. But yes, he did say that.

PaulDavisThe1st 5y ago

> I don’t think Ulrich Drepper could have foreseen all of the problems with DLL Hell (though there were signs on Windows), Drepper's article is from 2006. "DLL Hell" was beyond fully understood at that point, certainly by Drepper and large numbers of other well informed programmers.

ghoward 5y ago
Author here. In everything in the post, I attempted to give Drepper the benefit of the doubt, though there are cases where I don't want to since he sounds (to me) paternalistic in the way Apple and Microsoft do. So perhaps he understood, but I didn't want to assume that, especially since it might inflame the discussion, which is exactly what I was trying to avoid doing (again).
- PaulDavisThe1st 5y ago
  DLL Hell was well understood in the mid-90s. I think you need to give him more benefit of the doubt on this point.
  - ghoward 5y ago
    I'm having a hard time understanding because if DLL Hell was well understood by 2006, it seems I should have given him less benefit of the doubt, but you say I should give him more. Apologies.
  - torstenvl 5y ago
    If someone glosses over an issue, it's either innocent (didn't know about it, didn't understand it) or deceptive (intentionally ignored it). Giving Drepper the benefit of the doubt here means assuming he didn't know or understand the problem he failed to adequately address.
    - PaulDavisThe1st 5y ago
      Here is most of the relevant text from TFA: > Ulrich Drepper claims, rightly, that you only need to update a shared library to apply security fixes to all executables that rely on that library. > This is a good thing! It is also a good vision. > Unfortunately, it is only a vision because it’s not the whole story. > As of yet, there is no way to automatically determine if the ABI or API of a shared library changed between updates. This is called DLL Hell. The final two words are a link to the wikipedia page on "DLL Hell". The final sentence of the intro section there states: > DLL Hell is the Windows ecosystem-specific form of the general concept dependency hell. That is, TFA defines "DLL Hell" using a wikipedia page that explicitly states that it is a Windows-specific version of a more general problem. Drepper's article from 2006 fully tackles the general problem, albeit without (as the TFA puts it): "a way to automatically determine if the ABI or API of a shared library changed between updates." Drepper's primary suggestion there is a naming/versioning scheme which describes specifically whether the ABI/API has changed in ways that matter for shared linkage. It's not automatic - that part is true. But is is a solution, widely used in Linux and *nix more broadly. Did Drepper address the Windows specific parts of "DLL Hell". He did not, but that's because "DLL Hell" is Windows specific (as other comments here fully clarify), and he was not writing about how to fix the full scope of that particular nightmare. He likely understood "DLL Hell" as well as anyone, nevertheless.

rwmj 5y ago

I read this interesting article recently about all the security problems that static linking, bundling and pinning cause for Linux distributions: https://blogs.gentoo.org/mgorny/2021/02/19/the-modern-packagers-security-nightmare/ https://blogs.gentoo.org/mgorny/2021/02/19/the-modern-packag...

Aqueous 5y ago
But should distributing and ensuring the security of software that users install on their own systems even be the responsibility of operating system maintainers?
- eqvinox 5y ago
  Yes. If I installed software through my OS package manager, where else would I get security updates from?!
  - Aqueous 5y ago
    Like on every other major OS the publisher of those applications. And I mean they should be distributing the applications as well. Not the OS publishers.
- rwmj 5y ago
  Isn't it obvious? Or do you suggest that end users should be responsible for checking and patching every package on their system? Given that Red Hat has a large department just to do that for RHEL, I can't imagine where end users would begin.
  - Aqueous 5y ago
    No, I’m suggesting that the authors of the applications are responsible for patching and distributing their own application like on every other major OS.
ghoward 5y ago
Author here, and I am also a Gentoo user on my daily driver. I read that article when it came out, and it felt disingenuous. mgorny helps maintain Gentoo, which of all distributions, does rebuild software as necessary. It would be a simple change to portage to rebuild all dependencies of a library when that library is updated, regardless of if the user uses static linking or not. In fact, there is a project to have a statically-linked musl-based Gentoo. So it's possible and (I would argue) would be an easy change. mgorny would not have to worry about much because he could, in essence, treat all libraries as dynamic libraries. The only pain to be felt would be by users who would see their build times increase (to rebuild dependents). As a user, I would happily take that tradeoff. I already build my browser, which takes 5-8 hours; I'll happily rebuild dependents of updated libraries.

api 5y ago

My personal view is that only standard core system libraries should be dynamically linked. Anything that is not totally ubiquitous should be static. That seems to be the Go and Rust default approach, or close to it. Link libc and such, but build in more obscure things. The idea of dynamically linking everything doesn’t scale. It asks too much of distribution maintainers and makes them the choke point for every little upgrade.

PaulDavisThe1st 5y ago
So, you link against a GUI toolkit that can be used to display images. Images in many different formats, including some that haven't been used in more than a decade and are unlikely to ever be encountered. Do you want the GUI toolkit linkage to automatically include every object module for every possible image file format in your static image, or do you want the GUI toolkit to use runtime dynamic linkage (explicit or implicit) to import only the modules required for the image formats actually encountered?
- TillE 5y ago
  If I'm distributing binaries as a commercial product, I definitely want the former.
  - PaulDavisThe1st 5y ago
    There are many "commercial products" that run on Linux whose installation includes various *.so files that will be dynamically linked at run time. Are you saying this is a bad idea?
  - TeMPOraL 5y ago
    What about formats that don't exist yet? What about giving you the ability to supply your own encoder/decoder for some niche format you're using?
josefx 5y ago
> That seems to be the Go and Rust default approach I think Go tried to go fully static but ran into problems as most mainstream OSes except for Linux do not provide a stable system call interface, so applications have to link at least one dynamic library. The Linux system call interface is stable, but expects a C style stack layout and that also bit the Go devs. multiple times. So you can probably go fully static if you are only targeting Linux.

tinus_hn 5y ago

Considered harmful essays considered harmful: https://meyerweb.com/eric/comment/chech.html https://meyerweb.com/eric/comment/chech.html

colejohnson66 5y ago
I was wondering when something like this would be written. We’ve come full circle.
ghoward 5y ago
Author here. I read that essay a while ago and reread it before publishing my post. I don't think the author makes a good case. They make bare assertions without examples and without evidence, so it's unconvincing. For example, they say that a "Considered Harmful" essay: > often serves to inflame whatever debate is in progress, and thus makes it that much harder for a solution to be found through any means. But this could be true of any essay. More to the point, I think that the tone of the essay or post matters more. Case in point, my "Dynamic Linking Needs to Die" post probably inflames the debate more than this new "Considered Harmful" post. Maybe "Considered Harmful" essays have a reputation for inflaming the debate, and there's an argument to be made for that. But I personally don't think my post is one that follows that reputation. Also, because I was writing a post about a pre-existing "Considered Harmful" essay, it was natural to use it, especially since I needed to make it clear that my previous post is harmful too.
- jasode 5y ago
  >Maybe "Considered Harmful" essays have a reputation for inflaming the debate, and there's an argument to be made for that. But I personally don't think my post is one that follows that reputation. Neither your post title nor your content bothered me but just as fyi... some people are very irritated by "snowclones". My previous comment with various examples: https://news.ycombinator.com/item?id=19071532 https://news.ycombinator.com/item?id=19071532
  - ghoward 5y ago
    Yeah, I suspected people might be. But since I was answering a "Considered Harmful" essay, it was too easy. Eh, oh well.

gnufx 5y ago

There are examples of the sort of reason I want dynamic linking under https://news.ycombinator.com/item?id=28734875 https://news.ycombinator.com/item?id=28734875 and you also really want it for profiling/tracing, and possibly debugging, e.g. in HPC with the MPI PMPI mechanism. libabigail is one tool for checking that a library's ELF versioning hasn't lied about ABI compatibility when you're building packages, for instance. Unfortunately people typically don't follow Drepper's advice (as far I remember the article) on building shared libraries, particularly using symbol versioning a la glibc.

ghoward 5y ago
Author here. Yeah, there are reasons to use dynamic linking, but I'm still not sure why dynamic linking gives you better profiling and tracing. > Unfortunately people typically don't follow Drepper's advice (as far I remember the article) on building shared libraries, particularly using symbol versioning a la glibc. I mention this in the post, but I quoted another person, Ian Lance Taylor, who wrote the gold linker, as saying that symbol combining, a portion of symbol versioning, is an unsolved problem. Even if developers don't follow Drepper's advice, I wouldn't put the blame on them because of that fact; I would argue that it's more accurate to say they can't really follow his advice.
- dchapp 5y ago
  > Yeah, there are reasons to use dynamic linking, but I'm still not sure why dynamic linking gives you better profiling and tracing. It's not so much that the tracing becomes better, but that it becomes feasible at all. Two specific situations come to mind, both MPI-adjacent. (1) Running a PMPI-based tool on code you can't recompile yourself (e.g., you need a Q clearance to see the source, but not to actually execute it--weird I know, but not that uncommon in the DOE labs.); and (2) running multiple PMPI-based tools simultaneously which are composed at runtime via PnMPI.
  - ghoward 5y ago
    I addressed these arguments. I think that if you don't have access to the source, it's a security problem. That includes not having access to the source because of security clearances. As far as I am concerned, your superiors in the DOE are adversaries to you and your machine. Other people will have different viewpoints on that, and that's fine.

ghoward 5y ago

Author here. I don't know why my own submission of this [1] didn't catch and this one did. Oh well. XD I'm open to answering questions, and I'll be hanging in the thread. [1]: https://news.ycombinator.com/item?id=28728801 https://news.ycombinator.com/item?id=28728801

comex 5y ago
I am extremely skeptical of this calculation by Drew DeVault: > On average, dynamically linked executables use only 4.6% of the symbols on offer from their dependencies. A good linker will remove unused symbols. Percent of exported symbols used is a terrible proxy for percent of library functionality used. As an example, an SDL hello world program [1] uses 3 functions out of 750 exported by libSDL2. Does that mean it only uses 3/750 = 0.4% of libSDL2's functionality? Of course not; when I tried compiling it against a static libSDL2 instead of dynamic, the executable's binary size increased by a full 70% of the size of libSDL2.so. [2] Now, on one hand I admit that's an extreme example, as a real SDL program would use a higher number of functions. And SDL itself is not very amenable to dead code stripping; a good chunk of the hello-world program is for things like sound or input that the it doesn't actually use. Still, even the hello-world program legitimately needs a bunch of functionality, including support for both X11 and Wayland, and setting up hardware accelerated surfaces using OpenGL. And a real program would need more. Also, I'm not even counting the long list of shared libraries linked by SDL (ranging from libX11 to libFLAC); a SDL-based program will use 0% of their symbols (since it doesn't them directly), but may need a substantial portion of their functionality. More importantly, this general pattern, where exported APIs are just the tip of the iceberg, applies to many libraries. I tried a similar experiment with zpipe, the official example program for zlib – a much smaller and shallower library. It ended up using 23% of symbols but 53% of the implementation. On the other end of the spectrum, the problem would be far worse with a real GUI library, where even, say, a simple text input box truly requires massive amounts of functionality including display, layout, Unicode font rendering, menus, internationalization, and keyboard and mouse input. Plus, of course, a variety of input method editors if you want to support languages like Japanese. In a statically linked world you would probably want to implement most of that as IPC to shared daemons. Such a design could work; there isn't much that really needs the performance benefits of being in-process. But no major GUI libraries are written that way, because in a world with shared libraries there's no need. [1] https://gist.github.com/comex/c64b5a7e409d5d48ad6ebf37ec068b03 https://gist.github.com/comex/c64b5a7e409d5d48ad6ebf37ec068b... [2] Test harness also at [1]. This is with LTO; without LTO the percentage is much higher. I also had to disable SDL's "dynapi" feature that makes static linking secretly actually dynamically link. Which they put in for good reasons, albeit mostly applicable only to proprietary software where the user can't just recompile/relink.

cesarb 5y ago

The first section, on security fixes ("[...] you only need to update a shared library to apply security fixes to all executables that rely on that library"), starts talking about security fixes but then suddenly switches to talking about ABI and API breaks. But that sleight-of-hand hides the fact that many (perhaps even most) security fixes do not break the ABI or API; they are completely contained to the implementation (one obvious exception would be if the security issue was caused by bad API design, but even then often there are ways to fix it without breaking the ABI).

rewma 5y ago
> But that sleight-of-hand hides the fact that many (perhaps even most) security fixes do not break the ABI or API; they are completely contained to the implementation (one obvious exception would be if the security issue was caused by bad API design, but even then often there are ways to fix it without breaking the ABI). Right you are. I was also perplexed when I read that non sequitur. The author's reference to DLL Hell also suggests there's some confusion in his analysis of the underlying problem, given that DLL Hell is very specific to windows and at best is only orthogonally related to ABI. The author's references to API changes make even less sense, and definitely cast doubt over his insight into the issue.
- pjmlp 5y ago
  And was kind of fixed with application manifests, on XP.
- salawat 5y ago
  Differentiating DLL and SO hell is getting a bit beyond pedantic as they are implementations of the same fundamental abstraction. Any substantial difference in merely one of implementation details.
  - rewma 5y ago
    > Differentiating DLL and SO hell is getting a bit beyond pedantic (...) It really isn't. Unlike linking problems, where the problem is focused on how you need to fight your dependencies to be able to resolve symbols, DLL Hell has been for over a decade a dependency resolution problem that is solved at the packaging level. More importantly, on Windows, where sharing DLLs is not a thing, you can simply drop a DLL in the app dir and be done with it. In fact, it's customary for windows apps to just bundle all their dependencies.

lgg 5y ago

Every time I read one of these essays extolling the virtues of static linking it makes me pretty sad. Engineering is about trade offs, and sometimes it does make sense to statically link, but the fact that those trade offs are so lopsided on Linux has very little to do with dynamic linking vs static linking as generic concepts, and more to do with the fact that the design of ld.so is straight out of the 1990s and almost nothing has be done to either exploit the benefits dynamic linking brings, nor to mitigate the issues it causes. On Darwin derived systems (macOS, iOS, tvOS, watchOS) we have invested heavily in dynamic linking over the last two decades. That includes features we use to improve binary compatibility (things like two level namespaces (aka Direct Binding in ELF) and umbrella frameworks), middleware distribution (through techniques like bundling resources with their dylibs into frameworks), and mitigate the security issues (through technologies like __DATA_CONST). Meanwhile, we also substantially reduced the cost of dynamic linking through things like the dyld shared cache (which makes dynamic linking of most system frameworks almost free, and in practice it often reduces their startup cost to below the startup cost of statically linking them once you include the cost of rebasing for PIE), and mitigate much of the rest of the cost of dynamic linking through things like pre-calculated launch closures. It does not hurt that my team owns both the static and dynamic linkers. We have a tight design loop so that when we come up with ideas for how to change libraries to make the dynamic linker load them faster we have the ability to rapidly deploy those changes through the ecosystem. As I explained in here[1] dynamic linking on macOS is way faster than on Linux because we do it completely differently, and because of that it is used way more pervasively on our systems (a typical command line binary loads 80-150 dylibs, a GUI app around 300-800 depending on which of our OS you are talking about). IOW, by mitigating the costs we drove up the adoption which amplifies the benefits. And it is not like we have squeezed all the performance out of the dynamic linker that we can, not by a long shot. We have more ideas than we have time to pursue, as well as additional improvements to static linking. If you have any interest in either static or dynamic linking we're looking for people to work on both[2]. [1]: https://www.realworldtech.com/forum/?threadid=197081&curpostid=197486 https://www.realworldtech.com/forum/?threadid=197081&curpost... [2]: https://jobs.apple.com/en-us/details/200235669/systems-engineer-dyld https://jobs.apple.com/en-us/details/200235669/systems-engin... (edit: fixed link formatting)

ghoward 5y ago

osigurdson 5y ago

I really dislike the tired re-use of the term "considered harmful". A better title may be "I don't like dynamic linking - here's why".

Jach 5y ago

I'd like to see solutions like SDL2's become more widespread: https://old.reddit.com/r/linux_gaming/comments/1upn39/sdl2_adds_dynamic_api_magic_to_allow_updating_it/ https://old.reddit.com/r/linux_gaming/comments/1upn39/sdl2_a... At runtime, the first call into SDL sets up a jump table for all SDL functions afterwards. When SDL is statically linked, it uses those static functions by default, but it's possible to specify an environmental variable pointing at an SDL dynamic lib, and the jump table will use those functions instead.

forrestthewoods 5y ago

I care far more about about bundled vs unbundled than dynamic vs static. I am passionately in-favor of bundled. Unbundled distribution is what has led to software being so outrageously impossible to execute that the only sane path to distribution is to create a bundled docker image. https://xkcd.com/1987/ https://xkcd.com/1987/

themulticaster 5y ago

> Second, ldd, the dynamic linker, can be manipulated into executing arbitrary code. ldd is not the dynamic linker, it's only a tool to debug the process of dynamic linking. The dynamic linker is ld-linux.so (exact name depends on glibc version, architecture, etc.) Also, I think the linked article [1] about the security of ldd is somewhat useless. The ldd(1) manpage [2] is very explicit about the security of ldd and tells you not to run ldd on untrusted executables: > [...] Thus, you should never employ ldd on an untrusted executable, since this may result in the execution of arbitrary code. It's a little amusing how the linked blog post explains how to create a malicious executable that runs arbitrary code when inspected with ldd, noting that "I researched this subject thoroughly and found that it's almost completely undocumented. I have no idea how this could have gone unnoticed for such a long time." and concluding with "Never run ldd on unknown executables!" - all while the manpage literally mentions that precise known limitation in the third paragraph! To be fair, you could argue that this limitation is not widely known and people should be made aware of the risks of ldd, but on the other hand you can hardly criticize ldd when its manpage is that explicit about the issue. [1] https://catonmat.net/ldd-arbitrary-code-execution https://catonmat.net/ldd-arbitrary-code-execution [2] https://www.man7.org/linux/man-pages/man1/ldd.1.html https://www.man7.org/linux/man-pages/man1/ldd.1.html

ghoward 5y ago
Author here. Thank you for pointing out my mistake. I'll fix it.
- Hello71 5y ago
  ld isn't the dynamic linker either, it's the object linker. the name of the dynamic linker is ld.so or ld-linux.so. this is also documented in the ldd man page.
  - eqvinox 5y ago
    Fun fact: despite the library path and name, ld-linux.so is executable. Try running /lib64/ld-linux-x86-64.so.2 On your favorite Linux box. (Adjust path if needed for different architecture / distro.)

mst 5y ago

> Operating systems can statically map libraries. I have "fond" memories of running perebaseall on cygwin that this sentence reminded me of.

lamontcg 5y ago

I sorta think that static linking needs to die, but dynamic linking also needs to be a lot better. More of the choices that get made during autoconf needs to be made lazily at link time. That would make executables and dynamic libraries more like containers or lego blocks that could be rearranged without compilation. Which would fix a lot of issues that people hate about package managers and bloated distros along the way.

arduinomancer 5y ago

It seems a pain point of static linking is that if a library changes you need to rebuild everyone Maybe a stupid idea but has there ever been thought of distributing applications with source code included? That way you could have a system that automatically rebuilds the application if a static library changes. Meaning when a user sees “updating” it’s actually rebuilding the app underneath.