7 ms·

> Given these, it seems worthwhile to re-evaluate the tradeoffs that come with dynamic linking Perhaps this is just a Linux distro thing, but as someone who cl

> Given these, it seems worthwhile to re-evaluate the tradeoffs that come with dynamic linking Perhaps this is just a Linux distro thing, but as someone who closely monitors the Void packages repo, dynamic linking burdens distro maintainers with hundreds of additional packages which need to be individually tracked, tested and updated. (Packages which could otherwise be vendored by upstream and statically linked during the build.) Dynamic linking also adds complexity to distro upgrades, because dependant packages often need to be rebuilt when the libraries they dynamically link to are changed or upgraded. For example, Void’s recent switch from LibreSSL to OpenSSL borked my install script which wasn’t aware of the change. It resulted in a built system whose XBPS package manager couldn’t verify SSL certs. On Arch, AUR packages were notoriously prone to dynamic linking errors (back when I still used it). Personally, I don’t find the bandwidth efficiency and CVE arguments for dynamic linking to be all that convincing [1]: Will security vulnerabilities in libraries that have been statically linked cause large or unmanagable updates? Findings: not really Not including libc, the only libraries which had "critical" or "high" severity vulnerabilities in 2019 which affected over 100 binaries on my system were dbus, gnutls, cairo, libssh2, and curl. 265 binaries were affected by the rest. The total download cost to upgrade all binaries on my system which were affected by CVEs in 2019 is 3.8 GiB. This is reduced to 1.0 GiB if you eliminate glibc. It is also unknown if any of these vulnerabilities would have been introduced after the last build date for a given statically linked binary; if so that binary would not need to be updated. Many vulnerabilities are also limited to a specific code path or use-case, and binaries which do not invoke that code path in their dependencies will not be affected. A process to ascertain this information in the wake of a vulnerability could be automated. [1]: https://drewdevault.com/dynlib https://drewdevault.com/dynlib

rkeene2 5y ago
As someone who runs a Linux distribution, please don't vendor your dependencies for the benefit of distribution maintainers, it makes it much more difficult to build a coherent system. Running different versions of common libraries causes real annoyances.
- Ericson2314 5y ago
  Yes this can't be emphasized enough. static linking is fine, I don't really care either way. But please, please don't vendor. The point of software is to be composable, and vendoring kills that. Vendoring will cause free software to drown in its own complexity and is completely irresponsible.
  - chris_wot 5y ago
    Could someone explain what is meant by “vendoring”?
    - jrockway 5y ago
      Vendoring is when you just copy the dependencies into your own source tree, like in a third_party directory. Personally, I'm a big fan of just taking what you need. A little copying is better than a little dependency[1]. [1] https://www.youtube.com/watch?v=PAAkCSZUG1c&t=9m28s https://www.youtube.com/watch?v=PAAkCSZUG1c&t=9m28s
    - deleted 5y ago
      [deleted]
    - Ericson2314 5y ago
      It's one a package contains it's dependencies, and often their dependencies too (transitive deps).' packages are supposed to be nodes in a dependency graph (or better, partial order), and there are nice composition rules. But one the nodes themselves represent graph/order closures (or almost-closure) we use that nice property. People that are very application oriented tend to like vendoring --- "my application is its own island, who gives a shit". But the distro's point isn't just gather the software --- that's an anachronism --- but plan a coherent whole. And applications who only care about themselves are terrible for that. The larger point is free and open source software really should be library-first. Siloed Applicationns are a terrible UI and holdover from the economics of proprietary shrink-wrapped software which itself is a skeuomorphism from the physical world of nonreplicable goods.
dathinab 5y ago
Additionally I would say at least `dbus`, `gnutls`, `libssh2` and maybe?? `curl` are this kind of libraries which count as "widely used system libraries". I.e. the few for which Linus thinks dynamic linking still can make sense ;=)
- chasil 5y ago
  Shared library and .DLL usage is like freedom, and writing a constitution. Where you can agree, you share. Where you cannot agree, you table and hold. Perhaps we are trying to share too much. I have removed shared libraries from internal code for similar reasons.
- bch 5y ago
  I think none of those (at least two, nearly surely) might qualify for that exemption. On the vast majority of systems I bet libcurl, for example, is probably linked against by 1 other piece of software - curl(1). “Really useful” isn’t the qualifier here. I bet ~same for libssh. dbus and gnutls I’d have to see more examples to understand.
  - throwaway525142 5y ago
    I think you misunderstand how widely used libcurl is. On Arch Linux, it has 291 reverse dependencies: https://archlinux.org/packages/core/x86_64/curl/ https://archlinux.org/packages/core/x86_64/curl/. It is e.g. used by git, cargo (Rust's package manager) and cmake (C++ build system). It's literally installed on 99.99% of all Arch Linux systems: https://www.archlinux.de/packages/core/x86_64/curl https://www.archlinux.de/packages/core/x86_64/curl.
    - ahartmetz 5y ago
      I think that the kind of "knowledge" you are correcting is informing much of the discussion here. And it seems like the motivation is "I want to make MY thing as easy to ship as possible!". There are hundreds of binaries in a Linux system and no one wants to rebuild all of them when an important library is updated.
lobocinza 5y ago
If I am to install Pandoc (on Arch Linux) I have to install 95 Haskell dependencies besides the ones I already have installed.
- encryptluks2 5y ago
  This is due to the maintainers on Arch and I've complained about it before too. They also update them almost daily so they just add noise to system updates. I just end up running the official Pandoc Docker container.
- bscphil 5y ago
  That's the result of a perverse maintainer, there used to be a version that didn't depend on the development libraries.
  - lobocinza 5y ago
    Fortunately there is pandoc-bin at the AUR.
    - bhrgunatha 5y ago
      I discovered that quite quickly after they switched. More importantly (for any haskell developers using Arch), there are also static builds for stack, cabal and ghcup too although you then need to maintain your own set of GHC and libraries. I can't imagine what it must be like to use the official packages since the haskell library ecosystem is quite fast moving.
    - ttsiodras 5y ago
      Thank you! I had noticed the same problem, but was not aware of this excellent solution. I just applied it, and it works perfectly... and I removed the dozens of Haskell libraries that were there just for Pandoc. For anyone else interested, the same solution applies to shellcheck - just install shellcheck-bin from AUR.
- drran 5y ago
  > If I am to install Pandoc (on Arch Linux) I have to install 95 Haskell dependencies OMG, you really spent 95x more time, by installing of dependencies one by one? Why not just use a package manager?
hnlmorg 5y ago
> Perhaps this is just a Linux distro thing, but as someone who closely monitors the Void packages repo, dynamic linking burdens distro maintainers with hundreds of additional packages which need to be individually tracked, tested and updated. > (Packages which could otherwise be vendored by upstream and statically linked during the build.) The effort is approximately the same for distro maintainers. Shared libraries might mean more individual packages but the same number of projects need to be actively maintained because if a vulnerability is patched in libssl (for example) you'd need to rebuild all projects that are statically linked rather than rebuilding libssl. Upstream vendoring doesn't really help here. If anything, it add more issues than it solves with regards to managing a distro where you want to ensure every package has the latest security patches (vendoring is more useful for teams managing specific applications to protect them against breaking changes happening upstream than it is at a distro level where you can bundle different versions of the same .so if you absolutely need to and link against those)
rualca 5y ago
> (...) dynamic linking burdens distro maintainers with hundreds of additional packages which need to be individually tracked, tested and updated. To me this assertion makes no sense at all, because the whole point of a distro is to bundle together more than hundreds of packages individually tracked, tested, and updated. By default. That's the whole point of a distro, and the reason developers target specific distros. A release is just a timestamp that specifies a set of packages bundled together, and when you target a distro you're targeting those hundred of packages that are shipped together.

rkeene2 5y ago

As someone who runs a Linux distribution, please don't vendor your dependencies for the benefit of distribution maintainers, it makes it much more difficult to build a coherent system. Running different versions of common libraries causes real annoyances.

Ericson2314 5y ago
Yes this can't be emphasized enough. static linking is fine, I don't really care either way. But please, please don't vendor. The point of software is to be composable, and vendoring kills that. Vendoring will cause free software to drown in its own complexity and is completely irresponsible.
- chris_wot 5y ago
  Could someone explain what is meant by “vendoring”?
  - jrockway 5y ago
    Vendoring is when you just copy the dependencies into your own source tree, like in a third_party directory. Personally, I'm a big fan of just taking what you need. A little copying is better than a little dependency[1]. [1] https://www.youtube.com/watch?v=PAAkCSZUG1c&t=9m28s https://www.youtube.com/watch?v=PAAkCSZUG1c&t=9m28s
  - deleted 5y ago
    [deleted]
  - Ericson2314 5y ago
    It's one a package contains it's dependencies, and often their dependencies too (transitive deps).' packages are supposed to be nodes in a dependency graph (or better, partial order), and there are nice composition rules. But one the nodes themselves represent graph/order closures (or almost-closure) we use that nice property. People that are very application oriented tend to like vendoring --- "my application is its own island, who gives a shit". But the distro's point isn't just gather the software --- that's an anachronism --- but plan a coherent whole. And applications who only care about themselves are terrible for that. The larger point is free and open source software really should be library-first. Siloed Applicationns are a terrible UI and holdover from the economics of proprietary shrink-wrapped software which itself is a skeuomorphism from the physical world of nonreplicable goods.

dathinab 5y ago

Additionally I would say at least `dbus`, `gnutls`, `libssh2` and maybe?? `curl` are this kind of libraries which count as "widely used system libraries". I.e. the few for which Linus thinks dynamic linking still can make sense ;=)

chasil 5y ago
Shared library and .DLL usage is like freedom, and writing a constitution. Where you can agree, you share. Where you cannot agree, you table and hold. Perhaps we are trying to share too much. I have removed shared libraries from internal code for similar reasons.
bch 5y ago
I think none of those (at least two, nearly surely) might qualify for that exemption. On the vast majority of systems I bet libcurl, for example, is probably linked against by 1 other piece of software - curl(1). “Really useful” isn’t the qualifier here. I bet ~same for libssh. dbus and gnutls I’d have to see more examples to understand.
- throwaway525142 5y ago
  I think you misunderstand how widely used libcurl is. On Arch Linux, it has 291 reverse dependencies: https://archlinux.org/packages/core/x86_64/curl/ https://archlinux.org/packages/core/x86_64/curl/. It is e.g. used by git, cargo (Rust's package manager) and cmake (C++ build system). It's literally installed on 99.99% of all Arch Linux systems: https://www.archlinux.de/packages/core/x86_64/curl https://www.archlinux.de/packages/core/x86_64/curl.
  - ahartmetz 5y ago
    I think that the kind of "knowledge" you are correcting is informing much of the discussion here. And it seems like the motivation is "I want to make MY thing as easy to ship as possible!". There are hundreds of binaries in a Linux system and no one wants to rebuild all of them when an important library is updated.

lobocinza 5y ago

If I am to install Pandoc (on Arch Linux) I have to install 95 Haskell dependencies besides the ones I already have installed.

encryptluks2 5y ago
This is due to the maintainers on Arch and I've complained about it before too. They also update them almost daily so they just add noise to system updates. I just end up running the official Pandoc Docker container.
bscphil 5y ago
That's the result of a perverse maintainer, there used to be a version that didn't depend on the development libraries.
- lobocinza 5y ago
  Fortunately there is pandoc-bin at the AUR.
  - bhrgunatha 5y ago
    I discovered that quite quickly after they switched. More importantly (for any haskell developers using Arch), there are also static builds for stack, cabal and ghcup too although you then need to maintain your own set of GHC and libraries. I can't imagine what it must be like to use the official packages since the haskell library ecosystem is quite fast moving.
  - ttsiodras 5y ago
    Thank you! I had noticed the same problem, but was not aware of this excellent solution. I just applied it, and it works perfectly... and I removed the dozens of Haskell libraries that were there just for Pandoc. For anyone else interested, the same solution applies to shellcheck - just install shellcheck-bin from AUR.
drran 5y ago
> If I am to install Pandoc (on Arch Linux) I have to install 95 Haskell dependencies OMG, you really spent 95x more time, by installing of dependencies one by one? Why not just use a package manager?

hnlmorg 5y ago

> Perhaps this is just a Linux distro thing, but as someone who closely monitors the Void packages repo, dynamic linking burdens distro maintainers with hundreds of additional packages which need to be individually tracked, tested and updated. > (Packages which could otherwise be vendored by upstream and statically linked during the build.) The effort is approximately the same for distro maintainers. Shared libraries might mean more individual packages but the same number of projects need to be actively maintained because if a vulnerability is patched in libssl (for example) you'd need to rebuild all projects that are statically linked rather than rebuilding libssl. Upstream vendoring doesn't really help here. If anything, it add more issues than it solves with regards to managing a distro where you want to ensure every package has the latest security patches (vendoring is more useful for teams managing specific applications to protect them against breaking changes happening upstream than it is at a distro level where you can bundle different versions of the same .so if you absolutely need to and link against those)

rualca 5y ago

> (...) dynamic linking burdens distro maintainers with hundreds of additional packages which need to be individually tracked, tested and updated. To me this assertion makes no sense at all, because the whole point of a distro is to bundle together more than hundreds of packages individually tracked, tested, and updated. By default. That's the whole point of a distro, and the reason developers target specific distros. A release is just a timestamp that specifies a set of packages bundled together, and when you target a distro you're targeting those hundred of packages that are shipped together.