13 ms·

The main argument for using shared libraries isn’t memory or disk usage, but simply security. If you have a thousand packages linking statically against zlib,

The main argument for using shared libraries isn’t memory or disk usage, but simply security. If you have a thousand packages linking statically against zlib, you will have to update a thousand packages in case of a vulnerability. With a shared zlib, you will have to update only one package.

t-3 2y ago
If a vulnerability in a single library can cause security issues in more than one package, there are much more serious issues to consider with regards to that library than the need to recompile 1000 dependents. The monetary/energy/time savings of being able to update libraries without having to rebuild dependents are of far greater significance than the theoretical improvement in security.
sunshowers 2y ago
The solution here is to build tooling to track dependencies in statically linked binaries. There is no inherent reason that has to be tightly coupled to the dynamic dispatch model of shared objects. (In other words, the current situation is not an inherent fact about packaging. Rather, it is path-dependent.) For instance, many modern languages use techniques that are simply incompatible with dynamic dispatch. Some languages like Swift have focused on dynamic dispatch, but mostly because it was a fundamental requirement placed on their development teams by executives. While there is a place for dynamic dispatch in software, there is also no inherent justification for dynamic dispatch boundaries to be exactly at organizational ones. (For example, there is no inherent justification for the dynamic dispatch boundary to be exactly at the places a binary calls into zlib.) edit: I guess loading up a .so is more commonly called "dynamic binding". But it is fundamentally dynamic dispatch, ie figuring out what version of a function to call at runtime.
imoverclocked 2y ago
> The main argument for using shared libraries isn’t memory or disk usage, but simply security. “The” main argument? In a world filled with diverse concerns, there isn’t just one argument that makes a decision. Additionally, security is one of those things where practically everything is a trade off. Eg: by having lots of things link against a single shared library, that library becomes a juicy target. > With a shared zlib, you will have to update only one package. We are back to efficiency :)
Arnavion 2y ago
That's not a difference of security, only of download size (*). A statically-linking distro would queue up those thousand packages to be rebuilt and you would receive them via OS update, just as you would with a dynamically-linking distro. The difference is just in whether you have to download 1000 updated packages or 1. And on a distribution like OpenSUSE TW, the automation is set up such that those thousand packages do get rebuilt anyway, even though they're dynamically linked. (*): ... and build time on the distro package builders, of course.
- singron 2y ago
  The build time is pretty significant actually. E.g. NixOS takes this to the extreme and always rebuilds packages if their dependencies change. It can take days to rebuild affected packages, and you typically can't easily update in the meantime. This is worse if the rebuild breaks something since someone will have to fix or revert that before it can succeed. For non-security updates, they can do the rebuild in a branch (staging) so it's non-blocking, but you will feel the pain on a security update. In practice, users will apply a config mitigation if available or "graft" executables against an updated lib using system.replaceDependencies, which basically search and replaced the built artifacts to use a different file path.
  - Arnavion 2y ago
    True. I'm involved in Alpine, postmarketOS and OpenSUSE packaging and I've seen builders of the first two become noticeably slow on mass rebuilds. OpenSUSE tends to be fine, but it has a lot of builders so that's probably why.
  - JackSlateur 2y ago
    It shall be noted that there is no need to recompile everything, when a library's patched: you only need to relink the executables and rebuild the packages Still, this requires a somewhat bigger infrastructure
- robert_foss 2y ago
  Security + build time isn't the issue, but security + dev/testing time is. Maintaining a secure package of zlib takes linearly more time with more versions of it used. All distros are manpower limited.
nmz 2y ago
Does static compilation use the entire library instead of just the parts that are used? If I'm just using a single function from this library, why include everything?
pizlonator 2y ago
That's also a good argument for shared libraries, but the memory usage argument is a big one. I have 583 processes running on my Linux box right now (I'm posting from FF running on GNOME on Pop!_OS), and it's nice that when they run the same code (like zlib but also lots of things that are bigger than zlib), they load the same shared library, which means they are using the same mapped file and so the physical memory is shared. It means that memory usage due to code scales with the amount of code (linear), not with the amount of processes times the amount of code (quadratic). I think that having a sophisticated desktop OS where every process had distinct physical memory for duplicated uses of the same code would be problematic at scale, especially on systems with less RAM. At least that physical memory would still be disk-backed, but that only goes so far. That said, the security argument is also a good argument!
- MrDrMcCoy 2y ago
  Kernel samepage merging and zram/zswap are the answers to the memory usage issue. The only actual issues with static linking is that it's harder to ensure that bugs and security vulnerabilites get patched.
  - yjftsjthsd-h 2y ago
    I disagree: - KSM only works on pages that are identical; I am skeptical that it can actually identify 2 instances of a lib after it's been through an optimizing compiler+linker (mostly LTO) - KSM has performance overhead - zram/zswap only let you reduce the impact of running out of memory at the cost of performance; you're still swapping, no matter how much we improve the details, so it'll always be worse than outright deduplicating libraries
    - MrDrMcCoy 2y ago
      Fair points overall. I think KSM is still useful and worth the overhead for things like sprawling browser tabs and densely deployed microservices. I disagree that swap is only (or even primarily) for situations where you're running out of memory, though. It is useful for reducing I/O contention and improving memory utilization by replacing underutilized pages with file cache when it makes sense to do so. See this for more details: https://chrisdown.name/2018/01/02/in-defence-of-swap.html https://chrisdown.name/2018/01/02/in-defence-of-swap.html

t-3 2y ago

If a vulnerability in a single library can cause security issues in more than one package, there are much more serious issues to consider with regards to that library than the need to recompile 1000 dependents. The monetary/energy/time savings of being able to update libraries without having to rebuild dependents are of far greater significance than the theoretical improvement in security.

sunshowers 2y ago

The solution here is to build tooling to track dependencies in statically linked binaries. There is no inherent reason that has to be tightly coupled to the dynamic dispatch model of shared objects. (In other words, the current situation is not an inherent fact about packaging. Rather, it is path-dependent.) For instance, many modern languages use techniques that are simply incompatible with dynamic dispatch. Some languages like Swift have focused on dynamic dispatch, but mostly because it was a fundamental requirement placed on their development teams by executives. While there is a place for dynamic dispatch in software, there is also no inherent justification for dynamic dispatch boundaries to be exactly at organizational ones. (For example, there is no inherent justification for the dynamic dispatch boundary to be exactly at the places a binary calls into zlib.) edit: I guess loading up a .so is more commonly called "dynamic binding". But it is fundamentally dynamic dispatch, ie figuring out what version of a function to call at runtime.

imoverclocked 2y ago

> The main argument for using shared libraries isn’t memory or disk usage, but simply security. “The” main argument? In a world filled with diverse concerns, there isn’t just one argument that makes a decision. Additionally, security is one of those things where practically everything is a trade off. Eg: by having lots of things link against a single shared library, that library becomes a juicy target. > With a shared zlib, you will have to update only one package. We are back to efficiency :)

Arnavion 2y ago

That's not a difference of security, only of download size (*). A statically-linking distro would queue up those thousand packages to be rebuilt and you would receive them via OS update, just as you would with a dynamically-linking distro. The difference is just in whether you have to download 1000 updated packages or 1. And on a distribution like OpenSUSE TW, the automation is set up such that those thousand packages do get rebuilt anyway, even though they're dynamically linked. (*): ... and build time on the distro package builders, of course.

singron 2y ago
The build time is pretty significant actually. E.g. NixOS takes this to the extreme and always rebuilds packages if their dependencies change. It can take days to rebuild affected packages, and you typically can't easily update in the meantime. This is worse if the rebuild breaks something since someone will have to fix or revert that before it can succeed. For non-security updates, they can do the rebuild in a branch (staging) so it's non-blocking, but you will feel the pain on a security update. In practice, users will apply a config mitigation if available or "graft" executables against an updated lib using system.replaceDependencies, which basically search and replaced the built artifacts to use a different file path.
- Arnavion 2y ago
  True. I'm involved in Alpine, postmarketOS and OpenSUSE packaging and I've seen builders of the first two become noticeably slow on mass rebuilds. OpenSUSE tends to be fine, but it has a lot of builders so that's probably why.
- JackSlateur 2y ago
  It shall be noted that there is no need to recompile everything, when a library's patched: you only need to relink the executables and rebuild the packages Still, this requires a somewhat bigger infrastructure
robert_foss 2y ago
Security + build time isn't the issue, but security + dev/testing time is. Maintaining a secure package of zlib takes linearly more time with more versions of it used. All distros are manpower limited.

nmz 2y ago

Does static compilation use the entire library instead of just the parts that are used? If I'm just using a single function from this library, why include everything?

pizlonator 2y ago

That's also a good argument for shared libraries, but the memory usage argument is a big one. I have 583 processes running on my Linux box right now (I'm posting from FF running on GNOME on Pop!_OS), and it's nice that when they run the same code (like zlib but also lots of things that are bigger than zlib), they load the same shared library, which means they are using the same mapped file and so the physical memory is shared. It means that memory usage due to code scales with the amount of code (linear), not with the amount of processes times the amount of code (quadratic). I think that having a sophisticated desktop OS where every process had distinct physical memory for duplicated uses of the same code would be problematic at scale, especially on systems with less RAM. At least that physical memory would still be disk-backed, but that only goes so far. That said, the security argument is also a good argument!

MrDrMcCoy 2y ago
Kernel samepage merging and zram/zswap are the answers to the memory usage issue. The only actual issues with static linking is that it's harder to ensure that bugs and security vulnerabilites get patched.
- yjftsjthsd-h 2y ago
  I disagree: - KSM only works on pages that are identical; I am skeptical that it can actually identify 2 instances of a lib after it's been through an optimizing compiler+linker (mostly LTO) - KSM has performance overhead - zram/zswap only let you reduce the impact of running out of memory at the cost of performance; you're still swapping, no matter how much we improve the details, so it'll always be worse than outright deduplicating libraries
  - MrDrMcCoy 2y ago
    Fair points overall. I think KSM is still useful and worth the overhead for things like sprawling browser tabs and densely deployed microservices. I disagree that swap is only (or even primarily) for situations where you're running out of memory, though. It is useful for reducing I/O contention and improving memory utilization by replacing underutilized pages with file cache when it makes sense to do so. See this for more details: https://chrisdown.name/2018/01/02/in-defence-of-swap.html https://chrisdown.name/2018/01/02/in-defence-of-swap.html