5 ms·

I have some small critiques of some of the hyperbole in the article: "Package managers failed us due to shared libraries version differences causing dependency

I have some small critiques of some of the hyperbole in the article: "Package managers failed us due to shared libraries version differences causing dependency issues" Incorrect. The software administrators (read: The Users) failed to understand that installing duplicate incompatible software does not work, was never intended to happen, and shouldn't even be possible. But users are stubborn and will force a conflict if at all possible. Containers allow users to side-step package management. It doesn't replace it or help it at all, because it completely ignores all the work gone into the package. Imagine putting on tennis shoes, and then trying to put on snow boots. Containers give users a second pair of feet. And this is not a container innovation. Chroot environments have been providing the exact same functionality (installing side-by-side conflicting packaged software in a simple manner) for decades. You don't even need any extra software to use it. "Docker provides a self-contained image that is exactly that same image running on your laptop vs in the cloud while i.e. Puppet/Chef are procedural scripts that need to rerun to converge your cluster machines. This enables approaches also know as Immutable Infrastructure or Phoenix Deploys." Unless you designed your software to be immutable, it probably isn't. Software changes as it runs, and different hardware changes software differently, so at the best this claim is disingenuous. Different networks and systems interacting in different locations add complications. If you tested it on your laptop, do not expect it to run the same in production, period. "Before Docker, LXC would create a full copy of FileSystem when creating a container. This would be slow and take up a lot of space." Loop and COW filesystems (Unionfs, Aufs, Overlayfs, etc) on Linux pre-date Docker by a long time, and were used with containers and container alternatives. -- I thought i'd see more about linux container internals, not a description of how Docker works, but I guess the host name should have been a dead giveaway. Don't read this if you want to know about the kernel.

so0k 10y ago
Totally agree, the post was written almost a year ago across 3 days while preparing a talk (see intro) and while still learning about Docker. Thanks for the feedback :)
- deleted 10y ago
  [deleted]
digi_owl 10y ago
Nope, It is partially right. There already exist mechanics like soname to differentiate libraries from each other. Package managers do not take this into account, and insist on there only being one version present pr package name. That said, containers are shooting twee twee birds with AA guns.
- xorcist 10y ago
  > insist on there only being one version present pr package name That's a partial truth, at best. It is common for multiple versions of popular libraries to be installed at the same time. The whole of Debian or Red Hat isn't necessarily compiled with the same libcpp or Boost, for example, and that's expected. Some software run on Python 2 and some on 3. The packager would need to make sure they don't conflict, but Linux handles different sonames just fine and as long as you separate module paths you'll be just fine.
  - digi_owl 10y ago
    No it is not. If you want to install two minor versions of glib side by side, it can be done on the soname level (not to say that it is a smart idea, because Gnome devs are crap at keeping API stable). But it can't be done on the package level without playing musical chairs with package naming to work around collisions.
    - snuxoll 10y ago
      This is primarily why Debian and derivatives use the so version in the package name. Fedora is designed on the assumption you'll only ever have one version of a library installed and everything is built against it, special cases then get made for compat packages as needed.
      - digi_owl 10y ago
        Sadly more and more dev time is spent with Fedora as the target, leading to massive myopia. This while Debian has to adopt more and more Fedora-isms because they do no longer have the man hours to go their own way.
emmelaich 10y ago
You're going to far the the other way. > The Users) failed to understand that installing > duplicate incompatible software does not work I still think that means package managers failed, if only because that's the perception - that package management solves more that it actually does.
geofft 10y ago
> Incorrect. The software administrators (read: The Users) failed to understand that installing duplicate incompatible software does not work, was never intended to happen, and shouldn't even be possible. But users are stubborn and will force a conflict if at all possible. Why doesn't it work? By whom was it never intended to happen? Why should it not even be possible? I've shipped production software that - very carefully - links multiple versions of OpenSSL within the same process, so it's not a matter of some law of physics that I can't have two versions of OpenSSL on my system used by separate binaries. It's a design choice that this is how things are going to work. You don't need containers to pick a different design choice, yes, but neither do you need chroots - just careful use of shared library versioning and symbol versioning. Containers won because containerization tools made all of this easy. Nobody wants to piece together shell scripts to do things in chroots any more than they want to piece together shell scripts to set LD_LIBRARY_PATHs. (And way more commercial software actually does the latter, because they want to side-step package management because they have no idea what libraries are on your system.)
- peterwwillis 10y ago
  * > Why doesn't it work? By whom was it never intended to happen? Why should it not even be possible?* It doesn't work because it's incompatible, and so it's complicated. If I build A with B1, and you build C with B1.1, and the user wants both A and C, they need both B1 and B1.1. Which is fine - IF they built B* with unique symbol names, and built their apps against those unique symbol names, and if everyone else in the world follows exactly the same convention. Of course, if anything else changes (cpu architecture, features, ABI, whatever) everything may break anyway. But in general the biggest problem is not everyone builds software the same way. Both the software developers and the package managers never intended for incompatible software to be installed at the same time. The software devs could make it handle these cases, but they usually don't, so it doesn't work. The package managers could package their software uniquely every time, but that would be annoying, cumbersome and not very useful for managing systems ("do i need to remove db3 before i install db4? what are all the packages called? what's the order? what else will be affected? do i rename everything and rebuild everything with names specific to this one library package name?" etc). It shouldn't be possible to install conflicting software because the package should be built to fail to install if conflicting software exists, or remove the conflicting software before install. But sadly there also exists the ability to remove all these safeguards, or to install unpackaged software. Containers are just a wrapper around existing tools, such as package managers. They don't add functionality, they just simplify it. With Docker, you aren't linking to multiple versions of openssl within the same process: you're running one process in one environment with one version of openssl, unless you intentionally get really fancy, which really isn't easy. Package managers never failed, they simply weren't being used right. Containers won because someone finally realized users don't care how they do what they want, as long as they get to do it without having to know how it actually works. Devs get to pretend they know how to deploy software or manage systems and Ops people get less responsibility because they didn't build the shit so they don't support it. It's a win-win, but it's still a mess, and none of it is new or novel.

so0k 10y ago

Totally agree, the post was written almost a year ago across 3 days while preparing a talk (see intro) and while still learning about Docker. Thanks for the feedback :)

deleted 10y ago
[deleted]

digi_owl 10y ago

Nope, It is partially right. There already exist mechanics like soname to differentiate libraries from each other. Package managers do not take this into account, and insist on there only being one version present pr package name. That said, containers are shooting twee twee birds with AA guns.

xorcist 10y ago
> insist on there only being one version present pr package name That's a partial truth, at best. It is common for multiple versions of popular libraries to be installed at the same time. The whole of Debian or Red Hat isn't necessarily compiled with the same libcpp or Boost, for example, and that's expected. Some software run on Python 2 and some on 3. The packager would need to make sure they don't conflict, but Linux handles different sonames just fine and as long as you separate module paths you'll be just fine.
- digi_owl 10y ago
  No it is not. If you want to install two minor versions of glib side by side, it can be done on the soname level (not to say that it is a smart idea, because Gnome devs are crap at keeping API stable). But it can't be done on the package level without playing musical chairs with package naming to work around collisions.
  - snuxoll 10y ago
    This is primarily why Debian and derivatives use the so version in the package name. Fedora is designed on the assumption you'll only ever have one version of a library installed and everything is built against it, special cases then get made for compat packages as needed.
    - digi_owl 10y ago
      Sadly more and more dev time is spent with Fedora as the target, leading to massive myopia. This while Debian has to adopt more and more Fedora-isms because they do no longer have the man hours to go their own way.

emmelaich 10y ago

You're going to far the the other way. > The Users) failed to understand that installing > duplicate incompatible software does not work I still think that means package managers failed, if only because that's the perception - that package management solves more that it actually does.

geofft 10y ago

> Incorrect. The software administrators (read: The Users) failed to understand that installing duplicate incompatible software does not work, was never intended to happen, and shouldn't even be possible. But users are stubborn and will force a conflict if at all possible. Why doesn't it work? By whom was it never intended to happen? Why should it not even be possible? I've shipped production software that - very carefully - links multiple versions of OpenSSL within the same process, so it's not a matter of some law of physics that I can't have two versions of OpenSSL on my system used by separate binaries. It's a design choice that this is how things are going to work. You don't need containers to pick a different design choice, yes, but neither do you need chroots - just careful use of shared library versioning and symbol versioning. Containers won because containerization tools made all of this easy. Nobody wants to piece together shell scripts to do things in chroots any more than they want to piece together shell scripts to set LD_LIBRARY_PATHs. (And way more commercial software actually does the latter, because they want to side-step package management because they have no idea what libraries are on your system.)

peterwwillis 10y ago
* > Why doesn't it work? By whom was it never intended to happen? Why should it not even be possible?* It doesn't work because it's incompatible, and so it's complicated. If I build A with B1, and you build C with B1.1, and the user wants both A and C, they need both B1 and B1.1. Which is fine - IF they built B* with unique symbol names, and built their apps against those unique symbol names, and if everyone else in the world follows exactly the same convention. Of course, if anything else changes (cpu architecture, features, ABI, whatever) everything may break anyway. But in general the biggest problem is not everyone builds software the same way. Both the software developers and the package managers never intended for incompatible software to be installed at the same time. The software devs could make it handle these cases, but they usually don't, so it doesn't work. The package managers could package their software uniquely every time, but that would be annoying, cumbersome and not very useful for managing systems ("do i need to remove db3 before i install db4? what are all the packages called? what's the order? what else will be affected? do i rename everything and rebuild everything with names specific to this one library package name?" etc). It shouldn't be possible to install conflicting software because the package should be built to fail to install if conflicting software exists, or remove the conflicting software before install. But sadly there also exists the ability to remove all these safeguards, or to install unpackaged software. Containers are just a wrapper around existing tools, such as package managers. They don't add functionality, they just simplify it. With Docker, you aren't linking to multiple versions of openssl within the same process: you're running one process in one environment with one version of openssl, unless you intentionally get really fancy, which really isn't easy. Package managers never failed, they simply weren't being used right. Containers won because someone finally realized users don't care how they do what they want, as long as they get to do it without having to know how it actually works. Devs get to pretend they know how to deploy software or manage systems and Ops people get less responsibility because they didn't build the shit so they don't support it. It's a win-win, but it's still a mess, and none of it is new or novel.