4 ms·

Slightly off topic, but I'm really frustrated with the state of software packaging these days. Every solution tries to solve the same problems (reliable dependa

by edude03 11y ago

Slightly off topic, but I'm really frustrated with the state of software packaging these days. Every solution tries to solve the same problems (reliable dependancy resolution, reliable installation and removal) but they all try and solve it in different incompatible and flawed ways. Consider the following: I need a reliable way to deploy my app to clients. The popular options are: 1) Source code + Configuration system (automake for example) 2) Binaries built for popular platforms (debs, rpms, exes) 3) Docker Image Source code means that the client needs the whole build tool chain, which might be quite + computationally expensive (especially on mobile) packages have to be built and maintained, and don't fully solve the dependancy issues (IE I might expect Ubuntu 14.04 to have a specific version of libc, but the user might have upgraded it, and I can't install my own incompatible version) and of course docker, ship my clients a operating system and require them to have and know how to configure the runtime so they can run my 1mb compiled binary. Also doesn't work on embedded devices. Ideally, in the future most distributions will move to functional package managers, and at least for mobile will have binaries for every possible version of dependencies available, but at the moment that's just a pipe dream and things like snappy don't get us any closer to that

hasenj 11y ago
I think Docker is meant for dev ops, not for casual users.
- edude03 11y ago
  It's supposed to be for devops, but in practice developers use docker as a way to solve dependency management on the server more so than to seperate instances of apps from each other. Especially with "microservices" it makes more sense to have docker available on the server and allow each team to upload a image with all the deps already baked in.
  - falcolas 11y ago
    Yup. The tradeoff is that in operations we then need to work around all of the inherit disadvantages of running docker containers (network slowness, disk slowdown, logging, etc) to keep their systems performant.
ant6n 11y ago
You mean sort of like npm?
- IgorPartola 11y ago
  I wouldn't use npm as an example of a stellar package manager.
  - ant6n 11y ago
    Not using it as an example of a 'stellar' package manager. Just one where the dependencies can be defined with version. I was under the impression that apt-get will just use the most recent version.
    - IgorPartola 11y ago
      apt-get/dpkg do have the ability to specify package versions, including specifying versions equal to, greater than, or less than. It also has the ability to specify build time and run time dependencies separately.
    - slavik81 11y ago
      You can specify a version. The real trouble is that you generally cannot have multiple versions of the same package installed on the same system. Thus, if multiple packages specify different versions as dependencies, you're hooped.
      - ant6n 11y ago
        So that is something npm does, AFAIK. Isn't it even possible to have multiple different versions of a sub package in the dependency tree of one project?
- edude03 11y ago
  npm is an example of a bad solution to dep management. So I want you to run my 6kb node.js script you need to: 1) Build obtain node for your platform (and now you've hit the exact problem I'm talking about) 2) install npm (which I can't really define as a dep, you're just expected to have it) 3) install all the deps I've defined and hope they all work on the version / configuration of node you have (execjs only works on node 0.10 but you have v4? Too bad! You didn't apply the increased memory patch before building node? Too bad!) The ideal here to reiterate is given nothing but your installed OS, I should be able to give you a file that will go from nothing to a running app, and from the running app to a clean system if you decide to remove it, without you having to have any prerequisites (no node, no chef, no docker, etc)
davexunit 11y ago
I agree that functional package management is the way forward. Having a system that contains the precise details of how to go from source code to binary, in a completely automated fashion, for the entire set of packages all the way down to the bootstrap binaries is a critical foundation to build upon.
jhasse 11y ago
There are solutions to this problem (e. g. ZeroInstall [1]), but the distributions don't support them, since they want to keep you vendor locked-in. Guess why Ubuntu is developing Snappy and not using xdg-app [2]? Also purely functional package managers exist [3], distributions just ignore them. It's hard to use something not based on deb or rpm these days. [1] http://0install.net http://0install.net [2] https://wiki.gnome.org/Projects/SandboxedApps https://wiki.gnome.org/Projects/SandboxedApps [3] https://nixos.org/nix/ https://nixos.org/nix/
- vezzy-fnord 11y ago
  I find it hilarious how you make the implication that Snappy is some deliberate NIH of xdg-app, when xdg-app itself is GNOME/Red Hat's NIH of Nix. If anything, we should be lauding Canonical for keeping to themselves in their corner instead of seeking greater validity across the ecosystem with half-baked ideas.
VLM 11y ago
Seems to summarize down to a love/hate relationship with shared libraries. Most of the flaws are best explained as the devs, admins, users, security team, and distro maintainers sometimes having different positions on the spectrum of love/hate for shared libraries. Also don't forget architecture... you want that binary in 32 bit i386, 64 bit amd64, or arm?
- edude03 11y ago
  In a nutshell yes. The binary argument is true as well, however many developers don't want to make their sources avaialbe which means other than going source -> some obfuscated IR -> native, thre is no way to allow the end user to run your app on any arbitrary platform.
  - vezzy-fnord 11y ago
    There is. It's called fat binaries. Ryan C. Gordon was working on an ELF extension for it, but ended up quitting due to getting shit on by kernel hackers and common users alike.
ordinary 11y ago
> packages [...] don't fully solve the dependancy issues (IE I might expect Ubuntu 14.04 to have a specific version of libc, but the user might have upgraded it, and I can't install my own incompatible version) I feel that once a user circumvents the package system, it is unreasonable for that user to expect that the package system will continue to work correctly. In that case, the user has taken over some of the responsibilies of the package system, and software vendors should feel no obligation to ensure compatibility with such changes.
- edude03 11y ago
  I don't disagree, and of course swapping out libc might be an extreme example, but my point still stands - ideally I shouldn't have to rely on any specific dependencies being in place to deploy my software.
  - binarycrusader 11y ago
    Some amount of dependencies is a reasonable expectation; for example, requiring a certain minimum version of libc is entirely reasonable. Or, requiring a certain minimum version of the kernel. However, if newer versions of libc or the kernel are not backwards-compatible, that's a failure in the system itself (not the packaging). (Linus, as an example, has a strict rule about not breaking userland applications due to kernel changes.) It's impractical (and undesirable from a security and other standpoint) to statically link all dependencies into every binary simply to "simplify" packaging. Not only would that be a waste of storage space and memory, but it prevents fixes (security or otherwise) applied to the libraries used by the application from taking effect without patching everything that statically-linked it. Even if you ignore the storage space issue with some amount of hand-waving, larger binaries means greater I/O and network bandwidth requirements, which means it takes longer to update systems or create images for deployment. That in turn leads to increased downtime or increased sustaining costs, which effectively leads to reduced availability of the system. This can be mitigated somewhat by using difference-only-style update mechanisms, but it only partially mitigates the issues created by statically-linked binaries; it does not eliminate them. Dynamically-linking dependencies also ensures that future performance and security updates don't require a rebuild of existing applications; you can simply update the dependency, and if done properly, every application linked to it will automatically receive the benefits. Obviously, if a given dependency is only ever used by a single component, and the component is always rebuilt when the dependencies are, it doesn't matter if they're statically-linked. I'm referring only to the system dependencies common to many system components (such as libc). In the end, a carefully chosen set of dependencies that provide robust backwards-compatibility provides the least downtime and the most compelling administrative experience.
pnathan 11y ago
I feel your pain. I'm trying to figure out how to get some code with some unique dependencies shipping to a wide range of users (both in skills and platform), and I don't even know of a good (multiplatform) solution that doesn't look like "bake it all into a static zip and unzip into /opt" or "run a docker container". I need to think about that a bit more instead of solving the actual product problem at hand. :-/
fulafel 11y ago
You will be fine if the user installs a newer libc. Binaries and .debs linked against libc from 10 years ago work and install fine.
deleted 11y ago
[deleted]

by edude03 11y ago

hasenj 11y ago

I think Docker is meant for dev ops, not for casual users.

edude03 11y ago
It's supposed to be for devops, but in practice developers use docker as a way to solve dependency management on the server more so than to seperate instances of apps from each other. Especially with "microservices" it makes more sense to have docker available on the server and allow each team to upload a image with all the deps already baked in.
- falcolas 11y ago
  Yup. The tradeoff is that in operations we then need to work around all of the inherit disadvantages of running docker containers (network slowness, disk slowdown, logging, etc) to keep their systems performant.

ant6n 11y ago

You mean sort of like npm?

IgorPartola 11y ago
I wouldn't use npm as an example of a stellar package manager.
- ant6n 11y ago
  Not using it as an example of a 'stellar' package manager. Just one where the dependencies can be defined with version. I was under the impression that apt-get will just use the most recent version.
  - IgorPartola 11y ago
    apt-get/dpkg do have the ability to specify package versions, including specifying versions equal to, greater than, or less than. It also has the ability to specify build time and run time dependencies separately.
  - slavik81 11y ago
    You can specify a version. The real trouble is that you generally cannot have multiple versions of the same package installed on the same system. Thus, if multiple packages specify different versions as dependencies, you're hooped.
    - ant6n 11y ago
      So that is something npm does, AFAIK. Isn't it even possible to have multiple different versions of a sub package in the dependency tree of one project?
edude03 11y ago
npm is an example of a bad solution to dep management. So I want you to run my 6kb node.js script you need to: 1) Build obtain node for your platform (and now you've hit the exact problem I'm talking about) 2) install npm (which I can't really define as a dep, you're just expected to have it) 3) install all the deps I've defined and hope they all work on the version / configuration of node you have (execjs only works on node 0.10 but you have v4? Too bad! You didn't apply the increased memory patch before building node? Too bad!) The ideal here to reiterate is given nothing but your installed OS, I should be able to give you a file that will go from nothing to a running app, and from the running app to a clean system if you decide to remove it, without you having to have any prerequisites (no node, no chef, no docker, etc)

davexunit 11y ago

I agree that functional package management is the way forward. Having a system that contains the precise details of how to go from source code to binary, in a completely automated fashion, for the entire set of packages all the way down to the bootstrap binaries is a critical foundation to build upon.

jhasse 11y ago

There are solutions to this problem (e. g. ZeroInstall [1]), but the distributions don't support them, since they want to keep you vendor locked-in. Guess why Ubuntu is developing Snappy and not using xdg-app [2]? Also purely functional package managers exist [3], distributions just ignore them. It's hard to use something not based on deb or rpm these days. [1] http://0install.net http://0install.net [2] https://wiki.gnome.org/Projects/SandboxedApps https://wiki.gnome.org/Projects/SandboxedApps [3] https://nixos.org/nix/ https://nixos.org/nix/

vezzy-fnord 11y ago
I find it hilarious how you make the implication that Snappy is some deliberate NIH of xdg-app, when xdg-app itself is GNOME/Red Hat's NIH of Nix. If anything, we should be lauding Canonical for keeping to themselves in their corner instead of seeking greater validity across the ecosystem with half-baked ideas.

VLM 11y ago

Seems to summarize down to a love/hate relationship with shared libraries. Most of the flaws are best explained as the devs, admins, users, security team, and distro maintainers sometimes having different positions on the spectrum of love/hate for shared libraries. Also don't forget architecture... you want that binary in 32 bit i386, 64 bit amd64, or arm?

edude03 11y ago
In a nutshell yes. The binary argument is true as well, however many developers don't want to make their sources avaialbe which means other than going source -> some obfuscated IR -> native, thre is no way to allow the end user to run your app on any arbitrary platform.
- vezzy-fnord 11y ago
  There is. It's called fat binaries. Ryan C. Gordon was working on an ELF extension for it, but ended up quitting due to getting shit on by kernel hackers and common users alike.

ordinary 11y ago

> packages [...] don't fully solve the dependancy issues (IE I might expect Ubuntu 14.04 to have a specific version of libc, but the user might have upgraded it, and I can't install my own incompatible version) I feel that once a user circumvents the package system, it is unreasonable for that user to expect that the package system will continue to work correctly. In that case, the user has taken over some of the responsibilies of the package system, and software vendors should feel no obligation to ensure compatibility with such changes.

edude03 11y ago
I don't disagree, and of course swapping out libc might be an extreme example, but my point still stands - ideally I shouldn't have to rely on any specific dependencies being in place to deploy my software.
- binarycrusader 11y ago
  Some amount of dependencies is a reasonable expectation; for example, requiring a certain minimum version of libc is entirely reasonable. Or, requiring a certain minimum version of the kernel. However, if newer versions of libc or the kernel are not backwards-compatible, that's a failure in the system itself (not the packaging). (Linus, as an example, has a strict rule about not breaking userland applications due to kernel changes.) It's impractical (and undesirable from a security and other standpoint) to statically link all dependencies into every binary simply to "simplify" packaging. Not only would that be a waste of storage space and memory, but it prevents fixes (security or otherwise) applied to the libraries used by the application from taking effect without patching everything that statically-linked it. Even if you ignore the storage space issue with some amount of hand-waving, larger binaries means greater I/O and network bandwidth requirements, which means it takes longer to update systems or create images for deployment. That in turn leads to increased downtime or increased sustaining costs, which effectively leads to reduced availability of the system. This can be mitigated somewhat by using difference-only-style update mechanisms, but it only partially mitigates the issues created by statically-linked binaries; it does not eliminate them. Dynamically-linking dependencies also ensures that future performance and security updates don't require a rebuild of existing applications; you can simply update the dependency, and if done properly, every application linked to it will automatically receive the benefits. Obviously, if a given dependency is only ever used by a single component, and the component is always rebuilt when the dependencies are, it doesn't matter if they're statically-linked. I'm referring only to the system dependencies common to many system components (such as libc). In the end, a carefully chosen set of dependencies that provide robust backwards-compatibility provides the least downtime and the most compelling administrative experience.

pnathan 11y ago

I feel your pain. I'm trying to figure out how to get some code with some unique dependencies shipping to a wide range of users (both in skills and platform), and I don't even know of a good (multiplatform) solution that doesn't look like "bake it all into a static zip and unzip into /opt" or "run a docker container". I need to think about that a bit more instead of solving the actual product problem at hand. :-/

fulafel 11y ago

You will be fine if the user installs a newer libc. Binaries and .debs linked against libc from 10 years ago work and install fine.

deleted 11y ago

[deleted]