4 ms·

You're not wrong, exactly, but what ends up being brutal is not just packaging, it's packaging and supporting dependency management + post-install configuration

You're not wrong, exactly, but what ends up being brutal is not just packaging, it's packaging and supporting dependency management + post-install configuration. RPMs, for example, are pretty terrible at allowing you to specify custom configuration parameters, and you can't use them at all to install something that you don't have root privileges to access. Custom installers also allow you to have a lot more flexibility in terms of saying, "Okay, let's look to see if there's a mysql instance installed on the system, there's not, so rather than installing an OS one, we'll go ahead and install our preferred mysql version and treat it as an embedded db". RPM-driven upgrades also end up being a lot messier, since you can't do templated or merged configuration files easily. I'm not saying that the curl | bash strategy is good either, but forcing OS packages down people's throats isn't ideal for various situations. EDIT: also, you can't do multiple installs of the same RPM package on a system. So even if you wanted to install 5 instances of app X on a machine, possibly for testing reasons, you can't, and you end up having to use VMs or containers or hacking something together.

mattzito 11y ago
Gosh, and let's not forget that RPMs can also contain pre- and post- hook scripts, so you could be in a situation where you say, "Install this RPM and satisfy dependencies", and another RPM is downloaded and installed as root, with you never seeing the scripts that are going to be run as part of that process. You can inspect the scripts by hand by examining the package files, but in some ways it's less transparent than downloading a script and running that.
- IgorPartola 11y ago
  You are downloading software from the internet. How it executes code on your machine is irrelevant, you already trust it. Or do you review every OS package you install for shenanigans on every update? This is an especially bad point to bring up in the context Sandstorm, which runs stuff as root all the time. Do you mean to say that you trust them enough to run a bunch of stuff as root on your server, but not to run a post-install script?
  - tokenizerrr 11y ago
    No, but I do trust some software enough to run as non-root. Sadly to install this non-root software I need to install the RPM as root, and then I have no idea what it's going to do during the pre- and post- hook scripts.
    - icebraining 11y ago
      I don't know about RPM, but for deb you can just do "dpkg-deb --control <deb file>" to see those scripts.
    - brazzledazzle 11y ago
      You can export the scripts from an rpm without installing it.
    - IgorPartola 11y ago
      I suppose. Though have you considered the fact that if someone has non-root local access to your machine, they are just one local privilege escalation away form becoming root? And these things come out fairly frequently that you can pretty much count on one being found soon. In other words IMO you should either trust or not trust a piece of code to run on your machine. The distinction between root and non-root user running the code is more for maintainability and sanity and less for actual security.
- rkeene2 11y ago
  AppFS ( http://appfs.rkeene.org/ http://appfs.rkeene.org/ ) was created for this purpose. Everything is run as an unprivileged user because there is no install phase. You just run the software and the parts it needs are fetched lazily/JIT. Also because it uses PKI end-to-end, you trust the publisher of the application (as in signed RPM/DEB) not the site/mirror you are fetching it from (as in curl|bash).
IgorPartola 11y ago
The thing is that most of these types of projects have very rigid requirements for the OS they support. It's often not just Debian-like or RPM based, it's something like Ubuntu 14.04 64-bit only. If you are already doing that, you already bought into the OS. Support it properly. RPM is not my favorite format. I much prefer Debian-based distros specifically because dpkg and apt are so robust. I have never had a problem with managing configuration files there. You either provide a sane default config, put a config in /usr/share/doc/your-package/, or provide a configuration utility. All that works just fine. And you absolutely should use the system's packages for your dependencies. When you install your own copy of MySQL, nginx, or anything else, you assume responsibility for all upgrades to those subsystems. If next week MySQL releases a patch for remote code execution vulnerability, I want that patched right away. My OS's package manager will do that. Will your custom installer? If you must, break up your installer into two pieces. First, use a simple Makefile-like setup (avoid autotools though), with generic targets like build, install, clean, dist-clean. Then wrap that in a .deb and .rpm. This way you can distribute system packages that cover 90% of the use cases, and a signed self-contained tarball for the 10% of people who for whatever reason cannot use your pre-build packages. As an empirical point, hundreds of thousands of pieces of mature software get by with using RPM. Why is your project suddenly such a snowflake that it cannot fit that paradigm. Perhaps the people who develop apt/rpm/homebrew/whatever have thought of the problems you might have and already provided the tools for you to solve them so you don't have to write custom installers, upgrade scripts, migration scripts, dependency management scripts, etc. yourself.
- kentonv 11y ago
  > it's something like Ubuntu 14.04 64-bit only. Not in Sandstorm's case. We depend only on the kernel, coreutils, curl, and bash. You can even have them all statically linked, with no shared libraries installed. Sandstorm self-containerizes in a chroot to avoid dependencies. This is actually a big reason we prefer an installer script: it works everywhere. We also include an auto-updater and push often multiple times per week, so likely Sandstorm's deps will be more up-to-date than your system's.
mpdehaan2 11y ago
Packages are a GREAT option if you want to consider upgrades and repeatability than relying on random scripts, particularly because they are largely standardized. Dependency management checks are available and more easily managed than each script doing their own dependency management. This also makes it easier to establish mirrors for production deployment than having to mirror EVERY possible install source (though in many cases, folks will do that). While packages are not 100% standardized, as packages can also do whatever they want in their script sections, but generally distribution scripts are held to a higher degree of review. For instance, Fedora package review guidelines, which feed into CentOS and RHEL guidelines, are pretty thorough. I've generally seen a bit less rigor in Debian/Ubuntu packaging. For handling "merged configurations", look up how conf.d directories work, if you are unaware - it's an easy paradigm. Templating of config files should (or at least can easily be) be handled through a configuration management layer, if need be. I'd argue that no package should be doing things like running sed against existing files, as this is a goal to be solved at a higher level using some form of automation framework. Which of course makes a big case for packages itself - it's much easier to know that a package is not going to be interactive, or tell it to not be interactive, than random web scripts on the internet. As for seperation of differing configurations, VMs are nearly omnipresent at this point, and tend to simply management versus bare metal.

mattzito 11y ago

Gosh, and let's not forget that RPMs can also contain pre- and post- hook scripts, so you could be in a situation where you say, "Install this RPM and satisfy dependencies", and another RPM is downloaded and installed as root, with you never seeing the scripts that are going to be run as part of that process. You can inspect the scripts by hand by examining the package files, but in some ways it's less transparent than downloading a script and running that.

IgorPartola 11y ago
You are downloading software from the internet. How it executes code on your machine is irrelevant, you already trust it. Or do you review every OS package you install for shenanigans on every update? This is an especially bad point to bring up in the context Sandstorm, which runs stuff as root all the time. Do you mean to say that you trust them enough to run a bunch of stuff as root on your server, but not to run a post-install script?
- tokenizerrr 11y ago
  No, but I do trust some software enough to run as non-root. Sadly to install this non-root software I need to install the RPM as root, and then I have no idea what it's going to do during the pre- and post- hook scripts.
  - icebraining 11y ago
    I don't know about RPM, but for deb you can just do "dpkg-deb --control <deb file>" to see those scripts.
  - brazzledazzle 11y ago
    You can export the scripts from an rpm without installing it.
  - IgorPartola 11y ago
    I suppose. Though have you considered the fact that if someone has non-root local access to your machine, they are just one local privilege escalation away form becoming root? And these things come out fairly frequently that you can pretty much count on one being found soon. In other words IMO you should either trust or not trust a piece of code to run on your machine. The distinction between root and non-root user running the code is more for maintainability and sanity and less for actual security.
rkeene2 11y ago
AppFS ( http://appfs.rkeene.org/ http://appfs.rkeene.org/ ) was created for this purpose. Everything is run as an unprivileged user because there is no install phase. You just run the software and the parts it needs are fetched lazily/JIT. Also because it uses PKI end-to-end, you trust the publisher of the application (as in signed RPM/DEB) not the site/mirror you are fetching it from (as in curl|bash).

IgorPartola 11y ago

The thing is that most of these types of projects have very rigid requirements for the OS they support. It's often not just Debian-like or RPM based, it's something like Ubuntu 14.04 64-bit only. If you are already doing that, you already bought into the OS. Support it properly. RPM is not my favorite format. I much prefer Debian-based distros specifically because dpkg and apt are so robust. I have never had a problem with managing configuration files there. You either provide a sane default config, put a config in /usr/share/doc/your-package/, or provide a configuration utility. All that works just fine. And you absolutely should use the system's packages for your dependencies. When you install your own copy of MySQL, nginx, or anything else, you assume responsibility for all upgrades to those subsystems. If next week MySQL releases a patch for remote code execution vulnerability, I want that patched right away. My OS's package manager will do that. Will your custom installer? If you must, break up your installer into two pieces. First, use a simple Makefile-like setup (avoid autotools though), with generic targets like build, install, clean, dist-clean. Then wrap that in a .deb and .rpm. This way you can distribute system packages that cover 90% of the use cases, and a signed self-contained tarball for the 10% of people who for whatever reason cannot use your pre-build packages. As an empirical point, hundreds of thousands of pieces of mature software get by with using RPM. Why is your project suddenly such a snowflake that it cannot fit that paradigm. Perhaps the people who develop apt/rpm/homebrew/whatever have thought of the problems you might have and already provided the tools for you to solve them so you don't have to write custom installers, upgrade scripts, migration scripts, dependency management scripts, etc. yourself.

kentonv 11y ago
> it's something like Ubuntu 14.04 64-bit only. Not in Sandstorm's case. We depend only on the kernel, coreutils, curl, and bash. You can even have them all statically linked, with no shared libraries installed. Sandstorm self-containerizes in a chroot to avoid dependencies. This is actually a big reason we prefer an installer script: it works everywhere. We also include an auto-updater and push often multiple times per week, so likely Sandstorm's deps will be more up-to-date than your system's.

mpdehaan2 11y ago

Packages are a GREAT option if you want to consider upgrades and repeatability than relying on random scripts, particularly because they are largely standardized. Dependency management checks are available and more easily managed than each script doing their own dependency management. This also makes it easier to establish mirrors for production deployment than having to mirror EVERY possible install source (though in many cases, folks will do that). While packages are not 100% standardized, as packages can also do whatever they want in their script sections, but generally distribution scripts are held to a higher degree of review. For instance, Fedora package review guidelines, which feed into CentOS and RHEL guidelines, are pretty thorough. I've generally seen a bit less rigor in Debian/Ubuntu packaging. For handling "merged configurations", look up how conf.d directories work, if you are unaware - it's an easy paradigm. Templating of config files should (or at least can easily be) be handled through a configuration management layer, if need be. I'd argue that no package should be doing things like running sed against existing files, as this is a goal to be solved at a higher level using some form of automation framework. Which of course makes a big case for packages itself - it's much easier to know that a package is not going to be interactive, or tell it to not be interactive, than random web scripts on the internet. As for seperation of differing configurations, VMs are nearly omnipresent at this point, and tend to simply management versus bare metal.