16 ms·

First off, my problem has always been not just with `curl ... | bash` but specifically with `curl http://.. http://... | bash`. Notice the http, not https schem

First off, my problem has always been not just with `curl ... | bash` but specifically with `curl http://.. http://... | bash`. Notice the http, not https scheme there. Second, I feel like this is a huge anti-pattern. Why the heck are you writing a custom installer like this? Each platform has its own system for doing installs and upgrades. Package it for the OS you actually support, such as Debian, Red Hat, etc. It's pretty much a one time investment and it makes your software that much easier to maintain on my system. `curl | bash` makes me think that (a) your software will not play nicely with my OS and I'll have to figure out your weird locations for where you are putting your init scripts, config files, data, etc. and (b) that upgrades will be manual and painful. In other words `curl | bash` is for chumps (tm).

dangerlibrary 11y ago
In this particular case - https has always been used, and everything is in /opt/sandstorm. As far as curl|bash behavior goes, this is about the best I could ask for.
- pg_is_a_butt 11y ago
  yeah... and their solution is basically making a native universal app executable which internally do the exact same bash commands, and the app executable will be downloaded via HTTPS, so it isn't different in any measurable way, yet adds much more support and compiling costs that may break some installs and yield the actual installation script unauditable. you can't argue with these idiots. > /dev/null
mattzito 11y ago
You're not wrong, exactly, but what ends up being brutal is not just packaging, it's packaging and supporting dependency management + post-install configuration. RPMs, for example, are pretty terrible at allowing you to specify custom configuration parameters, and you can't use them at all to install something that you don't have root privileges to access. Custom installers also allow you to have a lot more flexibility in terms of saying, "Okay, let's look to see if there's a mysql instance installed on the system, there's not, so rather than installing an OS one, we'll go ahead and install our preferred mysql version and treat it as an embedded db". RPM-driven upgrades also end up being a lot messier, since you can't do templated or merged configuration files easily. I'm not saying that the curl | bash strategy is good either, but forcing OS packages down people's throats isn't ideal for various situations. EDIT: also, you can't do multiple installs of the same RPM package on a system. So even if you wanted to install 5 instances of app X on a machine, possibly for testing reasons, you can't, and you end up having to use VMs or containers or hacking something together.
- mattzito 11y ago
  Gosh, and let's not forget that RPMs can also contain pre- and post- hook scripts, so you could be in a situation where you say, "Install this RPM and satisfy dependencies", and another RPM is downloaded and installed as root, with you never seeing the scripts that are going to be run as part of that process. You can inspect the scripts by hand by examining the package files, but in some ways it's less transparent than downloading a script and running that.
  - IgorPartola 11y ago
    You are downloading software from the internet. How it executes code on your machine is irrelevant, you already trust it. Or do you review every OS package you install for shenanigans on every update? This is an especially bad point to bring up in the context Sandstorm, which runs stuff as root all the time. Do you mean to say that you trust them enough to run a bunch of stuff as root on your server, but not to run a post-install script?
    - tokenizerrr 11y ago
      No, but I do trust some software enough to run as non-root. Sadly to install this non-root software I need to install the RPM as root, and then I have no idea what it's going to do during the pre- and post- hook scripts.
      - icebraining 11y ago
        I don't know about RPM, but for deb you can just do "dpkg-deb --control <deb file>" to see those scripts.
      - brazzledazzle 11y ago
        You can export the scripts from an rpm without installing it.
      - IgorPartola 11y ago
        I suppose. Though have you considered the fact that if someone has non-root local access to your machine, they are just one local privilege escalation away form becoming root? And these things come out fairly frequently that you can pretty much count on one being found soon. In other words IMO you should either trust or not trust a piece of code to run on your machine. The distinction between root and non-root user running the code is more for maintainability and sanity and less for actual security.
_yy 11y ago
I fully agree with you. Though Sandstorm in particular explained why they do it this way: Q: Installing software not managed by the distro’s package manager can be inconvenient for system administrators, especially those who are deeply familiar with the inner workings of their distro. A: On these concerns: We hear you. For now, we believe that having our own install scripts allows us to iterate faster, compared to maintaining (and testing) half a dozen package formats for different distros. However, once Sandstorm reaches a stable release, we fully intend to offer more traditional packaging choices as well. We still have lots of work to do! So, there will be packages once it's stable. Right now, they'd rather spend their time adding new features.
- IgorPartola 11y ago
  They already spent more time creating a custom installer than they would have to package it properly. That's my point, just do it right and you will see a big ROI in terms of more users of your software.
- rythie 11y ago
  Translation: at some point after putting one hack on top of another at high speed, we will clean up the mess and do it properly.
voltagex_ 11y ago
At the end of the article: > Installing software not managed by the distro’s package manager can be inconvenient for system administrators, especially those who are deeply familiar with the inner workings of their distro. > On these concerns: We hear you. > For now, we believe that having our own install scripts allows us to iterate faster, compared to maintaining (and testing) half a dozen package formats for different distros. However, once Sandstorm reaches a stable release, we fully intend to offer more traditional packaging choices as well. We still have lots of work to do! This sounds like a cop-out. Packaging an RPM and a DEB will take care of most distros, and the distro maintainers should do the rest. Your build system should be creating the requisite package files. If not, there's always OpenSuse Build Service.
saidajigumi 11y ago
> Each platform has its own system for doing installs and upgrades. A system which is lacking either by platform convention and/or by technological capability to handle some now common use cases: thus the rise of second-level package managers. Homebrew, linuxbrew, rbenv/rvm, ndenv/nvm, etc. all came to the forefront because the platform's package management was not doing the job. You might pick out linuxbrew and say "ah, just use rpms/debs/etc!", but there are a lot of cases where you don't have the option to choose the distro flavor or version, which can severely limit your ability to pick up packages/versions that you need. Likewise, in most distros it's impossible to cherry-pick a single package back onto to an older release; an explicit backport must be made available. Too bad if it's not available!
- stephenr 11y ago
  > Homebrew Exists because OS X doesn't have a vendor-supplied "download and install this, including any dependencies" type package manager, and the existing attempts to fix it, were terrible. Frankly so is Homebrew. > linuxbrew using Beer terms is apparently cool for software packaging so lets port the idea of Homebrew back to Linux. > rbenv/rvm, ndenv/nvm rvm (and can i assume nvm, which i guess is rvm for nodejs?) is about installing multiple versions of Ruby and using them. There is nothing inherent about this concept that can't be achieved with a regular package - you're just placing compiled files in certain directories. Debian even had multiple versions of 1.x (1.8 and 1.9 from memory) that could be installed side-by-side. Even now the packages install binaries etc as e.g. `/usr/bin/ruby2.1`, so I don't buy the "traditional package managers were not doing the job". They just aren't written in hipster languages and they don't force you to compile every fucking thing you want to use on every fucking machine every fucking time you install it. > Likewise, in most distros it's impossible to cherry-pick a single package back onto to an older release; an explicit backport must be made available. Too bad if it's not available! Debian provides a pretty easy to follow guide about making your own unofficial backports, or guidance on creating official backports, and even has a basic 'auto' setup script and a "backport from a PPA" guide.
  - Chlorus 11y ago
    >They just aren't written in hipster languages and they don't force you to compile every fucking thing you want to use on every fucking machine every fucking time you install it. What? Homebrew has done binaries by default for ages now.
ikeboy 11y ago
`curl | bash` makes me think that (a) your software will not play nicely with my OS and I'll have to figure out your weird locations for where you are putting your init scripts, config files, data, etc. and (b) that upgrades will be manual and painful. Fwiw, the two programs I've used with such a script (Calibre and Youtube-dl) both update easily, and don't have problems with support. That said, it would be nice if there was a way to update them automatically. There seems to be a way to hook commands into apt-get, maybe you can add a command to update your other programs at the same time.
- finnn 11y ago
  youtube-dl is in PyPI, so you should be installing and updating with pip install youtube-dl and pip install --upgrade youtube-dl
- bumbledraven 11y ago
  youtube-dl -U # updates youtube-dl to latest version You could add this command to cron.
Karunamon 11y ago
Why the heck are you writing a custom installer like this? One possible example: User specific installs. Interaction with the system package manager requires root rights, and isn't something that is a good fit for something like, say, RVM or rbenv. If I'm a nonprivileged user on someone else's general purpose system, I shouldn't have to bug the administrator to install something useful to me.
- Filligree 11y ago
  Nix solves this problem neatly. Not saying that's answering your question, since Nix is woefully underused, but it's something to look at.
  - codemac 11y ago
    Yeah I've been using arch for my "root" and guix for my users and it's a wonderland of excellence. If you're interested in nix / guix, installing them and using them without the whole OS is a great way to get started.
- peterwwillis 11y ago
  All package managers allow user-specific installs.
  - Karunamon 11y ago
    As a special case, sure. http://askubuntu.com/questions/339/how-can-i-install-a-package-without-root-access http://askubuntu.com/questions/339/how-can-i-install-a-packa... http://askubuntu.com/questions/28619/how-do-i-install-an-application-by-deb-file-for-a-single-user-only http://askubuntu.com/questions/28619/how-do-i-install-an-app... But if you're going to extract deb files, why not just extract tarballs? Or better yet, just curl|bash? :)
    - peterwwillis 11y ago
      There's lots of reasons to use packages. Verification, versioning, bug tracking, offline use, system-install, user-install, dependency tracking, rollback, sanity checking, custom install steps, and being tested and certified for a particular repository and software branch, among others. But, you know, if you just want to install something and not know what it is, what it does, how to fix it or remove it, much less whether it's even the software you expected, curl|bash is okay.
msbarnett 11y ago
Bypassing the developers say "version 1.15 is out now" instead of "version 1.15 is out: 1) Now + however many years until the next Debian stable 2) Now + however many months until the next Ubuntu 3) Now for Arch users 4) Now + whatever for Fedora 5) Now + a different whatever for SUSE etc etc" Package-the-world-and-freeze distribution schemes are a black hole of suck for development tool projects that have their own ecosystem and want to iterate at a consistent pace. "Of course," you could say, "the devs should just run their own secondary repos instead of submitting to the official ones". So now instead of one bash script you just have to trust not to rootkit you, the devs distribute 10 different incompatible package formats you just have to trust not to rootkit you, at only 10x the maintenance, build script, and sysadmin burden as you run all these bespoke repos.
- derefr 11y ago
  Note that "package-the-world-and-freeze" distributions (Debian, RHEL, whoever else) have an entire infrastructure in place to creating their own line of packages from an "upstream's" basic-tarball releases. When Debian determines to, e.g., support Linux 3.6 for the next five years, it's not suddenly Linus's responsibility to backport kernel patches to Linux 3.6. Likewise, it's not your responsibility to maintain a stable branch of your software so that Debian can do less work packaging it. The other kind of model, of Ubuntu PPAs and such, where they're expected to track the developer's releases, frequently incentivize the developer to get those packages created as part of their own build infrastructure. But there's still no expectation that they will. The culture of the "distribution" is set up to allow developers to just develop, and distributors to be the ones to think about distribution.
  - msbarnett 11y ago
    > Note that "package-the-world-and-freeze" distributions (Debian, RHEL, whoever else) have an entire infrastructure in place to creating their own line of packages from an "upstream's" basic-tarball releases. > When Debian determines to, e.g., support Linux 3.6 for the next five years, it's not suddenly Linus's responsibility to backport kernel patches to Linux 3.6. Likewise, it's not your responsibility to maintain a stable branch of your software so that Debian can do less work packaging it. That's great and all, but if I'm a Ruby dev my life isn't any better just because the burden of packaging a 4-year-out-of-date version of the runtime isn't on me. The Rails community wants to move to the latest Rails version, which wants to user modern Ruby syntax, which depends on a modern Ruby version, and the community doesn't just want to bifurcate and fork every single gem for half a decade because Debian's dragging it's feet about the next release. And I can't just order my users to use Debian Sid in a production environment. The fundamental problem here is that Distro-packaging is optimized to distribute software for an ecosystem comprised of that Distro's users. The problem a lot of dev tool projects run into is that they serve a cross-cutting community of many different OS's users, all of whom want to stay in sync with the rest of the cross-cutting community, and who couldn't give two shits about what some other person's OS's package repo's timelines are like. That's the niche that NPM, RVM, et al are serving and it's a niche that distro packages are completely unsuited to handle. And RVM et al tend to "package" as scripts because we're right back in this mess if Debian RVM users are 4 years behind OS X RVM users.

dangerlibrary 11y ago

In this particular case - https has always been used, and everything is in /opt/sandstorm. As far as curl|bash behavior goes, this is about the best I could ask for.

pg_is_a_butt 11y ago
yeah... and their solution is basically making a native universal app executable which internally do the exact same bash commands, and the app executable will be downloaded via HTTPS, so it isn't different in any measurable way, yet adds much more support and compiling costs that may break some installs and yield the actual installation script unauditable. you can't argue with these idiots. > /dev/null

mattzito 11y ago

You're not wrong, exactly, but what ends up being brutal is not just packaging, it's packaging and supporting dependency management + post-install configuration. RPMs, for example, are pretty terrible at allowing you to specify custom configuration parameters, and you can't use them at all to install something that you don't have root privileges to access. Custom installers also allow you to have a lot more flexibility in terms of saying, "Okay, let's look to see if there's a mysql instance installed on the system, there's not, so rather than installing an OS one, we'll go ahead and install our preferred mysql version and treat it as an embedded db". RPM-driven upgrades also end up being a lot messier, since you can't do templated or merged configuration files easily. I'm not saying that the curl | bash strategy is good either, but forcing OS packages down people's throats isn't ideal for various situations. EDIT: also, you can't do multiple installs of the same RPM package on a system. So even if you wanted to install 5 instances of app X on a machine, possibly for testing reasons, you can't, and you end up having to use VMs or containers or hacking something together.

mattzito 11y ago
Gosh, and let's not forget that RPMs can also contain pre- and post- hook scripts, so you could be in a situation where you say, "Install this RPM and satisfy dependencies", and another RPM is downloaded and installed as root, with you never seeing the scripts that are going to be run as part of that process. You can inspect the scripts by hand by examining the package files, but in some ways it's less transparent than downloading a script and running that.
- IgorPartola 11y ago
  You are downloading software from the internet. How it executes code on your machine is irrelevant, you already trust it. Or do you review every OS package you install for shenanigans on every update? This is an especially bad point to bring up in the context Sandstorm, which runs stuff as root all the time. Do you mean to say that you trust them enough to run a bunch of stuff as root on your server, but not to run a post-install script?
  - tokenizerrr 11y ago
    No, but I do trust some software enough to run as non-root. Sadly to install this non-root software I need to install the RPM as root, and then I have no idea what it's going to do during the pre- and post- hook scripts.
    - icebraining 11y ago
      I don't know about RPM, but for deb you can just do "dpkg-deb --control <deb file>" to see those scripts.
    - brazzledazzle 11y ago
      You can export the scripts from an rpm without installing it.
    - IgorPartola 11y ago
      I suppose. Though have you considered the fact that if someone has non-root local access to your machine, they are just one local privilege escalation away form becoming root? And these things come out fairly frequently that you can pretty much count on one being found soon. In other words IMO you should either trust or not trust a piece of code to run on your machine. The distinction between root and non-root user running the code is more for maintainability and sanity and less for actual security.

_yy 11y ago

I fully agree with you. Though Sandstorm in particular explained why they do it this way: Q: Installing software not managed by the distro’s package manager can be inconvenient for system administrators, especially those who are deeply familiar with the inner workings of their distro. A: On these concerns: We hear you. For now, we believe that having our own install scripts allows us to iterate faster, compared to maintaining (and testing) half a dozen package formats for different distros. However, once Sandstorm reaches a stable release, we fully intend to offer more traditional packaging choices as well. We still have lots of work to do! So, there will be packages once it's stable. Right now, they'd rather spend their time adding new features.

IgorPartola 11y ago
They already spent more time creating a custom installer than they would have to package it properly. That's my point, just do it right and you will see a big ROI in terms of more users of your software.
rythie 11y ago
Translation: at some point after putting one hack on top of another at high speed, we will clean up the mess and do it properly.

voltagex_ 11y ago

At the end of the article: > Installing software not managed by the distro’s package manager can be inconvenient for system administrators, especially those who are deeply familiar with the inner workings of their distro. > On these concerns: We hear you. > For now, we believe that having our own install scripts allows us to iterate faster, compared to maintaining (and testing) half a dozen package formats for different distros. However, once Sandstorm reaches a stable release, we fully intend to offer more traditional packaging choices as well. We still have lots of work to do! This sounds like a cop-out. Packaging an RPM and a DEB will take care of most distros, and the distro maintainers should do the rest. Your build system should be creating the requisite package files. If not, there's always OpenSuse Build Service.

saidajigumi 11y ago

> Each platform has its own system for doing installs and upgrades. A system which is lacking either by platform convention and/or by technological capability to handle some now common use cases: thus the rise of second-level package managers. Homebrew, linuxbrew, rbenv/rvm, ndenv/nvm, etc. all came to the forefront because the platform's package management was not doing the job. You might pick out linuxbrew and say "ah, just use rpms/debs/etc!", but there are a lot of cases where you don't have the option to choose the distro flavor or version, which can severely limit your ability to pick up packages/versions that you need. Likewise, in most distros it's impossible to cherry-pick a single package back onto to an older release; an explicit backport must be made available. Too bad if it's not available!

stephenr 11y ago
> Homebrew Exists because OS X doesn't have a vendor-supplied "download and install this, including any dependencies" type package manager, and the existing attempts to fix it, were terrible. Frankly so is Homebrew. > linuxbrew using Beer terms is apparently cool for software packaging so lets port the idea of Homebrew back to Linux. > rbenv/rvm, ndenv/nvm rvm (and can i assume nvm, which i guess is rvm for nodejs?) is about installing multiple versions of Ruby and using them. There is nothing inherent about this concept that can't be achieved with a regular package - you're just placing compiled files in certain directories. Debian even had multiple versions of 1.x (1.8 and 1.9 from memory) that could be installed side-by-side. Even now the packages install binaries etc as e.g. `/usr/bin/ruby2.1`, so I don't buy the "traditional package managers were not doing the job". They just aren't written in hipster languages and they don't force you to compile every fucking thing you want to use on every fucking machine every fucking time you install it. > Likewise, in most distros it's impossible to cherry-pick a single package back onto to an older release; an explicit backport must be made available. Too bad if it's not available! Debian provides a pretty easy to follow guide about making your own unofficial backports, or guidance on creating official backports, and even has a basic 'auto' setup script and a "backport from a PPA" guide.
- Chlorus 11y ago
  >They just aren't written in hipster languages and they don't force you to compile every fucking thing you want to use on every fucking machine every fucking time you install it. What? Homebrew has done binaries by default for ages now.

ikeboy 11y ago

`curl | bash` makes me think that (a) your software will not play nicely with my OS and I'll have to figure out your weird locations for where you are putting your init scripts, config files, data, etc. and (b) that upgrades will be manual and painful. Fwiw, the two programs I've used with such a script (Calibre and Youtube-dl) both update easily, and don't have problems with support. That said, it would be nice if there was a way to update them automatically. There seems to be a way to hook commands into apt-get, maybe you can add a command to update your other programs at the same time.

finnn 11y ago
youtube-dl is in PyPI, so you should be installing and updating with pip install youtube-dl and pip install --upgrade youtube-dl
bumbledraven 11y ago
youtube-dl -U # updates youtube-dl to latest version You could add this command to cron.

Karunamon 11y ago

Why the heck are you writing a custom installer like this? One possible example: User specific installs. Interaction with the system package manager requires root rights, and isn't something that is a good fit for something like, say, RVM or rbenv. If I'm a nonprivileged user on someone else's general purpose system, I shouldn't have to bug the administrator to install something useful to me.

Filligree 11y ago
Nix solves this problem neatly. Not saying that's answering your question, since Nix is woefully underused, but it's something to look at.
- codemac 11y ago
  Yeah I've been using arch for my "root" and guix for my users and it's a wonderland of excellence. If you're interested in nix / guix, installing them and using them without the whole OS is a great way to get started.
peterwwillis 11y ago
All package managers allow user-specific installs.
- Karunamon 11y ago
  As a special case, sure. http://askubuntu.com/questions/339/how-can-i-install-a-package-without-root-access http://askubuntu.com/questions/339/how-can-i-install-a-packa... http://askubuntu.com/questions/28619/how-do-i-install-an-application-by-deb-file-for-a-single-user-only http://askubuntu.com/questions/28619/how-do-i-install-an-app... But if you're going to extract deb files, why not just extract tarballs? Or better yet, just curl|bash? :)
  - peterwwillis 11y ago
    There's lots of reasons to use packages. Verification, versioning, bug tracking, offline use, system-install, user-install, dependency tracking, rollback, sanity checking, custom install steps, and being tested and certified for a particular repository and software branch, among others. But, you know, if you just want to install something and not know what it is, what it does, how to fix it or remove it, much less whether it's even the software you expected, curl|bash is okay.

msbarnett 11y ago

Bypassing the developers say "version 1.15 is out now" instead of "version 1.15 is out: 1) Now + however many years until the next Debian stable 2) Now + however many months until the next Ubuntu 3) Now for Arch users 4) Now + whatever for Fedora 5) Now + a different whatever for SUSE etc etc" Package-the-world-and-freeze distribution schemes are a black hole of suck for development tool projects that have their own ecosystem and want to iterate at a consistent pace. "Of course," you could say, "the devs should just run their own secondary repos instead of submitting to the official ones". So now instead of one bash script you just have to trust not to rootkit you, the devs distribute 10 different incompatible package formats you just have to trust not to rootkit you, at only 10x the maintenance, build script, and sysadmin burden as you run all these bespoke repos.

derefr 11y ago
Note that "package-the-world-and-freeze" distributions (Debian, RHEL, whoever else) have an entire infrastructure in place to creating their own line of packages from an "upstream's" basic-tarball releases. When Debian determines to, e.g., support Linux 3.6 for the next five years, it's not suddenly Linus's responsibility to backport kernel patches to Linux 3.6. Likewise, it's not your responsibility to maintain a stable branch of your software so that Debian can do less work packaging it. The other kind of model, of Ubuntu PPAs and such, where they're expected to track the developer's releases, frequently incentivize the developer to get those packages created as part of their own build infrastructure. But there's still no expectation that they will. The culture of the "distribution" is set up to allow developers to just develop, and distributors to be the ones to think about distribution.
- msbarnett 11y ago
  > Note that "package-the-world-and-freeze" distributions (Debian, RHEL, whoever else) have an entire infrastructure in place to creating their own line of packages from an "upstream's" basic-tarball releases. > When Debian determines to, e.g., support Linux 3.6 for the next five years, it's not suddenly Linus's responsibility to backport kernel patches to Linux 3.6. Likewise, it's not your responsibility to maintain a stable branch of your software so that Debian can do less work packaging it. That's great and all, but if I'm a Ruby dev my life isn't any better just because the burden of packaging a 4-year-out-of-date version of the runtime isn't on me. The Rails community wants to move to the latest Rails version, which wants to user modern Ruby syntax, which depends on a modern Ruby version, and the community doesn't just want to bifurcate and fork every single gem for half a decade because Debian's dragging it's feet about the next release. And I can't just order my users to use Debian Sid in a production environment. The fundamental problem here is that Distro-packaging is optimized to distribute software for an ecosystem comprised of that Distro's users. The problem a lot of dev tool projects run into is that they serve a cross-cutting community of many different OS's users, all of whom want to stay in sync with the rest of the cross-cutting community, and who couldn't give two shits about what some other person's OS's package repo's timelines are like. That's the niche that NPM, RVM, et al are serving and it's a niche that distro packages are completely unsuited to handle. And RVM et al tend to "package" as scripts because we're right back in this mess if Debian RVM users are 4 years behind OS X RVM users.