9 ms·

> Installing software by piping from curl to bash is obviously a bad idea and a knowledgable user will most likely check the content first It's funny how so ma

by ithkuil 11y ago

> Installing software by piping from curl to bash is obviously a bad idea and a knowledgable user will most likely check the content first It's funny how so many obvious things are all but obvious when you think a little bit more about it. Interesting read on the subject: https://sandstorm.io/news/2015-09-24-is-curl-bash-insecure-pgp-verified-install https://sandstorm.io/news/2015-09-24-is-curl-bash-insecure-p... (I don't want to enter the `curl | bash` good or bad rabbit hole; just that the topic cannot be just dismissed as "obvious")

kenny_r 11y ago
For me personally, the security aspect isn't even the reason I dislike curl | bash installers. There's no standard behaviour for curl|sh, and it re-introduces many problems that were solved by package managers decades ago. Each time I'm about to run `curl https://install.newhotness.io|sudo https://install.newhotness.io|sudo sh` I'm left with the following questions: * Will it work? I'm on Arch Linux. This installer was probably tested on OSX and Ubuntu and deemed "portable". I've seen curl|sudo sh installers trying to apt-get installing dependencies. * Where does it place files? I've got a ~/.meteor folder that's 784MB. Why would you install software libraries and binaries to $HOME? I now have to tell my backup tool to ignore it when backing up my home partition, great. The FHS was established for a reason, people. * Correlary to the previous question: How do I uninstall it? Maybe all files were installed to a hidden folder in ~/, maybe there's some stuff in /opt, or /usr/local, or who knows where else. If the installer doesn't implement uninstall functionality itself, I now have to go hunt for the stuff it placed in my filesystem. I understand the need to be able to distribute your software without having to implement native packaging for OSX/Debian/Ubuntu/Red Hat/Fedora/Arch/... Docker solved this problem. If you're feeling lazy when it comes to distributing your app, just ship a Docker container and be done with it.
- TeMPOraL 11y ago
  > Correlary to the previous question: How do I uninstall it? Maybe all files were installed to a hidden folder in ~/, maybe there's some stuff in /opt, or /usr/local, or who knows where else. How does one uninstall stuff on Linux anyway? My Linux boxes are always like thrash cans - I add and add new stuff until it's time to upgrade either the system or the machine, so it gets wiped and replaced. Never figured out how to delete stuff in a way that will not leave a ton of things behind...
  - dsp1234 11y ago
    The following generally works for me. It removes the installed files, including configuration files. Additionally it will remove any dependencies that were only used by packagename sudo apt-get purge --auto-remove packagename
  - HappyTypist 11y ago
    Your package manager should have a remove operation.
  - serf 11y ago
    >How does one uninstall stuff on Linux anyway? with a package manager suited for your system, or with the knowledge of what files the software in question touches.
    - RijilV 11y ago
      It's a best effort though, you're relying on the package maintainer to have tracked all the installed files and clean up anything they might have modified, which hopefully you or something else hasn't modified after the package was installed. The packages authored by distributions tend to be pretty good, but I've seen RPMs with nothing more than tar zxf in them and no tracking of the files installed. The major package managers expect the package to manage the artifacts installed, this works if everyone plays nice and does their job...
      - alphapapa 11y ago
        You're right that this requires well-maintained packages, and this is one of the strengths of Debian: its policies that require well-maintained packages. If you install a package that's part of Debian, you can count on it being well-maintained, as opposed to one put out by a random web site or developer, which may install but may not integrate well or clean up after itself.
      - dsmithatx 11y ago
        One point I'd like to make is that this is the fault of the developer(s) and not Linux. I've seen this problem in OSX and Windows as well.
- crdoconnor 11y ago
  >For me personally, the security aspect isn't even the reason I dislike curl | bash installers. There's no standard behaviour for curl|sh, and it re-introduces many problems that were solved by package managers decades ago. Every time I've seen it used, curl | bash is simply a way to deal with the fact that the users all have different package managers (often with outdated software) and quite complicated workflows to get stuff set up with them because there were problems that the package manager didn't solve. >Docker solved this problem. Ironic. Docker is one of those applications. And they used to ask for it to be installed with a curl pipe. They seem to have replaced it with what is effectively a script executed with human hands. Not really an improvement.
  - kenny_r 11y ago
    I was genuinly confused with regards to your last point. There's a section on their website[0] there are installation instructions for various distributions using repositories and package managers. In other words, the right(tm) way of doing software distribution on Linux. But then on a different part of their website I found the curl | sh instructions you allude to[1]. Bizarre that they still support that installation method. To make matters worse, the big orange "Get Started" call to action button actually leads to the curl|sh installation instructions. In light of that I certainly can't blame you for not knowing about the proper distribution packages. [0] https://docs.docker.com/engine/installation/ https://docs.docker.com/engine/installation/ [1] https://docs.docker.com/linux/step_one/ https://docs.docker.com/linux/step_one/
    - crdoconnor 11y ago
      That curled script actually installs the 'proper' distribution packages. It has a lot of clever stuff to figure out which environment it's running on and then does whatever commands are necessary - including, e.g., adding keys so that the external repos where docker is hosted are trusted. Point being that even if you install "the right way" on linux using package managers it's still a multi-step complicated process that benefits from being automated in a bash script.
      - 11y ago
- lgessler 11y ago
  FWIW, this is Meteor's justification in /usr/local/bin/meteor: # This is the script that we install somewhere in your $PATH (as "meteor") # when you run # $ curl https://install.meteor.com/ | sh # It's the only file that we install globally on your system; each user of # Meteor gets their own personal package and tools repository, called the # warehouse (or, for 0.9.0 and newer, the "tropohouse"), in ~/.meteor/. This # means that a user can share packages among multiple apps and automatically # update to new releases without having to have permissions to write them to # anywhere global.
  - TallGuyShort 11y ago
    I used to do Linux packaging full-time and constantly hit limitations in what Linux packaging formats support or allow. There's still a lot of progress that could be made in that area, but unfortunately any attempt to improve will likely become that 15th competing standard XKCD refers to...
  - pbhjpbhj 11y ago
    They could be in ~/.local/share/meteor following "XDG Base Directory Specification" (https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html https://specifications.freedesktop.org/basedir-spec/basedir-... - seems well used on my Kubuntu but it's a case of "pick a standard" I guess).
- alphapapa 11y ago
  For dealing with this problem, as well as anything that's not already packaged, the checkinstall utility is very helpful. It intercepts filesystem calls and builds deb/rpm/tgz packages that you install with your distro's package manager. It's been packaged in Debian for a long time. Highly recommended.
- nksjndfdf 11y ago
  > Why would you install software libraries and binaries to $HOME? Why not? It makes more sense than global installations sometimes.
Sean1708 11y ago
What amuses me is that the people who are vehemently against using a bash script are often the same people who will run $ ./configure $ make without a second thought.
- merb 11y ago
  Actually 99% of the Time this will install it to /usr/local
  - fjarlq 11y ago
    I disagree. Many (probably most) makefiles conform to the GNU Coding Standards, which recommend declaring "all" to be the default make target. `make all` does not install; that's what `make install` is for. https://www.gnu.org/prep/standards/html_node/Standard-Targets.html https://www.gnu.org/prep/standards/html_node/Standard-Target...
    - coldtea 11y ago
      The idea of the original parent comment on Make is not about well conforming makefiles. The idea is that if one doesn't trust a random curl-ed install file, they shouldn't trust a downloaded makefile either -- and for the same reasons, yet tons of people complaint about the first, but have been using the latter for decades without complain...
  - coldtea 11y ago
    The idea is that the Makefile (and maybe configure?) could just as well as a "curled" install script contain ANY command, even "sudo rm -rf /". It's not about where it will install it if it works fine.
- koolba 11y ago
  Don't forget the most important part: sudo make install
- pif 11y ago
  Yes, but not as root!
nailer 11y ago
Agreed 100%. Things installed on your machine have privileges regardless of how you install them. Install software from verified sources. Whether you use curl or not is irrelevant.
- raesene6 11y ago
  I'd be interested in knowing, how do you assess whether something is a "trusted source" or not?
  - coldtea 11y ago
    The same way we trust people. By reputation and familiarity. If something is from e.g. Red Hat or a large project repository like Apache, it's more trusted than a random website with some unknown tarball... If many people in websites and forums say "don't trust this program", well, don't trust it. Etc.
    - raesene6 11y ago
      Ahh but that's one of the hardest pieces of the trust puzzle. On the internet, how do you know who's who. you download a package from npm. Who wrote it, what's their background, where do they work? What are their security practices like, what are the security practices of the hosting company? so we say Redhat/Apache are more trusted sure, that's one of the reasons that I dislike curl|bash from random sites, there's no way of assessing trust in that software or its origin...
      - coldtea 11y ago
        >On the internet, how do you know who's who. you download a package from npm. Who wrote it, what's their background, where do they work? What are their security practices like, what are the security practices of the hosting company? Check the npm package's ratings, and the github repository watchers and stars. Try to find and read a few posts (and their comments) on the package. The more respected the source (e.g. HN comments vs comments on Digg), the better.
jfindley 11y ago
I don't think that's nearly as clever a counter-argument against curl|bash as they appear to. A few points of the shortcomings of curl|bash vs RPM and others: * curl|bash is not transactional. If you have an install/upgrade process in a different tab, and you close your terminal accidentally, how do you know if the install/upgrade completed? * There's no obvious uninstall method, and you need to read the docs to see what, if anything, you should remove outside the main directory. * There's no way of verifying if files have been tampered with post installation. This has negative security and operational consequences. * It's much more vulnerable to MiTM attacks - a proxy with a successfully faked certificate can trivially modify a bash script on the wire and add a malicious command to it. This is hard for a user to detect. Packages are (on most distros) installed via package managers that verify GPG signing keys, making in-flight modification very very greatly more difficult. * Packages are much more auditable - I can download a package and inspect it, and know that it will run the same actions on every machine it's deployed to. curl|bash can trivially serve different instructions to different requests, based on whatever secret criteria the server operator decides. If you're going to try and argue that instead of doing what the generally accepted $good_thing is, and instead use $quick_and_easy solution that's widely perceived to be insecure, when you argue in defence of your position, you need to make very sure that you're not just trying to justify laziness. I don't think sandstorm have really thought about this hard enough.
- ithkuil 11y ago
  I think the many discussions about this conflates two things: 1. "curl|bash" vs "curl >install.sh; sh install.sh" 2. install scripts vs package managers The OP talks about how to detect a streaming execution of a script and exploit it. This proof of concept exploit aims at discouraging the the use of curl|bash as a means to installing code, and instead promotes that at least you should save the script (and review it) (i.e. the "curl >install.sh; sh install.sh"). All of the points you mentioned are valid and important pain points that affect any arbitrary executable install script (not verified cryptographically using a side channel). Deciding whether to use an install script or a packaging system is really a tradeoff. The "obvious" problem the OP was talking about was that smell you feel when you see piping code to bash. I think OP made an excellent job showing how you can make it harder for people to spot and audit malicious code and how this is pretty serious downside for piping code into bash; It's more like: "I have a gut feeling this is horrible, so let's think about a way to prove that gut feeling", and the proof is insightful and non obvious.
donquichotte 11y ago
Recently I stumbled across hashpipe: "hashpipe helps you venture into the unknown. It reads from stdin, checks the hash of the content, and outputs it IF AND ONLY IF it matches the provided hash checksum." [https://github.com/jbenet/hashpipe https://github.com/jbenet/hashpipe]
- coredog64 11y ago
  I was going to try it, but it didn't have a curl|bash install instruction.
  - kenny_r 11y ago
    I know you're being facetious but `go get` is just as convenient and is a controversial installation method for different reasons.
NDizzle 11y ago
Every time I see that I just wget the script and inspect it. If it does anything too clever, I don't install it that way. If I can't reason about the shell script something is wrong. It's like being emailed a .bat file in 1999. Would you blindly run it on your Windows box?! I hope not!
- tamana 11y ago
  The script downloads a binary onto your system. How would you inspect that?
goldbrick 11y ago
That article is good, but keybase.io is a non-starter until they open registration. I have been on the pending queue for over a year now.
- kentonv 11y ago
  You don't need a Keybase account to follow the Sandstorm PGP-verified installation instructions.

by ithkuil 11y ago

kenny_r 11y ago

For me personally, the security aspect isn't even the reason I dislike curl | bash installers. There's no standard behaviour for curl|sh, and it re-introduces many problems that were solved by package managers decades ago. Each time I'm about to run `curl https://install.newhotness.io|sudo https://install.newhotness.io|sudo sh` I'm left with the following questions: * Will it work? I'm on Arch Linux. This installer was probably tested on OSX and Ubuntu and deemed "portable". I've seen curl|sudo sh installers trying to apt-get installing dependencies. * Where does it place files? I've got a ~/.meteor folder that's 784MB. Why would you install software libraries and binaries to $HOME? I now have to tell my backup tool to ignore it when backing up my home partition, great. The FHS was established for a reason, people. * Correlary to the previous question: How do I uninstall it? Maybe all files were installed to a hidden folder in ~/, maybe there's some stuff in /opt, or /usr/local, or who knows where else. If the installer doesn't implement uninstall functionality itself, I now have to go hunt for the stuff it placed in my filesystem. I understand the need to be able to distribute your software without having to implement native packaging for OSX/Debian/Ubuntu/Red Hat/Fedora/Arch/... Docker solved this problem. If you're feeling lazy when it comes to distributing your app, just ship a Docker container and be done with it.

TeMPOraL 11y ago
> Correlary to the previous question: How do I uninstall it? Maybe all files were installed to a hidden folder in ~/, maybe there's some stuff in /opt, or /usr/local, or who knows where else. How does one uninstall stuff on Linux anyway? My Linux boxes are always like thrash cans - I add and add new stuff until it's time to upgrade either the system or the machine, so it gets wiped and replaced. Never figured out how to delete stuff in a way that will not leave a ton of things behind...
- dsp1234 11y ago
  The following generally works for me. It removes the installed files, including configuration files. Additionally it will remove any dependencies that were only used by packagename sudo apt-get purge --auto-remove packagename
- HappyTypist 11y ago
  Your package manager should have a remove operation.
- serf 11y ago
  >How does one uninstall stuff on Linux anyway? with a package manager suited for your system, or with the knowledge of what files the software in question touches.
  - RijilV 11y ago
    It's a best effort though, you're relying on the package maintainer to have tracked all the installed files and clean up anything they might have modified, which hopefully you or something else hasn't modified after the package was installed. The packages authored by distributions tend to be pretty good, but I've seen RPMs with nothing more than tar zxf in them and no tracking of the files installed. The major package managers expect the package to manage the artifacts installed, this works if everyone plays nice and does their job...
    - alphapapa 11y ago
      You're right that this requires well-maintained packages, and this is one of the strengths of Debian: its policies that require well-maintained packages. If you install a package that's part of Debian, you can count on it being well-maintained, as opposed to one put out by a random web site or developer, which may install but may not integrate well or clean up after itself.
    - dsmithatx 11y ago
      One point I'd like to make is that this is the fault of the developer(s) and not Linux. I've seen this problem in OSX and Windows as well.
crdoconnor 11y ago
>For me personally, the security aspect isn't even the reason I dislike curl | bash installers. There's no standard behaviour for curl|sh, and it re-introduces many problems that were solved by package managers decades ago. Every time I've seen it used, curl | bash is simply a way to deal with the fact that the users all have different package managers (often with outdated software) and quite complicated workflows to get stuff set up with them because there were problems that the package manager didn't solve. >Docker solved this problem. Ironic. Docker is one of those applications. And they used to ask for it to be installed with a curl pipe. They seem to have replaced it with what is effectively a script executed with human hands. Not really an improvement.
- kenny_r 11y ago
  I was genuinly confused with regards to your last point. There's a section on their website[0] there are installation instructions for various distributions using repositories and package managers. In other words, the right(tm) way of doing software distribution on Linux. But then on a different part of their website I found the curl | sh instructions you allude to[1]. Bizarre that they still support that installation method. To make matters worse, the big orange "Get Started" call to action button actually leads to the curl|sh installation instructions. In light of that I certainly can't blame you for not knowing about the proper distribution packages. [0] https://docs.docker.com/engine/installation/ https://docs.docker.com/engine/installation/ [1] https://docs.docker.com/linux/step_one/ https://docs.docker.com/linux/step_one/
  - crdoconnor 11y ago
    That curled script actually installs the 'proper' distribution packages. It has a lot of clever stuff to figure out which environment it's running on and then does whatever commands are necessary - including, e.g., adding keys so that the external repos where docker is hosted are trusted. Point being that even if you install "the right way" on linux using package managers it's still a multi-step complicated process that benefits from being automated in a bash script.
    - 11y ago
lgessler 11y ago
FWIW, this is Meteor's justification in /usr/local/bin/meteor: # This is the script that we install somewhere in your $PATH (as "meteor") # when you run # $ curl https://install.meteor.com/ | sh # It's the only file that we install globally on your system; each user of # Meteor gets their own personal package and tools repository, called the # warehouse (or, for 0.9.0 and newer, the "tropohouse"), in ~/.meteor/. This # means that a user can share packages among multiple apps and automatically # update to new releases without having to have permissions to write them to # anywhere global.
- TallGuyShort 11y ago
  I used to do Linux packaging full-time and constantly hit limitations in what Linux packaging formats support or allow. There's still a lot of progress that could be made in that area, but unfortunately any attempt to improve will likely become that 15th competing standard XKCD refers to...
- pbhjpbhj 11y ago
  They could be in ~/.local/share/meteor following "XDG Base Directory Specification" (https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html https://specifications.freedesktop.org/basedir-spec/basedir-... - seems well used on my Kubuntu but it's a case of "pick a standard" I guess).
alphapapa 11y ago
For dealing with this problem, as well as anything that's not already packaged, the checkinstall utility is very helpful. It intercepts filesystem calls and builds deb/rpm/tgz packages that you install with your distro's package manager. It's been packaged in Debian for a long time. Highly recommended.
nksjndfdf 11y ago
> Why would you install software libraries and binaries to $HOME? Why not? It makes more sense than global installations sometimes.

Sean1708 11y ago

What amuses me is that the people who are vehemently against using a bash script are often the same people who will run $ ./configure $ make without a second thought.

merb 11y ago
Actually 99% of the Time this will install it to /usr/local
- fjarlq 11y ago
  I disagree. Many (probably most) makefiles conform to the GNU Coding Standards, which recommend declaring "all" to be the default make target. `make all` does not install; that's what `make install` is for. https://www.gnu.org/prep/standards/html_node/Standard-Targets.html https://www.gnu.org/prep/standards/html_node/Standard-Target...
  - coldtea 11y ago
    The idea of the original parent comment on Make is not about well conforming makefiles. The idea is that if one doesn't trust a random curl-ed install file, they shouldn't trust a downloaded makefile either -- and for the same reasons, yet tons of people complaint about the first, but have been using the latter for decades without complain...
- coldtea 11y ago
  The idea is that the Makefile (and maybe configure?) could just as well as a "curled" install script contain ANY command, even "sudo rm -rf /". It's not about where it will install it if it works fine.
koolba 11y ago
Don't forget the most important part: sudo make install
pif 11y ago
Yes, but not as root!

nailer 11y ago

Agreed 100%. Things installed on your machine have privileges regardless of how you install them. Install software from verified sources. Whether you use curl or not is irrelevant.

raesene6 11y ago
I'd be interested in knowing, how do you assess whether something is a "trusted source" or not?
- coldtea 11y ago
  The same way we trust people. By reputation and familiarity. If something is from e.g. Red Hat or a large project repository like Apache, it's more trusted than a random website with some unknown tarball... If many people in websites and forums say "don't trust this program", well, don't trust it. Etc.
  - raesene6 11y ago
    Ahh but that's one of the hardest pieces of the trust puzzle. On the internet, how do you know who's who. you download a package from npm. Who wrote it, what's their background, where do they work? What are their security practices like, what are the security practices of the hosting company? so we say Redhat/Apache are more trusted sure, that's one of the reasons that I dislike curl|bash from random sites, there's no way of assessing trust in that software or its origin...
    - coldtea 11y ago
      >On the internet, how do you know who's who. you download a package from npm. Who wrote it, what's their background, where do they work? What are their security practices like, what are the security practices of the hosting company? Check the npm package's ratings, and the github repository watchers and stars. Try to find and read a few posts (and their comments) on the package. The more respected the source (e.g. HN comments vs comments on Digg), the better.

jfindley 11y ago

I don't think that's nearly as clever a counter-argument against curl|bash as they appear to. A few points of the shortcomings of curl|bash vs RPM and others: * curl|bash is not transactional. If you have an install/upgrade process in a different tab, and you close your terminal accidentally, how do you know if the install/upgrade completed? * There's no obvious uninstall method, and you need to read the docs to see what, if anything, you should remove outside the main directory. * There's no way of verifying if files have been tampered with post installation. This has negative security and operational consequences. * It's much more vulnerable to MiTM attacks - a proxy with a successfully faked certificate can trivially modify a bash script on the wire and add a malicious command to it. This is hard for a user to detect. Packages are (on most distros) installed via package managers that verify GPG signing keys, making in-flight modification very very greatly more difficult. * Packages are much more auditable - I can download a package and inspect it, and know that it will run the same actions on every machine it's deployed to. curl|bash can trivially serve different instructions to different requests, based on whatever secret criteria the server operator decides. If you're going to try and argue that instead of doing what the generally accepted $good_thing is, and instead use $quick_and_easy solution that's widely perceived to be insecure, when you argue in defence of your position, you need to make very sure that you're not just trying to justify laziness. I don't think sandstorm have really thought about this hard enough.

ithkuil 11y ago
I think the many discussions about this conflates two things: 1. "curl|bash" vs "curl >install.sh; sh install.sh" 2. install scripts vs package managers The OP talks about how to detect a streaming execution of a script and exploit it. This proof of concept exploit aims at discouraging the the use of curl|bash as a means to installing code, and instead promotes that at least you should save the script (and review it) (i.e. the "curl >install.sh; sh install.sh"). All of the points you mentioned are valid and important pain points that affect any arbitrary executable install script (not verified cryptographically using a side channel). Deciding whether to use an install script or a packaging system is really a tradeoff. The "obvious" problem the OP was talking about was that smell you feel when you see piping code to bash. I think OP made an excellent job showing how you can make it harder for people to spot and audit malicious code and how this is pretty serious downside for piping code into bash; It's more like: "I have a gut feeling this is horrible, so let's think about a way to prove that gut feeling", and the proof is insightful and non obvious.

donquichotte 11y ago

Recently I stumbled across hashpipe: "hashpipe helps you venture into the unknown. It reads from stdin, checks the hash of the content, and outputs it IF AND ONLY IF it matches the provided hash checksum." [https://github.com/jbenet/hashpipe https://github.com/jbenet/hashpipe]

coredog64 11y ago
I was going to try it, but it didn't have a curl|bash install instruction.
- kenny_r 11y ago
  I know you're being facetious but `go get` is just as convenient and is a controversial installation method for different reasons.

NDizzle 11y ago

Every time I see that I just wget the script and inspect it. If it does anything too clever, I don't install it that way. If I can't reason about the shell script something is wrong. It's like being emailed a .bat file in 1999. Would you blindly run it on your Windows box?! I hope not!

tamana 11y ago
The script downloads a binary onto your system. How would you inspect that?

goldbrick 11y ago

That article is good, but keybase.io is a non-starter until they open registration. I have been on the pending queue for over a year now.

kentonv 11y ago
You don't need a Keybase account to follow the Sandstorm PGP-verified installation instructions.