10 ms·
Show HN: Checksum.sh verify every install script
The pattern of downloading and executing installation scripts without verifying them has bothered me for a while.
I started messing around with a way to verify the checksum of scripts before I execute them. I've found it a really useful tool for installing things like Rust or Deno.
It's written entirely as a shell script, and it's easy to read and understand what's happening.
I hope it may be useful to someone else!
- ithkuil 4y agoAwesome. I made something similar in https://github.com/mkmik/runck https://github.com/mkmik/runck But I didn't but a fancy domain name :-)
- nerdponx 4y agoWhy not use the -c option? Especially if you're using Bash or Zsh which has "here-strings": checksum() { hash="$1" file="$2" sha256sum -c <<< "${hash} ${file}" } Or if you need to use a POSIX-ish shell: checksum() { hash="$1" file="$2" printf '%s %s' "$hash" "$file" | sha256sum -c } Of course you can add a `--binary` option (uses '%s *%s' instead of '%s %s'), options to use different hash functions, etc. I also think it's weird to use `alias` inside a function, instead of just using a parameter to store the name of the program to execute.
- gavinuhma 4y agoGreat point on alias, thanks. I think that was a relic of an older iteration. I'll work through these suggestions. Appreciate it. Feel free to send a PR if you want. For the here string I think that won't work because the file isn't being saved locally, it's just being piped (so $2 is a URL). I can't do the usual `shasum -c <<< "132e320edb0027470bfd836af8dadf174e4fee00 install.sh" which takes a local filename but not the file content. As far as I could tell anyway. I'll try it some more
- koolba 4y agoJust remember that any script that fetches anything else remotely would still pass the checksum as only the initial script is checked.
- gavinuhma 4y agoDefinitely. Important to note. There is a long long supply chain
- ChadNauseam 4y agoYep. As an example, rustup happens to be in this category as the checksums for rustc, cargo, etc. aren't checked.
- gavinuhma 4y agoIt's really interesting. There should be a massive ledger of checksums for software
- jandrese 4y agoIt's called apt. Or dnf. Or most any package manager. Having a gigantic general list runs into the problem of how do you update it and how do you verify the updates?
- yjftsjthsd-h 4y agoYou use GPG and trust the people publishing things, who sign the artifact that you actually download. Which is internally how every package manager I've seen works internally, anyways.
- bigiain 4y ago> You use GPG “and now you have two problems.” —jwz We haven’t been able to trust public pgp keyservers for a decade or more (possibly never, really). So now we’re back at having to trust where-ever we get the proof from, whether that’s the file hash, or the public key. (Which, as you say, is what package managers provide, and if you don’t trust your system’s apt/yum/pacman/whatever, then you have a bigger problem that trusting any random install shell script)
- dundarious 4y agoThere are two big problems with the use of `echo $s` in bash/POSIX sh: 1. Never use echo to output untrusted content as the first argument Let's say `s='-e 1\n2'`, then `echo $s` will output: > 1 > 2 Instead of: > -e 1\n2 Always use printf if you want to start output with untrusted content, e.g., `printf %s\\n "$s"`. 2. Never use unquoted variable expansion when trying to exactly reproduce contents of the variable Similarly, unquoted variable expansion re-tokenizes the contents and will not preserve spaces appropriately. Say `s='"a<space><space>b"'` (where each <space> is a literal ' ', HN seems to be collapsing 2 spaces down to 1), then `echo $s` will output: > "a<space>b" Instead of: > "a<space><space>b" You can get the latter with `echo "$s"` but use `printf %s\\n "$s"` to fix both issues. PS: If you fail to use quoted expansion with printf, for example like so, `printf %s\\n $s`, then you'll notice the problem right away, as it will effectively turn that into `for i in $s ; do printf %s\\n "$i" ; done`. That's actually a very useful feature of printf if you know to use it. Edit: These problems exist for bash/POSIX sh at least. Perhaps you're using a shell that works differently, like zsh, because otherwise issue 2 would probably have led to some checksum fails for you already.
- gavinuhma 4y agoThis is awesome. Thank you! I've been through so many iterations but it's been fun to improve
- gavinuhma 4y agoLike this? https://github.com/gavinuhma/checksum.sh/pull/2 https://github.com/gavinuhma/checksum.sh/pull/2
- dundarious 4y agoMissed the other `echo $s` piped into shasum. But I echo the sentiment of the another commenter that I'd rather rely on `shasum --check` to give the OK or not.
- gavinuhma 4y agoGot it. Thanks. Re --check, I suppose the way to do that would be to download the file to disk, which --check requires as fair as I can tell. So I could download the file to disk, --check, and then remove it. I think most of these installs scripts are trying not to leave any artifacts around from install, other than the resulting binary.
- thewataccount 4y agoSerious question - What is the benefit of verifying a hash? Are we really worried about file integrity? Why don't people use GPG? The hash only verifies file integrity, and that the content of the url doesn't switch the script later. But keep in mind in most scenerios, and attacker would also just change the hash listed too (they're usually on the same website). This only mitigates one very specific attack. Why don't we use GPG here? That way we can verify ownership and file integrity with at minimum TOFU, plus optional manual verification? If we're going through the work of adding a wrapper and all that, we may as well no? This has the benefit that you only need to import the owner's cert once, all future changes have the same cert. Where hashes are obviously different every time, you have to trust the source of the hash every time it changes. With GPG at the very least you have TOFU with certs - and very best can have better assurance of the initial download too. EDIT: Just want to clarify - I'm openly asking why the "developer community" is going the direction of hashes for script verification vs GPG signatures. I don't mean to diminish your project, your project looks fun, and does make verifying hashes easier :)
- tomrod 4y agoI'm not terribly deep in this space. What is the conceptual difference of hash vs GPG sig?
- atoav 4y agoA hash is the same when the values of the content are the same. But when you get a new (maliciously hacked) install script chances are that you won't have an old hash lying around to check whether the script changed. Any attacker who could swap the sceipt could also swap the hash, unless it is a different channel. With GPG the developer has a key pair (one private, one public). They can then sign all their scripts with their private key and publish the public one wherever. You can then take that public key and verify that the script has been indeed signed by the developers private key.
- thewataccount 4y agoAdmittedly this is likely the main reason GPG isn't more common place because of the complexity. This is the overview: Developer generates a private/public key they use for all of their projects. You import their public key once - you can verify this from their github, twitter, etc but that's optional. They can sign a file with their key. You can check this signature against their public key. This will guarantee the file was signed by using that key and is unmodified. If someone hijacks the website after this point and signs the new downloads with their own key - then you will be able to see it's invalid. If you manually verify the key then you'll know your initial download is valid - if you trust on first use then you at least know all future files signed from that developer with that cert are valid. They also are effectively a hash for file integrity. tl;dr - hashes tell you if a file is changed. Signatures tell you if the file is changed, and who the person that made the file is.
- throwawaaarrgh 4y agoIf we kept a mirrored or distributed decentralized network of just cryptographic hashes, that might solve a huge number of problems around distributing files securely.
- dontbenebby 4y ago>The pattern of downloading and executing installation scripts without verifying them has bothered me for a while. Thanks for sharing this work OP! I didn't see a license mentioned -- did you intend this to go into the public domain? I like how you set up a cool domain name and did some sick graphics, but I'm not sure how I can legally use your code in the future. That being said, I appreciate the work you put into this project. I'm not going to list off specific examples, but MANY open source projects serve either PGP keys or hashes in the clear. Or they serve just hashes over HTTPS and now you have a trust issue. Or, in one case, my favorite -- they had lovingly listed out the MD5 sum for the program... but they served both that checksum, and the code itself... over HTTPS. Now, to be fair, HTTPS does provide an integrity check, so there's a benefit beyond privacy or whatever but... this is a RAMPANT problem in the open source community. I ran into it mostly when trying to find esoteric security tools when I was attempting OSCP and interviewing around for penetration testing roles. I got the sense rapidly shifting from "I was so scared of the CFAA I did an entire master's thesis on the design of censorship circumvention tools" to "Oh gee, I used to be such a narcissis, demanding a high falutin salary when I couldmn't even fire up Metasploit to wipe a server." (The implication being that some folks abused their access when my powers were week, and now, in time for spooky season, it's time lean in to letting people take whatever drug they want if they feel scared -- reality scares me too some days.)
- gavinuhma 4y agoGood catch. Let me add a license
- dontbenebby 4y agoThanks, it wasn't meant in a gotcha way.
- gavinuhma 4y agoI totally just forgot to add one. Added MIT just now. Appreciate it!
- orf 4y agoI feel like bash/sh should have this built in
- neeh0 4y agoI wrote hundreds of those checks in scripts, makefiles, CI and whatever else. After I found Nix (and NixOS) it's ridiculous not to use it. Use it.
- gavinuhma 4y agoI hadn’t heard of NixOS. Super cool
- NovemberWhiskey 4y agoI don't know; what's the threat model here? If the script is deliberately malicious as originally published, then the publisher will provide a valid checksum; so it doesn't help. If the script source is subverted by an attacker, then it only helps if the attacker doesn't also have the means to change the published checksum too. If an attacker can modify the site which publishes the URL for the script and the checksum, they can modify both at the same time.
- gavinuhma 4y agoThat’s right. The checksum shouldn’t be provided by the site. I’m producing the checksum myself after reviewing the install scripts manually. Once I produce the checksum I can keep relying on it. The install scripts don’t tend to change very often.
- jwong_ 4y agoso you’re storing the checksums locally for each script then? is that much different than just storing the verified copies of the scripts?
- IshKebab 4y agoThat makes some kind of sense. The original post makes you sound like you're one of those crazy people who thinks e.g. Flatpak is fine but curl | bash is horribly insecure. However I'm still not sure it really makes sense. Do you also manually review the code of the binaries that the bash scripts download?
- woodruffw 4y agoI think this is a worthy cause, but maybe a little misguided: the problem with "curl-piping" isn't so much the fact that you're throwing a random shell script into your shell, but the fact that you're downloading arbitrary code in a way that's disconnected from the normal integrity/authenticity guarantees of a package manager. In other words: you can be confident in the bootstrapping script you've just downloaded because it passed its checksum, but that script is just going to download more binaries from the Internet.
- michaelmior 4y ago> that script is just going to download more binaries from the Internet Not necessarily. A number of these scripts either configure a package manager or the shell script contains the binary itself which is unpacked when the script is run.
- woodruffw 4y agoSure, I suppose that's possible. Most of the ones I'm familiar with just download an architecture-compatible binary from a CDN somewhere. Even if there's a shar-style[1] packed binary in the script, you have no idea what that binary does when you verify that the checksum is correct.
- michaelmior 4y agoWell-written scripts I've seen also contain the hash of binaries that are downloaded. So as long as the hash function is good, checking the hash of the script should still ensure that the binary downloaded is what you want. > you have no idea what that binary does when you verify that the checksum is correct. This isn't any different from using a package manager. You're still downloading a binary that could do anything and you have to have some level of trust in the source.
- cryptonector 4y ago> the fact that you're downloading arbitrary code in a way that's disconnected from the normal integrity/authenticity guarantees of a package manager. I'm old enough to remember when apt packaging got burned because it used http instead of https even though apt packages get signed. If you're downloading software from websites protected with HTTPS, and that's good enough for you, then downloading and executing a script from those same websites using HTTPS is also good enough. Would it be better if those things were signed with a key for which there is a code signing certificate? Eh, maybe, yes, if the PKI for the code signing is sufficiently better than WebPKI, which... is not necessarily obvious. Meanwhile, access to that PKI is probably sufficiently harder to come by than WebPKI TLS server certificates that a lot of people don't bother, and rightly so. Now suppose you say "I don't trust this, I'm just going to clone their github repo and build from source". Do you get more protection that way? Maybe, maybe not. Now, if you get packages from Debian and the like, you get them signed, and maybe the person who contributed the package to their repository did a thorough code review and audit of the upstream they are packaging, or maybe not, who knows. This is why containerizing this stuff helps. But it's not really accessible to people yet. What might be nice is that any program that a user executes automatically gets some level of isolation corresponding to how it was delivered, authored by whom, etc. So programs from the OS get the least isolation, and programs written by the user less isolation, and programs of unknown provenance get the most isolation.
- Genghis_9000 4y agoand where do we get the checksum? HN has a hall monitor mentality issue
- gavinuhma 4y agoI generate the checksum myself after reviewing a given install script. Then I add it to the readme. And then anytime I go to install something I reuse the checksum
- kazinator 4y agoThis function is flawed, containing unquoted variable interpolations: s=$(curl -fsSL $1) ... c=$(echo $s | shasum | awk '{print $1}') what it means is that the checksum is being calculated on a whitespace-mangled version of the data that is pulled down from the web. It appears to work because the author calculated the checksums with the same script and is just validating that they are not changing. In other words, it's possible to make whitespace changes such that the hash won't change. Here are two scripts: a harmless one and a malicious one, which produce the same whitespace-ignorant SHA256: $ foo='# this is a comment > # rm -rf /' $ echo $foo | sha256sum 8b87547d4d214038b153ce57d929be4c835b7690c930c1e83a25fc1509390cf9 - $ foo='# this is a comment # > rm -rf /' $ echo $foo | sha256sum 8b87547d4d214038b153ce57d929be4c835b7690c930c1e83a25fc1509390cf9 - The first foo contains two comments. The "rm -rf /" command is commented out. The second foo moves the hash mark of the second comment into the previous line, uncommenting the command. (I know about GNU Coreutils' safeguard in rm against removing / recursively, by the way.)
- charcircuit 4y agoThis just shifts the trust to the checksum. How do you know you downloaded the right checksum? Checksum the checksum? Whatever you are doing to protect sending the checksum can also be used for protecting the script itself.
- gavinuhma 4y agoAgree. Although checksums are smaller and easier to copy/paste. Same with a url. I download the script from A, and the checksum from B. And then I verify them locally. So A and B both need to be compromised. It all assumes the script was safe to begin with, and this just verifies that nothing has changed
- tkk23 4y agoChecksum the checksum_s_ Checksum.sh could keep track of checksums. Then an attacker has to alter the original script and checksum.sh.
- remram 4y agoAn idea might be to get the checksum from the URL, for example: checksum https://sh.rustup.rs/#8327fa6ce106d2a387fdd02cb95c3607193c2edb | sh Otherwise I don't understand why your script is loaded as a function rather than run as a script.
- Arnavion 4y ago>I've found it a really useful tool for installing things like Rust or Deno. For Rust you can ignore sh.rustup.rs and just download and set up rustup manually. CARGO_HOME="${CARGO_HOME:-$HOME/.cargo}" mkdir -p "$CARGO_HOME/bin" curl -Lo "$CARGO_HOME/bin/rustup" 'https://static.rust-lang.org/rustup/dist/x86_64-unknown-linux-gnu/rustup-init' chmod +x "$CARGO_HOME/bin/rustup" hash -r rustup set auto-self-update disable rustup set profile minimal rustup default stable rustup update --force rustup self update # Create hardlinks under $CARGO_HOME/bin/
- IshKebab 4y agoAwesome, that avoids downloading and running executable code from the Rust project!
- steveklabnik 4y agorustup is still executable code you are downloading from the Rust Project. It then downloads Cargo and rustc, which are both executable code, downloaded via the Rust project. The only difference here is that you’re running a few commands by hand instead of running them in a single invocation of a shell script.
- Arnavion 4y agoYeah, the only thing this achieves is not having to worry about the `curl | sh` step. The rest of the threat model is exactly the same.
- IshKebab 4y agoOf course; I was being sarcastic.
- steveklabnik 4y agoPoe’s law strikes again, my bad!
- muterad_murilax 4y agoOP may want to take a look at "Shell script best practices" (https://news.ycombinator.com/item?id=33354286 https://news.ycombinator.com/item?id=33354286) submitted two days ago. :)
- gavinuhma 4y agoNice! Thanks for sharing. I also learned about shellcheck thanks to this thread, which has been super useful