5 ms·

> Use an SSD: If you can get your hands on a buttload of memory, try using a tmpfs mount instead. For i/o heavy builds it's like turning on ludicrous speed. >

> Use an SSD: If you can get your hands on a buttload of memory, try using a tmpfs mount instead. For i/o heavy builds it's like turning on ludicrous speed. > trying to automate everything in a language where failures are silent set -u -e -o pipefail > and can do exciting things like call "rm -rf /" when you meant "rm -rf ${foo}/${bar}" I mean, if you're calling any recursive-file-unlinking thing from Python you face the same problem. Same solution applies: normalize the path (readlink, realpath) and test if it's "/" before adding an argument to the command. (Also don't use -f unless you need to, it ignores errors) set -e -u -o pipefail _rm_r () { declare -a rmargs for arg in "$@" ; do fullpath="$(readlink -f "$arg")" if [ ! "$fullpath" = "/" ] ; then rmargs+=("$fullpath") fi done rm -r "${rmargs[@]}" } _rm_r "${foo}/${bar}" But I agree with OP that you should use any language you're most familiar with to produce the best results possible in the least amount of time.

gravypod 5y ago
In addition: you can use overlayfs + tmpfs to run the same build with multiple build args without using any extra memory. I had an experiment I was working on to run tests for mutation testing that I want to open source at some point that used this same principle.
ezekiel68 5y ago
Yeah, the rancor for cheap laughs schtick fell flat for me >Avoid shell hackery. This is probably controversial You bet. It's less than 5 minutes to learn how to make shell _not_ fail silently and an additional 10 minutes (I'm overstating the time) to figure out how to check variables for whether they will evaluate to empty strings -- to avoid deletion catastropes. Not saying anyone should be "forced" to learn this. It's simply more accurate and helpful to say, "I don't know and I don't want to learn" instead of blaming a language.
- moyix 5y ago
  Sure, sorry if I was too harsh on bash here :) It really wasn't the right tool for the job in some parts of my use case though – parsing info out of Sources.gz using awk and sed and friends is just plain bad compared to using python-debian to get the info directly from the apt database.
  - ezekiel68 5y ago
    Thanks for taking my criticism as constructive! (that's how I meant it.) I found myself very interested in the exploration and discoveries that guided your process.
  - mst 5y ago
    I read it as somebody who knows they're sufficiently bad at bash to not be safe with it admitting that and doing something more sensible for their specific context. NB: When I say "sufficiently bad at bash" I think of myself first. Just because I know about 'set -e' and friends doesn't mean I trust myself to get it right nor do I believe anybody else should trust me to either.
  - chubot 5y ago
    But Python works great when called from shell! I learned Python first, then shell, so my shell scripts largely use Python instead of sed/awk (except for the most basic cases). This includes both inline invocations like python -c, but more commonly simply a .py file in the same directory as the shell script. So I think of shell as the main(), and then it often calls Python and R. (One big reason I started using shell was for preparing data in Python and then analyzing it in R.) That said, I think shell needs better support for JSON and TSV so you can interoperate with languages like Python, JS, and R better. Things like the 'read' builtin aren't great for this.
- Grimburger 5y ago
  > "I don't know and I don't want to learn" instead of blaming a language. I'm a staunch proponent of "anything over 100 lines should not be written in bash". You can do many more things easier and with less google-fu to find your niche SO answer. Unless you have absolutely no clue about other scripting languages that already exist on Debian, just don't do it, there's so many better options that actually last and can even be read by other people.
  - 0xbadcafebee 5y ago
    I used to say the same thing, back when the only bash I knew was from whatever tidbits I picked up as I needed them. Then I got more familiar with bash. Now, nearly everything I write is bash. I have realized that more complex languages mostly just make things.... more complex, rather than simplifying them. For simple solutions I stick to simple tools, even if it takes more than 100 lines. Here's one tool I wrote in Bash. It's an alternative to Terragrunt. 616 lines. https://github.com/pwillis-els/terraformsh https://github.com/pwillis-els/terraformsh Here's another one. A static-app package manager/versioner/pinner thing. 409 lines of POSIX shell (the code is intentionally ugly, don't take this as an example of good code). https://github.com/peterwwillis/clinst/blob/main/clinst https://github.com/peterwwillis/clinst/blob/main/clinst For both of these tools, 90% of what they do is just managing environment variables and calling other programs, and that's what shell scripts are best at.
rwmj 5y ago
> If you can get your hands on a buttload of memory, try using a tmpfs mount instead. If you don't have enough memory, using an nbdkit tmpdisk (https://libguestfs.org/nbdkit-tmpdisk-plugin.1.html https://libguestfs.org/nbdkit-tmpdisk-plugin.1.html) is a middle ground. It's backed by disk, but disables flushes which gives you a good 30% speed boost. (Obviously only use this for stuff you don't care about / can reproduce, because disabling flushes is dangerous and will lose data if the machine crashes).
moyix 5y ago
I do have a buttload(ish) of memory (256GB), but given that I'm running 32 builds in parallel (which themselves often run in parallel) I was a bit worried about the amount of RAM needed for compilation (depending on the linker, things like LLVM can consume huge amounts of RAM during compilation). It's a cool idea though, maybe I'll give it a try for my next rebuild :) The pipefail bit is a nice trick; I should remember to do this by habit when writing bash scripts (and to send the script through shellcheck)!
- mortenlarsen 5y ago
  Maybe you should also look into eatmydata. It can speed up apt (when installing dependencies) by an order of magnitude. It will also speed up your builds in general. (I am guessing you don't care about a half built package in case of a power failure)
  - moyix 5y ago
    This does look pretty nice, although I am not sure how well it would play with bear, which also uses LD_PRELOAD to intercept some calls.
- gravypod 5y ago
  Have you thought of using something like container snapshots to build the full dep tree of things off shared container images? For example: you build ncurses, snapshot the layer it produces, then use that to build nano, vi, etc.
  - moyix 5y ago
    At the moment all the build-deps are installed as binary packages, so I think this wouldn't work? I thought of trying to figure out which packages had shared build dependencies and building a tree of images that way, but solving the dependency constraints to get an efficient tree sounded unpleasant. I did experiment with just preinstalling the top 300 or so most common build dependencies, but this ended up causing more problems due to conflicts (and not just version conflicts – some packages behave differently at build time depending on what's already installed; an example is any R package which will use my old nemesis xvfb-run if it's present on the system [1]) than it saved in time or space. I do feel a bit guilty about how much I'm hammering the debian repos though, so I will probably set up a local apt mirror and point my sources.list to that instead next time. [1] https://salsa.debian.org/r-pkg-team/dh-r/-/blob/master/dh/R.pm#L165-171 https://salsa.debian.org/r-pkg-team/dh-r/-/blob/master/dh/R....
neilv 5y ago
`tmpfs` for output files of huge distro builds has been great, IME. (This was the first thing I tried when I got a big new compute server at home.) For source files, I'm thinking SSD, and leave some of the RAM for filesystem caches.
nerdponx 5y ago
Not exactly the same issue, but this general category of silliness is why I use Zsh for most of my scripts, instead of Bash or "Posix" Sh, whenever I can get away with it. The programmer experience in Zsh is much better than Bash around anything related to parameter expansion, quoting, and arrays, while still more or less using the same idioms and syntax. In addition, the Zsh equivalent of the Bash "strict mode" is a bit stricter. Here's mine: emulate zsh setopt \ err_exit \ pipe_fail \ warn_create_global \ warn_nested_var \ no_unset https://git.sr.ht/~wintershadows/dotfiles/tree/master/item/lib/zsh/strict-mode.zsh https://git.sr.ht/~wintershadows/dotfiles/tree/master/item/l...

gravypod 5y ago

In addition: you can use overlayfs + tmpfs to run the same build with multiple build args without using any extra memory. I had an experiment I was working on to run tests for mutation testing that I want to open source at some point that used this same principle.

ezekiel68 5y ago

Yeah, the rancor for cheap laughs schtick fell flat for me >Avoid shell hackery. This is probably controversial You bet. It's less than 5 minutes to learn how to make shell _not_ fail silently and an additional 10 minutes (I'm overstating the time) to figure out how to check variables for whether they will evaluate to empty strings -- to avoid deletion catastropes. Not saying anyone should be "forced" to learn this. It's simply more accurate and helpful to say, "I don't know and I don't want to learn" instead of blaming a language.

moyix 5y ago
Sure, sorry if I was too harsh on bash here :) It really wasn't the right tool for the job in some parts of my use case though – parsing info out of Sources.gz using awk and sed and friends is just plain bad compared to using python-debian to get the info directly from the apt database.
- ezekiel68 5y ago
  Thanks for taking my criticism as constructive! (that's how I meant it.) I found myself very interested in the exploration and discoveries that guided your process.
- mst 5y ago
  I read it as somebody who knows they're sufficiently bad at bash to not be safe with it admitting that and doing something more sensible for their specific context. NB: When I say "sufficiently bad at bash" I think of myself first. Just because I know about 'set -e' and friends doesn't mean I trust myself to get it right nor do I believe anybody else should trust me to either.
- chubot 5y ago
  But Python works great when called from shell! I learned Python first, then shell, so my shell scripts largely use Python instead of sed/awk (except for the most basic cases). This includes both inline invocations like python -c, but more commonly simply a .py file in the same directory as the shell script. So I think of shell as the main(), and then it often calls Python and R. (One big reason I started using shell was for preparing data in Python and then analyzing it in R.) That said, I think shell needs better support for JSON and TSV so you can interoperate with languages like Python, JS, and R better. Things like the 'read' builtin aren't great for this.
Grimburger 5y ago
> "I don't know and I don't want to learn" instead of blaming a language. I'm a staunch proponent of "anything over 100 lines should not be written in bash". You can do many more things easier and with less google-fu to find your niche SO answer. Unless you have absolutely no clue about other scripting languages that already exist on Debian, just don't do it, there's so many better options that actually last and can even be read by other people.
- 0xbadcafebee 5y ago
  I used to say the same thing, back when the only bash I knew was from whatever tidbits I picked up as I needed them. Then I got more familiar with bash. Now, nearly everything I write is bash. I have realized that more complex languages mostly just make things.... more complex, rather than simplifying them. For simple solutions I stick to simple tools, even if it takes more than 100 lines. Here's one tool I wrote in Bash. It's an alternative to Terragrunt. 616 lines. https://github.com/pwillis-els/terraformsh https://github.com/pwillis-els/terraformsh Here's another one. A static-app package manager/versioner/pinner thing. 409 lines of POSIX shell (the code is intentionally ugly, don't take this as an example of good code). https://github.com/peterwwillis/clinst/blob/main/clinst https://github.com/peterwwillis/clinst/blob/main/clinst For both of these tools, 90% of what they do is just managing environment variables and calling other programs, and that's what shell scripts are best at.

rwmj 5y ago

> If you can get your hands on a buttload of memory, try using a tmpfs mount instead. If you don't have enough memory, using an nbdkit tmpdisk (https://libguestfs.org/nbdkit-tmpdisk-plugin.1.html https://libguestfs.org/nbdkit-tmpdisk-plugin.1.html) is a middle ground. It's backed by disk, but disables flushes which gives you a good 30% speed boost. (Obviously only use this for stuff you don't care about / can reproduce, because disabling flushes is dangerous and will lose data if the machine crashes).

moyix 5y ago

I do have a buttload(ish) of memory (256GB), but given that I'm running 32 builds in parallel (which themselves often run in parallel) I was a bit worried about the amount of RAM needed for compilation (depending on the linker, things like LLVM can consume huge amounts of RAM during compilation). It's a cool idea though, maybe I'll give it a try for my next rebuild :) The pipefail bit is a nice trick; I should remember to do this by habit when writing bash scripts (and to send the script through shellcheck)!

mortenlarsen 5y ago
Maybe you should also look into eatmydata. It can speed up apt (when installing dependencies) by an order of magnitude. It will also speed up your builds in general. (I am guessing you don't care about a half built package in case of a power failure)
- moyix 5y ago
  This does look pretty nice, although I am not sure how well it would play with bear, which also uses LD_PRELOAD to intercept some calls.
gravypod 5y ago
Have you thought of using something like container snapshots to build the full dep tree of things off shared container images? For example: you build ncurses, snapshot the layer it produces, then use that to build nano, vi, etc.
- moyix 5y ago
  At the moment all the build-deps are installed as binary packages, so I think this wouldn't work? I thought of trying to figure out which packages had shared build dependencies and building a tree of images that way, but solving the dependency constraints to get an efficient tree sounded unpleasant. I did experiment with just preinstalling the top 300 or so most common build dependencies, but this ended up causing more problems due to conflicts (and not just version conflicts – some packages behave differently at build time depending on what's already installed; an example is any R package which will use my old nemesis xvfb-run if it's present on the system [1]) than it saved in time or space. I do feel a bit guilty about how much I'm hammering the debian repos though, so I will probably set up a local apt mirror and point my sources.list to that instead next time. [1] https://salsa.debian.org/r-pkg-team/dh-r/-/blob/master/dh/R.pm#L165-171 https://salsa.debian.org/r-pkg-team/dh-r/-/blob/master/dh/R....

neilv 5y ago

`tmpfs` for output files of huge distro builds has been great, IME. (This was the first thing I tried when I got a big new compute server at home.) For source files, I'm thinking SSD, and leave some of the RAM for filesystem caches.

nerdponx 5y ago

Not exactly the same issue, but this general category of silliness is why I use Zsh for most of my scripts, instead of Bash or "Posix" Sh, whenever I can get away with it. The programmer experience in Zsh is much better than Bash around anything related to parameter expansion, quoting, and arrays, while still more or less using the same idioms and syntax. In addition, the Zsh equivalent of the Bash "strict mode" is a bit stricter. Here's mine: emulate zsh setopt \ err_exit \ pipe_fail \ warn_create_global \ warn_nested_var \ no_unset https://git.sr.ht/~wintershadows/dotfiles/tree/master/item/lib/zsh/strict-mode.zsh https://git.sr.ht/~wintershadows/dotfiles/tree/master/item/l...