3 ms·

I do have a buttload(ish) of memory (256GB), but given that I'm running 32 builds in parallel (which themselves often run in parallel) I was a bit worried about

by moyix 5y ago

I do have a buttload(ish) of memory (256GB), but given that I'm running 32 builds in parallel (which themselves often run in parallel) I was a bit worried about the amount of RAM needed for compilation (depending on the linker, things like LLVM can consume huge amounts of RAM during compilation). It's a cool idea though, maybe I'll give it a try for my next rebuild :) The pipefail bit is a nice trick; I should remember to do this by habit when writing bash scripts (and to send the script through shellcheck)!

mortenlarsen 5y ago
Maybe you should also look into eatmydata. It can speed up apt (when installing dependencies) by an order of magnitude. It will also speed up your builds in general. (I am guessing you don't care about a half built package in case of a power failure)
- moyix 5y ago
  This does look pretty nice, although I am not sure how well it would play with bear, which also uses LD_PRELOAD to intercept some calls.
gravypod 5y ago
Have you thought of using something like container snapshots to build the full dep tree of things off shared container images? For example: you build ncurses, snapshot the layer it produces, then use that to build nano, vi, etc.
- moyix 5y ago
  At the moment all the build-deps are installed as binary packages, so I think this wouldn't work? I thought of trying to figure out which packages had shared build dependencies and building a tree of images that way, but solving the dependency constraints to get an efficient tree sounded unpleasant. I did experiment with just preinstalling the top 300 or so most common build dependencies, but this ended up causing more problems due to conflicts (and not just version conflicts – some packages behave differently at build time depending on what's already installed; an example is any R package which will use my old nemesis xvfb-run if it's present on the system [1]) than it saved in time or space. I do feel a bit guilty about how much I'm hammering the debian repos though, so I will probably set up a local apt mirror and point my sources.list to that instead next time. [1] https://salsa.debian.org/r-pkg-team/dh-r/-/blob/master/dh/R.pm#L165-171 https://salsa.debian.org/r-pkg-team/dh-r/-/blob/master/dh/R....
0xbadcafebee 5y ago
Holy crap that's a lot of parallel builds. One thing to try would be fewer parallel builds, but increase the parallelization of the build tool per package, e.g. make -j16. Combined with the tmpfs and less resource contention from other parallel builds, it may end up faster, and you can clean up after each package so your tmpfs only has a couple of packages' worth of build files at once. Also, are you using ccache / distcc?
- moyix 5y ago
  The debian build system already runs `make -j$(nproc)` whenever the package supports it. So in principle sometimes I'm running 1024 (32*32) concurrent compilations, but it's rare for that many to actually be active at once (most builds have large portions that can't be parallelized, like running ./configure or tarring up the build artifacts at the end) – right now I have 280 processes running in 32 containers. Not using ccache/distcc since I'm just running this on my own workstation. I'm not sure ccache would help here because each build runs in an isolated container environment, to prevent cross-contamination with other builds. I'm also instrumenting each build using bear to generate a compilation database, and I think distcc would mess with that? (I can't imagine distcc knows how to transfer over the bear wrapper binaries / shared libs?)
  - 0xbadcafebee 5y ago
    Well the idea with ccache/distcc is to store the cache on the host or a shared volume and reference it from your containers. Your container builds remain isolated but don't waste time rebuilding objects that were already compiled (and re-compiling obvs gets much faster). Assuming same architecture, compiler, linker, etc I think it won't affect the isolated-build-ness. Here's some examples: https://cinaq.com/blog/2020/05/10/speed-up-docker-builds-with-distcc-ccache-and-kubernetes/ https://cinaq.com/blog/2020/05/10/speed-up-docker-builds-wit... https://stackoverflow.com/questions/39650056/using-ccache-when-building-inside-of-docker https://stackoverflow.com/questions/39650056/using-ccache-wh... To be honest it's been... a very long time... since I've used these. But the theory is that it all "just works", in that if it can compile something with cache or remotely, it does, but if not, it just works as normal. So I think it would work for your use case? Maybe a Gentoo user can correct me :)

by moyix 5y ago

mortenlarsen 5y ago

Maybe you should also look into eatmydata. It can speed up apt (when installing dependencies) by an order of magnitude. It will also speed up your builds in general. (I am guessing you don't care about a half built package in case of a power failure)

moyix 5y ago
This does look pretty nice, although I am not sure how well it would play with bear, which also uses LD_PRELOAD to intercept some calls.

gravypod 5y ago

Have you thought of using something like container snapshots to build the full dep tree of things off shared container images? For example: you build ncurses, snapshot the layer it produces, then use that to build nano, vi, etc.

moyix 5y ago
At the moment all the build-deps are installed as binary packages, so I think this wouldn't work? I thought of trying to figure out which packages had shared build dependencies and building a tree of images that way, but solving the dependency constraints to get an efficient tree sounded unpleasant. I did experiment with just preinstalling the top 300 or so most common build dependencies, but this ended up causing more problems due to conflicts (and not just version conflicts – some packages behave differently at build time depending on what's already installed; an example is any R package which will use my old nemesis xvfb-run if it's present on the system [1]) than it saved in time or space. I do feel a bit guilty about how much I'm hammering the debian repos though, so I will probably set up a local apt mirror and point my sources.list to that instead next time. [1] https://salsa.debian.org/r-pkg-team/dh-r/-/blob/master/dh/R.pm#L165-171 https://salsa.debian.org/r-pkg-team/dh-r/-/blob/master/dh/R....

0xbadcafebee 5y ago

Holy crap that's a lot of parallel builds. One thing to try would be fewer parallel builds, but increase the parallelization of the build tool per package, e.g. make -j16. Combined with the tmpfs and less resource contention from other parallel builds, it may end up faster, and you can clean up after each package so your tmpfs only has a couple of packages' worth of build files at once. Also, are you using ccache / distcc?

moyix 5y ago
The debian build system already runs `make -j$(nproc)` whenever the package supports it. So in principle sometimes I'm running 1024 (32*32) concurrent compilations, but it's rare for that many to actually be active at once (most builds have large portions that can't be parallelized, like running ./configure or tarring up the build artifacts at the end) – right now I have 280 processes running in 32 containers. Not using ccache/distcc since I'm just running this on my own workstation. I'm not sure ccache would help here because each build runs in an isolated container environment, to prevent cross-contamination with other builds. I'm also instrumenting each build using bear to generate a compilation database, and I think distcc would mess with that? (I can't imagine distcc knows how to transfer over the bear wrapper binaries / shared libs?)
- 0xbadcafebee 5y ago
  Well the idea with ccache/distcc is to store the cache on the host or a shared volume and reference it from your containers. Your container builds remain isolated but don't waste time rebuilding objects that were already compiled (and re-compiling obvs gets much faster). Assuming same architecture, compiler, linker, etc I think it won't affect the isolated-build-ness. Here's some examples: https://cinaq.com/blog/2020/05/10/speed-up-docker-builds-with-distcc-ccache-and-kubernetes/ https://cinaq.com/blog/2020/05/10/speed-up-docker-builds-wit... https://stackoverflow.com/questions/39650056/using-ccache-when-building-inside-of-docker https://stackoverflow.com/questions/39650056/using-ccache-wh... To be honest it's been... a very long time... since I've used these. But the theory is that it all "just works", in that if it can compile something with cache or remotely, it does, but if not, it just works as normal. So I think it would work for your use case? Maybe a Gentoo user can correct me :)