5 ms·

Now just make "-j" the default and all is good :) Why shouldn't make use the available resources by default?

by Aardwolf 7y ago

bicolao 7y ago
It exposes bugs in your make rules.
- IshKebab 7y ago
  You mean not using `-j` exposes bugs? Doubtful. If you are missing a dependency edge in your build graph then there are no guarantees that it will cause an issue with or without `-j`. The only way to properly solve missing dependencies is to hide targets unless they are explicitly depended on. The only systems I know that do that are Nix and Bazel (and its clones).
  - deleted 7y ago
    [deleted]
  - asveikau 7y ago
    If your rules have side-effects that aren't reflected in the dependency part of the declaration but nonetheless depended upon, running the jobs in parallel means the order of execution varies instead of being sequential, so sometimes you may get a file not found on something produced in an earlier rule. If make knows about all the inputs and outputs of every rule, it can make sure dependencies are built before they are used. The problem occurs when the makefile doesn't tell make about every dependency. Also the rule scripts themselves can have race conditions. A trivial example is if they end up using the same temp filename when doing work.
    - IshKebab 7y ago
      Yes I know. But the order of execution can vary even if you don't run in parallel. For example if you have two targets, A and B, and B depends on A's output but you forgot to say so, then `make` can build B before A even if you don't use the `-j` option.
      - asveikau 7y ago
        I thought the behavior I have seen is it's always in the order that the targets are specified. So if you have: all: A B Then if both need a rebuild A is always first, except when -j breaks that. But I can't say I have played close attention to every time I have typed make.
        IshKebab 7y ago
        Ah right, but that's just a lucky coincidence. Make is perfectly within its rights to randomise the order of A and B even if you don't specify `-j`. And if you're driving Make from some other system, e.g. CMake, you really have no idea what order stuff is going to end up in the makefiles anyway.
- klyrs 7y ago
  I prefer processes that expose bugs over those that obscure them...
  - jschwartzi 7y ago
    I've learned over my engineering career to prefer rules which are based on the facts of the situation at hand rather than rules which are applied dogmatically. Dogma is what gave us Chernobyl.
0xcde4c3db 7y ago
I've had a "make -j" fail due to virtual memory exhaustion (granted, on a 32-bit VM). This was on a C++ project, so it might be less of a problem when building the kernel, but it's definitely not a reasonable default across the board.
- uep 7y ago
  I've had similar issues, though it was actually caused by Ninja, and not Make. I was doing a build of clang+llvm itself, and somehow all the link steps ended up happening about the same time. It really exhausted the memory on my system. My system has 32 GiB, of which I rarely go above 10GiB used except for those giant builds.
swiley 7y ago
I really hate it when people do that and don’t provide a way to disable it. There are a number of python libraries I can’t install on my laptop easily because the native code runs out of physical (4GB!) memory spawning compiler jobs and everything starts thrashing, it’s just dumb to go forking without asking the user first. People who really want it can give make the -j flag.
- alfalfasprout 7y ago
  Bazel is the worst at this. When you try to eg; build tensorflow from source it maxes out all the cores, uses up as much memory as it can, and then crashes when the OOM killer eventually kills it. This is even on a 64GB 64 core machine. Manually specifying resources doesn't work well either because it uses 2x+ the resources you allocate it.
- stassats 7y ago
  You can do "taskset --cpu-list 0 make -j"
war1025 7y ago
Memory exhaustion for one. In my experience build systems will happily eat up all of your memory and put you into a swap hell that is difficult to escape from. Especially painful when one of your coworkers has a beefier machine than you and changes the number of parallel jobs offhand one day, causing your computer to freeze until the OOM killer gets the right build process.

Now just make "-j" the default and all is good :) Why shouldn't make use the available resources by default?

bicolao 7y ago

It exposes bugs in your make rules.

IshKebab 7y ago
You mean not using `-j` exposes bugs? Doubtful. If you are missing a dependency edge in your build graph then there are no guarantees that it will cause an issue with or without `-j`. The only way to properly solve missing dependencies is to hide targets unless they are explicitly depended on. The only systems I know that do that are Nix and Bazel (and its clones).
- deleted 7y ago
  [deleted]
- asveikau 7y ago
  If your rules have side-effects that aren't reflected in the dependency part of the declaration but nonetheless depended upon, running the jobs in parallel means the order of execution varies instead of being sequential, so sometimes you may get a file not found on something produced in an earlier rule. If make knows about all the inputs and outputs of every rule, it can make sure dependencies are built before they are used. The problem occurs when the makefile doesn't tell make about every dependency. Also the rule scripts themselves can have race conditions. A trivial example is if they end up using the same temp filename when doing work.
  - IshKebab 7y ago
    Yes I know. But the order of execution can vary even if you don't run in parallel. For example if you have two targets, A and B, and B depends on A's output but you forgot to say so, then `make` can build B before A even if you don't use the `-j` option.
    - asveikau 7y ago
      I thought the behavior I have seen is it's always in the order that the targets are specified. So if you have: all: A B Then if both need a rebuild A is always first, except when -j breaks that. But I can't say I have played close attention to every time I have typed make.
      - IshKebab 7y ago
        Ah right, but that's just a lucky coincidence. Make is perfectly within its rights to randomise the order of A and B even if you don't specify `-j`. And if you're driving Make from some other system, e.g. CMake, you really have no idea what order stuff is going to end up in the makefiles anyway.
klyrs 7y ago
I prefer processes that expose bugs over those that obscure them...
- jschwartzi 7y ago
  I've learned over my engineering career to prefer rules which are based on the facts of the situation at hand rather than rules which are applied dogmatically. Dogma is what gave us Chernobyl.

0xcde4c3db 7y ago

I've had a "make -j" fail due to virtual memory exhaustion (granted, on a 32-bit VM). This was on a C++ project, so it might be less of a problem when building the kernel, but it's definitely not a reasonable default across the board.

uep 7y ago
I've had similar issues, though it was actually caused by Ninja, and not Make. I was doing a build of clang+llvm itself, and somehow all the link steps ended up happening about the same time. It really exhausted the memory on my system. My system has 32 GiB, of which I rarely go above 10GiB used except for those giant builds.

swiley 7y ago

I really hate it when people do that and don’t provide a way to disable it. There are a number of python libraries I can’t install on my laptop easily because the native code runs out of physical (4GB!) memory spawning compiler jobs and everything starts thrashing, it’s just dumb to go forking without asking the user first. People who really want it can give make the -j flag.

alfalfasprout 7y ago
Bazel is the worst at this. When you try to eg; build tensorflow from source it maxes out all the cores, uses up as much memory as it can, and then crashes when the OOM killer eventually kills it. This is even on a 64GB 64 core machine. Manually specifying resources doesn't work well either because it uses 2x+ the resources you allocate it.
stassats 7y ago
You can do "taskset --cpu-list 0 make -j"

war1025 7y ago

Memory exhaustion for one. In my experience build systems will happily eat up all of your memory and put you into a swap hell that is difficult to escape from. Especially painful when one of your coworkers has a beefier machine than you and changes the number of parallel jobs offhand one day, causing your computer to freeze until the OOM killer gets the right build process.