6 ms·
An optimization to help code compilation times on big CPUs
- Aardwolf 7y agoNow just make "-j" the default and all is good :) Why shouldn't make use the available resources by default?
- bicolao 7y agoIt exposes bugs in your make rules.
- IshKebab 7y agoYou mean not using `-j` exposes bugs? Doubtful. If you are missing a dependency edge in your build graph then there are no guarantees that it will cause an issue with or without `-j`. The only way to properly solve missing dependencies is to hide targets unless they are explicitly depended on. The only systems I know that do that are Nix and Bazel (and its clones).
- deleted 7y ago[deleted]
- asveikau 7y agoIf your rules have side-effects that aren't reflected in the dependency part of the declaration but nonetheless depended upon, running the jobs in parallel means the order of execution varies instead of being sequential, so sometimes you may get a file not found on something produced in an earlier rule. If make knows about all the inputs and outputs of every rule, it can make sure dependencies are built before they are used. The problem occurs when the makefile doesn't tell make about every dependency. Also the rule scripts themselves can have race conditions. A trivial example is if they end up using the same temp filename when doing work.
- IshKebab 7y agoYes I know. But the order of execution can vary even if you don't run in parallel. For example if you have two targets, A and B, and B depends on A's output but you forgot to say so, then `make` can build B before A even if you don't use the `-j` option.
- asveikau 7y agoI thought the behavior I have seen is it's always in the order that the targets are specified. So if you have: all: A B Then if both need a rebuild A is always first, except when -j breaks that. But I can't say I have played close attention to every time I have typed make.
- IshKebab 7y agoAh right, but that's just a lucky coincidence. Make is perfectly within its rights to randomise the order of A and B even if you don't specify `-j`. And if you're driving Make from some other system, e.g. CMake, you really have no idea what order stuff is going to end up in the makefiles anyway.
- klyrs 7y agoI prefer processes that expose bugs over those that obscure them...
- jschwartzi 7y agoI've learned over my engineering career to prefer rules which are based on the facts of the situation at hand rather than rules which are applied dogmatically. Dogma is what gave us Chernobyl.
- 0xcde4c3db 7y agoI've had a "make -j" fail due to virtual memory exhaustion (granted, on a 32-bit VM). This was on a C++ project, so it might be less of a problem when building the kernel, but it's definitely not a reasonable default across the board.
- uep 7y agoI've had similar issues, though it was actually caused by Ninja, and not Make. I was doing a build of clang+llvm itself, and somehow all the link steps ended up happening about the same time. It really exhausted the memory on my system. My system has 32 GiB, of which I rarely go above 10GiB used except for those giant builds.
- swiley 7y agoI really hate it when people do that and don’t provide a way to disable it. There are a number of python libraries I can’t install on my laptop easily because the native code runs out of physical (4GB!) memory spawning compiler jobs and everything starts thrashing, it’s just dumb to go forking without asking the user first. People who really want it can give make the -j flag.
- alfalfasprout 7y agoBazel is the worst at this. When you try to eg; build tensorflow from source it maxes out all the cores, uses up as much memory as it can, and then crashes when the OOM killer eventually kills it. This is even on a 64GB 64 core machine. Manually specifying resources doesn't work well either because it uses 2x+ the resources you allocate it.
- stassats 7y agoYou can do "taskset --cpu-list 0 make -j"
- war1025 7y agoMemory exhaustion for one. In my experience build systems will happily eat up all of your memory and put you into a swap hell that is difficult to escape from. Especially painful when one of your coworkers has a beefier machine than you and changes the number of parallel jobs offhand one day, causing your computer to freeze until the OOM killer gets the right build process.
- boshomi 7y agothe git commit: pipe: use exclusive waits when reading or writing[1] [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0ddad21d3e99c743a3aa473121dc5561679e26bb https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
- georgyo 7y agoI am always amazed at the quality of linux kernel commit messages.
- wadkar 7y agoIndeed, my personal rule of thumb is if the ratio of commit message to diff is greater than one then it’s a good commit. Larger the ratio better the commit and you would do better to read the commit (as opposed to skimming through them). Chances are, you’ll learn something new.
- kees99 7y agoIndeed. Those two lines in the test-case are particularly excellent: /* 64 processes */ fork(); fork(); fork(); fork(); fork(); fork(); I can totally see most people, myself included, writing a dozen of lines instead, with a for loop, checking for parent/child/error return code, etc. Which might be preferable for some code somewhere deep inside the app logic that will potentially run on a resource-constrained machine. But for the explanation code to be read by a human, this one-liner is such a good choice!
- zvrba 7y agoFrom the linked text: > In particular, the GNU make jobserver code ends up using a pipe as a way to limit parallelism, where each job consumes a token by reading a byte from the jobserver pipe, and releases the token by writing a byte back to the pipe. Umm, that's what semaphores are for.
- jws 7y agoGNU make runs on a lot of different systems. There is not a semaphore API shared between them, but there will (almost certainly) be a usable pipe() on all of them. But… if I could, I would probably use POSIX semaphores, no wait SYS-V, no, hang on, let me read some web pages… um… let me check my repositories and see which I've used before… both… hmm… Seriously, I can't see getting so frustrated choosing that I'd just go with pipe(), but there is room for improvement in portable semaphore APIs.
- teddyh 7y agoGNU Make has been fixed, but I wonder if there are other programs which would also have similar bugs in their code? I am reminded of when systemd exposed bugs in system startup scripts, and everybody instead started hating systemd for breaking their systems.
- sumanthvepa 7y agoIt's not GNU make that got fixed. It's the pipe system call that Linus optimized. Make uses pipe as a synchronization mechanism. So all programs that use pipe this will benefit.
- mmastrac 7y agoWhat the parent comment is referring to is a pipe handling bug in early versions of GNU make that prevented kernel optimizations from happening earlier on.
- deleted 7y ago[deleted]
- uep 7y agoI think this happens a lot with the Linux kernel. Where they struggle to maintain backwards compatibility, in the face of applications relying on undocumented implementation quirks. I think this case is a bit different, in that the bug already triggers sometimes without this change, the change just makes it happen more often. Which is why the GNU Make bug report predates this improvement.
- chris_wot 7y agoSuddenly LibreOffice build time’s increase substantially.
- FartyMcFarter 7y agoWhat do you mean?
- chris_wot 7y agoLibreOffice relies extensively on parallel build from Gnu make. Edit: crap, that autocorrected on my iPhone - that should have been decrease!
- boris 7y agoGood example of how we develop software these days: instead of fixing GNU make by ripping out that antiquated pipe-based jobserver and replacing it with proper multi-threaded parallelism[1], we rather optimize the operating system to make the antiquated stuff work better. [1] Having hacked on GNU make I know this won't be easy (it's quite a mess). In fact, it would probably be easier to replace it entirely and fix some of its other deficiencies in the process.
- jankotek 7y agoMaybe there is a good reason for that. There are already dozens of alternatives to GNU Make. Linux already did something similar with Git.
- s3cur3 7y agoReplacing Make in “your” project is relatively easy. Replacing it in all your dependencies is both a technical pain in the ass and a massive political problem if you want to upstream the change. If you have a lot of dependencies, it’s probably easier just to improve Make. ;)
- temac 7y agoI would not call avoiding a classic thundering herd problem a bad choice, especially when the top maintainer of Linux does it himself in Linux: which is arguably more probable than him rewriting GNU Make (well we all know he could do it, but this would still be less probable...) Plus it will not only optimize GNU Make workloads, but potentially other programs that do that. And to be honest, playing with pipe for signaling (and more rarely token counting) is STILL a must-have way to do IPC portably, esp. if you want to integrate with polling on fds. Various OSes have also various new things, but they are non-portable -- maybe someone should try to get eventfd&friends in POSIX? Anyway, if that's a "good example of how we develop software these days", I would say we managed to develop software the right way at last, making basic stuff work well before succumbing to the tentation to rewrite everything all the time just to play with new shinny techs, getting fresh bugs in the process. Also you know what has evolved in the last few years, motivating this perf fix? Availability of HIGHLY parallel computers for not too expensive. Yes, I continue to find it highly refreshing that work is done to get all the boring historical software work better in that context. Maybe you think we can afford rewriting all the things all the time to follow supposedly radical technological changes (and using less well designed OSes?), but if so then this thought is not shared widely or at least not followed with action. Rightly so because economically the outcome would be doubtful: just look at the patch, it is way smaller than an hypothetical shiny new rewrite of GNU Make...
- kzrdude 7y agoCould the fix have any potential negative effects on other workloads?
- the8472 7y agoSome other build systems use the pipe token approach too, rust's cargo for example.
- ridiculous_fish 7y agoDoes this affect select() or only read()? Today if N procs select() on a single pipe, all will be woken up if the pipe becomes readable. Will this now mean that only one will be awoken?