11 ms·
GNU make insanity: finding the value of the -j parameter
- chaosfactor 12y agoThis probably won't work if you just pass -j. If I needed to solve this problem, I would have just grepped the output of `ps' to get the make invocation. If I wasn't sure which invocation was correct, I would have used heuristics like the output of `pstree' to figure it out.
- Someone 12y agoI would think a process launched by make can easily obtain the process ID of its parent process. If so, no heuristics are needed to find the correct instance of make. On the other hand, reading the command line isn't sufficient. The -j argument does not require a job count, defaulting to "as many as possible". Also, you would have to check your OS, too since the -j argument is ignored on MS-DOS (https://www.gnu.org/software/make/manual/make.html#Options-Summary https://www.gnu.org/software/make/manual/make.html#Options-S...)
- raverbashing 12y agoIf you change the maximum value (I put 1024) on line 10 it prints: 319 on Mac OS X 10.8 337 on Centos 6
- foxhill 12y agoi think the true insanity lies within the mind of the developer whose build system depends on the number of jobs being ran in parallel..! interesting academic exercise though. unless i'm mistaken, this will only detect up to 32 build jobs? i'd hate to see the Makefile that works for arbitrary values..
- chaosfactor 12y agoIt makes some sense to want to know this. You might know that your build process doesn't benefit from more than 4 jobs, and want to alert this to the developer who tries to use more.
- moonbug 12y agoNot all problems need a solution in code.
- nsajko 12y agoAwesome! This goes in my fortune file :D
- stormbrew 12y agoI really have to echo the person who replied to you on a different thread and ask.. why? What do you think you're saving the user from exactly?
- xorcist 12y agoPlease avoid such things. Useless warnings just create noise that mask real ones. Just this one may not matter, but the same rationality would apply to hundreds more.
- jordigh 12y agoI was the one who asked the original question. My purpose was so that Mercurial's test suite could specify the amount of parallelism from the Makefile. The "test" rule in the Makefile invokes "run-tests.py". This latter script itself accepts a -j option. It's possible to invoke it directly, but this misses the second part of the "test" rule in the Makefile, which executes some tests that run-tests.py can't do.
- pavpanchekha 12y agoSay one of your build rules generates a data file by performing some expensive computations. The parallelism is internal to the code (say, it's a Monte-Carlo simulation), so it should run on a lot of cores, but if you're doing it multiple times (with, say, different parameters) in parallel, you don't want the total number of threads spawned to be more than the machine has.
- foxhill 12y agotrue, however my only complaint - > you don't want the total number of threads spawned to be more than the machine has that may not be necessarily true, and regardless, it might be a sane decision for yourself, but it might not be for others.
- rlpb 12y agoWhy would you want to do it multiple times with different parameters in parallel? If your code can already scale to multiple cores, then just have each parameter set run one at a time, and you're already maximizing your CPU use. Or integrate with make's jobserver as someone else has described.
- raindrop777 12y agoReading this only pops one question: What is the best replacement of make in 2015? :)
- Puts 12y agoI hear Ruby in a Docker container running on RaspberryPI actually can replace a lot of the functionality of make...
- vezzy-fnord 12y agotup [1], redo [2] and ninja [3] are probably the build systems with the most potential. redo is the simplest, which is unsurprising given that it was originally proposed by djb. Alternatively, you could look into mk, which is the make replacement used by Plan 9, Inferno and derivatives. [1] http://gittup.org/tup/ http://gittup.org/tup/ [2] http://homepage.ntlworld.com/jonathan.deboynepollard/FGA/introduction-to-redo.html http://homepage.ntlworld.com/jonathan.deboynepollard/FGA/int... [3] https://martine.github.io/ninja/ https://martine.github.io/ninja/
- raindrop777 12y agoAre any of these cross-platform like make?
- joeyh 12y agoThe real insanity with make is that it has no interface to check whether a Makefile contains a given target. Best you can do is to run make -n target to probe, but it's possible to write Makefiles that run code even though -n is used, which will defeat such probes at distribution scale. It quickly becomes an exercise in heuristics and output parsing.
- chaosfactor 12y agoWrite a target that greps the Makefile itself and extracts the targets.
- caust1c 12y agoThen write a parser to regenerate all the targets! It's a good thing make is turing complete. :-)
- TheDong 12y agoAs shown in the post, targets are not always grep-able. The post's makefile contained the target "par-30" even though that string appears nowhere on the page or in the makefile. You need to write a full parser to discover all the targets.
- Hello71 12y ago$ make -pqrf <(curl https://gist.githubusercontent.com/jgrahamc/2cc6df7fd5cd61c4d93f/raw/parallel) | grep '^[^ ]*:' | cut -d : -f 1 par-% par-22 par-30 par-5 par-27 par-31 all FORCE /proc/self/fd/13 par-13 par-9 par-23 par-19 .SUFFIXES par-15 par-7 parallel par-29 par-28 .parallel par-4 par-25 par-6 par-14 .DEFAULT par-24 par-0 par par-1 par-10 par-16 par-2 par-11 par-17 par-20 par-8 par-21 par-12 par-26 par-18 par-3
- TheDong 12y agoRight, you used 'make' as the parser. He said "grep the makefile". You're not grepping the makefile. Nice solution though!
- JoshTriplett 12y agoIt's an amusing exercise, but for actual practical use, the right solution is to share make's jobserver. If you have a build rule that wants to run multiple things in parallel, put + in front of the command, and then your command will receive the file descriptors for the jobserver pipe on the command line. Then, when you want to start a parallel job, do a blocking read of a byte from the read fd, and when you're done with that job, write a byte to the write fd.
- chaosfactor 12y agoThis does not generalize to arbitrary purposes. If you just want to detect that the user has run more jobs than your build system can possibly benefit from and print a warning, there is no reason to start messing with the jobserver. This is not the Ockham solution :)
- JoshTriplett 12y agoWhy print such a warning? For most projects, make -j65536 will be effectively equivalent to make -j, both of which are perfectly fine. Suppose you're on a system with more CPUs than you have build rules in your build system; you shouldn't print a warning just because someone does make -j$(nproc).
- oso2k 12y agoI get what you're saying but not seeing how to do that in make syntax. Mind giving an example?
- JoshTriplett 12y ago$ cat Makefile test: +env | grep MAKEFLAGS $ make -j4 test env | grep MAKEFLAGS MAKEFLAGS= -j --jobserver-fds=3,4 If you want to act like a sub-make, then parse MAKEFLAGS from your environment, and if you see -j and --jobserver-fds=R,W , then parse R and W as file descriptor numbers, block waiting for a byte from R before starting an additional parallel job, and write a byte to W when done with a parallel job.
- moonbug 12y agoCarp all you want about make, I'll take it over any of its ostensible "replacements".
- Negitivefrags 12y agoI don't think anyone has a problem with make. The thing you need to replace is autotools.
- davexunit 12y agoAs crufty as autotools is, nothing comes close to being a more robust build system. It does everything. Learning to use the autotools is difficult, but what's most frustrating is trying to build software that uses clever Makefiles with a bunch of hardcoded assumptions about the platform you are building on that you have to hack to get to work.
- saboot 12y ago"As crufty as autotools is, nothing comes close to being a more robust build system." .. CMake? http://www.cmake.org/ http://www.cmake.org/
- michael_h 12y agoI wish this were true. Getting cmake to do something off the wall like building static executable binaries is...frustrating. I had high hopes for scons, but that turned into a quagmire. Any build system that's going to unseat autotools will need to obviate the need for almost all custom scripts. AKA, specify the 'what', and let the user apply a 'recipe' for the 'how'. Or something...if I knew what it was supposed to look like I'd build it.
- MereInterest 12y agoOut of curiosity, what are the quagmire-ish aspects of scons? I've been using it for a while, and it hasn't seemed too bad, which probably means that I haven't run into them yet.
- Quanticles 12y agoOkay, I'll bite. This is my attempt at it: ---------------------- PID := $(shell cat /proc/$$$$/status | grep PPid | awk '{print $$2}') JOBS := $(shell ps -p ${PID} -f | tail -n1 | grep -oP '\-j *\d+' | sed 's/-j//') ifeq "${JOBS}" "" JOBS := 1 endif all: echo ${JOBS} ---------------------- quanticles@glados $ make -j8 8 quanticles@glados $ make 1 ---------- edit: for silly mistake
- anon4 12y agoNote: you can forego the ps and simply read /proc/$PID/cmdline, which is a \0-delimited list of strings. So, grep -oP '\-j\x00\d+' /proc/$PID/cmdline Edit: and no need for sed when a simple cut -b4- does the same job.
- protomyth 12y agoDoes that work on anything other than Linux?
- myhf 12y agoNot in exactly the same way, but every unixy system has some way of finding the parent pid and looking up its command line.
- protomyth 12y agoIs Quanticles's code portable to the other unix systems?
- myhf 12y agoIt should work on most systems that have a /proc filesystem: http://en.wikipedia.org/wiki/Procfs http://en.wikipedia.org/wiki/Procfs If you don't have /proc but do have ps (1), you could get the parent pid like this: PID := $(ps -p $$$$ -o ppid=)
- Someone 12y ago
- msie 12y agoI need more information as to why it's required to get the value of that parameter. Can't it be simultaneously be passed in as an environment variable?
- rcthompson 12y agoThere is no reason your makefile should ever depend on the parallelism level, with the possible exception of requiring no parallelism at all (i.e. "-j1"). That's why make doesn't bother to expose it. This little hack is just for fun.
- jordigh 12y agoI was the one who asked the original question. My purpose was so that Mercurial's test suite could specify the amount of parallelism from the Makefile. The "test" rule in the Makefile invokes "run-tests.py". This latter script itself accepts a -j option. It's possible to invoke it directly, but this misses the second part of the "test" rule in the Makefile, which executes some tests that run-tests.py can't do.
- jwatte 12y agoThose who don't learn make are doomed to reimplement it. Poorly. Also: never, ever, build a make system that relies on recursive make invocations. Automake is a particularly egregious violator is this rule (and other sanity rules) and has probably hurt make more than anything else.
- santaclaus 12y agoSomeone should implement make without the silly tabs.
- babuskov 12y agoAnd I thought I was the only one burned by this ;)
- burstmode 12y agoAt least every 2nd C programmer working on a non-unix platform has been burned by this at least once. It's like a 10ft deep pothole on Broadway that won't get fixed because traditionalists see it as a 'national monument'.
- kryptiskt 12y ago"Why the tab in column 1? Yacc was new, Lex was brand new. I hadn't tried either, so I figured this would be a good excuse to learn. After getting myself snarled up with my first stab at Lex, I just did something simple with the pattern newline-tab. It worked, it stayed. And then a few weeks later I had a user population of about a dozen, most of them friends, and I didn't want to screw up my embedded base. The rest, sadly, is history." -- Stuart Feldman
- JoshTriplett 12y agoIf you don't care about compatibility with existing Makefiles, you can do this already with GNU make; just set ".RECIPEPREFIX" to some other character.
- pdkl95 12y ago
- sillysaurus3 12y agoWhy do you need to know the value of the -j parameter in a makefile?
- saboot 12y agoIt will reduce total compile times for large projects by running make in parallel. Files dont compile faster, but make will act (compile) on N files at the same time together.
- babuskov 12y agoI didn't downvote you, but I assume why it happened: What you wrote is correct, but does not answer the question.
- jordigh 12y agoI was the one who asked the original question. My purpose was so that Mercurial's test suite could specify the amount of parallelism from the Makefile. The "test" rule in the Makefile invokes "run-tests.py". This latter script itself accepts a -j option. It's possible to invoke it directly, but this misses the second part of the "test" rule in the Makefile, which executes some tests that run-tests.py can't do.
- tobik 12y agoFor FreeBSD users: FreeBSD's make provides the value of -j through ${.MAKE.JOBS}
- mempko 12y agoGNU make is free software. If you are a capable programmer, you can make a change and send a patch.
- snoopybbt 12y agoI don't know how many hours the author has spent/wasted on this topic (and how many people are now wasting their time coming up with other solutions that in the end do NOT fix the real problem), but... Quite frankly, I find this whole "look i did a horrible hack and let's see who can make the best worst horrible hack" thing quite stupid and silly. GNU Make is free software released under a free license, my opinion is that instead of doing that crazy thing, the author could have just written a patch for GNU Make in order to make it export a "JOBS" environment variable to all its child processes. Oh but yes, "I felt this feature was missing so I added it" is way way way less cool than "geez the gnu make folks are insane lollerplex they have no way to know how many jobs they're running". ALSO: http://www.catb.org/esr/faqs/hacker-howto.html#believe2 http://www.catb.org/esr/faqs/hacker-howto.html#believe2 Here it is a patched version of GNU Make, providing a #J variable that holds the number of jobs as passed via -j/--jobs: http://santoro.tk/~manu/gnumake.png As can be seen in the screenshot, it can be passed as environment variable to programs called by make Source: https://github.com/esantoro/make https://github.com/esantoro/make
- snoopybbt 12y agoSource from sample Makefile: default: qualcosa.sh JOBS=${#J} ./qualcosa.sh Source for sample shell-script: #!/bin/bash echo "According to \$JOBS environment variable, the argument of -j/--jobs is $JOBS"
- DangerousPie 12y agoIgnoring the fact that I doubt that the original post was serious: Of course you can patch the software and in the long term that will of course be the best solution. But it can take a long time until that patch actually makes it into the main repo, and even longer until it will be in the default `make` on all major distributions. So if you want to use this feature right now then writing a patch is useless for you, unless you only ever use it on your own machine. The big advantage of the workaround is that it actually works with current versions of make.
- snoopybbt 12y agoOh, I see your point. Anyway, it was my small contribution to the discussion :-)
- cafebeen 12y agoOn a semi-related note... do any of the make gurus out there know if there's anything similar to "-j" that instead sends parallel jobs off to a cluster, e.g. Sun Grid?
- adrusi 12y agoThat wouldn't be appropriate as a specialized component of make. You could write a script that sends a command off to a cluster and just set CC=script/send_to_cluster gcc Then specify the number of jobs you'd like to run simultaneously on your cluster with the "-j" flag.
- cafebeen 12y agoThanks for the thought. That sounds fine for simple compiler makefiles, although I'm using it for general purpose work (data analysis and visualization). I'm hoping for something that wouldn't require each command to be wrapped up, as there are many.