5 ms·

The GNU make jobserver Implementation (2015)

83 points by fanf2 2y ago

chasil 2y ago
Does the xargs -P option also work this way?
- nwellnhof 2y ago
  xargs doesn't recursively invoke itself, so you don't have the problem that make has.
  - Polizeiposaune 2y ago
    The xargs command can invoke arbitrary commands, possibly including scripts that themselves invoke xargs, so a system using xargs -P recursively could very well benefit from a governor limiting the total number of jobs run in parallel. It might be unlikely or uncommon (I've certainly never seen it in practice) but nothing prevents it from happening. And if you're using xargs somewhere in the middle of a parallel build, having xargs -P respect the jobserver's limits would be a nice touch.
kazinator 2y ago
> convincing myself that reads and writes of one-byte tokens to a pipe were atomic and robust :)
- crest 2y ago
  Transfers up to PIPE_BUF are atomic on pipes according to POSIX.
badmintonbaseba 2y ago
It's interesting that waiting on a signal and a pipe read at the same time is the hard part. Would be interesting to track down how this happens to be implemented if you do this from a higher level async framework, like python's asyncio.
- awesomerob 2y ago
  I'd guess it would be similar to (if not the same as) the internal pipe + select() approach mentioned in the post (although most likely with epoll or whatever instead of select).
- nerdponx 2y ago
  As far as I know, none of the tricky Unix details here would be handled by Python. The "async" part isn't the problem, it's the coordination around syscalls.
  - badmintonbaseba 2y ago
    I wonder how this fails then: import asyncio import os import signal import sys async def get_stdin_reader(): loop = asyncio.get_running_loop() reader = asyncio.StreamReader() protocol = asyncio.StreamReaderProtocol(reader) await loop.connect_read_pipe(lambda: protocol, sys.stdin) return reader async def amain(): print(f'pid: {os.getpid()}') sig_queue: asyncio.Queue[None] = asyncio.Queue() def sig_handler() -> None: sig_queue.put_nowait(None) loop= asyncio.get_running_loop() loop.add_signal_handler(signal.SIGUSR1, sig_handler) reader = await get_stdin_reader() task1 = loop.create_task(reader.read(1)) task2 = loop.create_task(sig_queue.get()) pending = {task1, task2} while True: done, pending = await asyncio.wait(pending, return_when=asyncio.FIRST_COMPLETED) if task1 in done: task1 = loop.create_task(reader.read(1)) pending.add(task1) print('got char on stdin') if task2 in done: task2 = loop.create_task(sig_queue.get()) pending.add(task2) print('got signal') asyncio.run(amain())
- aumerle 2y ago
  It's not even slightly tricky, just use self pipe. I have no idea why the maintainer of make rejected it. He says select has different signatures, but select is in POSIX, so unless he is porting to a non POSIX platform, it's irrelevant and even if he is, I doubt it is that hard to write a wrapper to abstract the non POSIX compatible implementation of select. Then he complains about needing to do CLOEXEC on the self pipe. This is trivial (one line) on Linux using pipe2 and about 5 lines of code on other platforms. Given that he says make does not use threads its also perfectly robust without pipe2. Opting for a harder algorithm just to avoid a few lines of compatibility shims seems like very much the wrong tradeoff.
redleader55 2y ago
Make is awesome, but it's such a hack ... Writing and designing a build for a complicated project is such a pain because of the recursive calling.
- db48x 2y ago
  Anything you want to build can be done without using recursive invocations of make. You should probably go ahead and recursively run make when building a dependency that is built with make, but almost any other use is a mistake.
- ajross 2y ago
  Recursive make invocation isn't a required thing or a design point, it's not really even recommended. You see it commonly only in the "obvious" situation where you're building a dependency that has its own build integration. And there, make works better than competing tools for precisely the reason that it inherits the jobserver environment and can thus make better use of the build hardware. I continue to believe that make is and remains the superior technology. Everyone else does like one or two things better (usually just limited to the developer-facing configuration language) and everything else worse.
- 1718627440 2y ago
  Recursive make is known to be problematic since 1997 (https://petermiller.work/pmiller/books/rmch/ https://petermiller.work/pmiller/books/rmch/).
pornel 2y ago
Rust/Cargo has adopted this protocol: https://lib.rs/jobserver https://lib.rs/jobserver
- ajross 2y ago
  Frustrating that almost nothing else has. In particular Ninja has been dragging its feet on this for most of a decade (they have an epic GitHub feature request and like six submitted attempts, none merged). All the "build system" components in the ecosystem want to be the Root of all Execution and don't feel like they should have to play in a sandbox with their competitors. Really, the older I get and the more I see of the cmake/ninja/Bazel/whatever world... GNU Make was a better tool even 20 years ago.
  - rout39574 2y ago
    Everyone wants to write the build tool in their favorite language, and then stops when the work gets hard. :)
  - danudey 2y ago
    There's a lot to hate about make, but looking for good alternatives there are very, very few out there and none of them provided anything extra I was looking for. The one thing I wish were possible (trivially) in make, and I understand why it's not, is the ability to define a dependency not on whether a file exists or not but on the result of executing a process. For example, do I need to rebuild this docker container? The current method to determine that is to create a stamp file; create the docker container then `touch .docker-image-created.stamp`. Unfortunately, those stamps can become out of date if the docker container is removed (or old, etc.) so it leads to confusing situations where make's interpretation of the current state is separate from the reality, and in large complex projects where there are lots of interconnected parts, sometimes your only feasible route is to remove everything and completely rebuild from scratch. There's also an inexplicable lack of debugging output insofar as printing what target make is currently running. It has a ton of debug output, but no option I can find that says "I am running this target right now", "okay I'm done this target now" without manually adding $(info ...) statements to your makefile.
deleted 2y ago
[deleted]
stabbles 2y ago
This is somewhat outdated. GNU make now uses named pipes by default on platforms that support it. https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html https://www.gnu.org/software/make/manual/html_node/POSIX-Job... It's a bit more robust. For example when you have a Makefile like this: target: +python3 script.py $@ where `script.py` itself spawns a sub-process that is jobserver aware (like make, gcc, cargo, ...), you don't have to worry about whether the fds of the python process, needed for communicating with the jobserver, are inherited in the subprocess. Previously you had to run `subprocess.Popen(..., close_fds=False)` and make sure MAKEFLAGS was inherited. Now you only have to ensure MAKEFLAGS is propagated, and that the sub-process can access the fifo by path (which is sometimes problematic, for example when running docker).
- xyzzy_plugh 2y ago
  Frustratingly there is still value in the old style jobserver because old versions of GNU make are prevalent in the wild, i.e. on macOS. I'm at the point where if you're going to require users to install something, rather than a newer version of make, I'd rather require a more modern build tool and disregard make entirely.
- o11c 2y ago
  Any program that closes FDs it doesn't know the origin of is fundamentally broken, and should have bugs filed against it. If "close all FDs" is a workaround for other bugs, fix those bugs too.
  - int_19h 2y ago
    This particular argument is about closing inherited FDs in a forked child process. This is a perfectly sensible thing to do if the child doesn't need them, and most child processes do not need FDs for various random files opened by the parent.
    - o11c 2y ago
      "Random files opened by the parent" were, as a rule, deliberately given to the child. You would use CLOEXEC if you don't want that. Please don't let your children trash something you gave them just because they don't appreciate it yet.
abbbi 2y ago
so that comes quite right. Im currently working on a codebase that has evolved over 30-ish years. The makefiles are quite a mess and since ages, the software has been built sequentially which results in build times beeing much longer than required. Of course the project consists of multiple module which you should be able to build seperately, thus recursive calls are quite common amongst the makefiles. First test with only a hand full of jobs (-j) already failed in the beginning, i could fix these quite fast (missing makefile targets). Now i have the situation that on some build systems (with faster CPU) i still see races, where the same makfile target for a subproject runs at the same time and overwrites each others target files. On other build systems it works without any issue. However, ive still failed to reproduce the failure manually, it usually happens during automatic build invoked by jenkins or gitlab. Is there a way to make "make" simulate those builds so one could tell where the cause for the races is in detail?
- oso2k 2y ago
  Is there a way to make "make" simulate those builds so one could tell where the cause for the races is in detail? Have you tried? make —-dry-run

83 points by fanf2 2y ago

chasil 2y ago

Does the xargs -P option also work this way?

nwellnhof 2y ago
xargs doesn't recursively invoke itself, so you don't have the problem that make has.
- Polizeiposaune 2y ago
  The xargs command can invoke arbitrary commands, possibly including scripts that themselves invoke xargs, so a system using xargs -P recursively could very well benefit from a governor limiting the total number of jobs run in parallel. It might be unlikely or uncommon (I've certainly never seen it in practice) but nothing prevents it from happening. And if you're using xargs somewhere in the middle of a parallel build, having xargs -P respect the jobserver's limits would be a nice touch.

kazinator 2y ago

> convincing myself that reads and writes of one-byte tokens to a pipe were atomic and robust :)

crest 2y ago
Transfers up to PIPE_BUF are atomic on pipes according to POSIX.

badmintonbaseba 2y ago

It's interesting that waiting on a signal and a pipe read at the same time is the hard part. Would be interesting to track down how this happens to be implemented if you do this from a higher level async framework, like python's asyncio.

awesomerob 2y ago
I'd guess it would be similar to (if not the same as) the internal pipe + select() approach mentioned in the post (although most likely with epoll or whatever instead of select).
nerdponx 2y ago
As far as I know, none of the tricky Unix details here would be handled by Python. The "async" part isn't the problem, it's the coordination around syscalls.
- badmintonbaseba 2y ago
  I wonder how this fails then: import asyncio import os import signal import sys async def get_stdin_reader(): loop = asyncio.get_running_loop() reader = asyncio.StreamReader() protocol = asyncio.StreamReaderProtocol(reader) await loop.connect_read_pipe(lambda: protocol, sys.stdin) return reader async def amain(): print(f'pid: {os.getpid()}') sig_queue: asyncio.Queue[None] = asyncio.Queue() def sig_handler() -> None: sig_queue.put_nowait(None) loop= asyncio.get_running_loop() loop.add_signal_handler(signal.SIGUSR1, sig_handler) reader = await get_stdin_reader() task1 = loop.create_task(reader.read(1)) task2 = loop.create_task(sig_queue.get()) pending = {task1, task2} while True: done, pending = await asyncio.wait(pending, return_when=asyncio.FIRST_COMPLETED) if task1 in done: task1 = loop.create_task(reader.read(1)) pending.add(task1) print('got char on stdin') if task2 in done: task2 = loop.create_task(sig_queue.get()) pending.add(task2) print('got signal') asyncio.run(amain())
aumerle 2y ago
It's not even slightly tricky, just use self pipe. I have no idea why the maintainer of make rejected it. He says select has different signatures, but select is in POSIX, so unless he is porting to a non POSIX platform, it's irrelevant and even if he is, I doubt it is that hard to write a wrapper to abstract the non POSIX compatible implementation of select. Then he complains about needing to do CLOEXEC on the self pipe. This is trivial (one line) on Linux using pipe2 and about 5 lines of code on other platforms. Given that he says make does not use threads its also perfectly robust without pipe2. Opting for a harder algorithm just to avoid a few lines of compatibility shims seems like very much the wrong tradeoff.

redleader55 2y ago

Make is awesome, but it's such a hack ... Writing and designing a build for a complicated project is such a pain because of the recursive calling.

db48x 2y ago
Anything you want to build can be done without using recursive invocations of make. You should probably go ahead and recursively run make when building a dependency that is built with make, but almost any other use is a mistake.
ajross 2y ago
Recursive make invocation isn't a required thing or a design point, it's not really even recommended. You see it commonly only in the "obvious" situation where you're building a dependency that has its own build integration. And there, make works better than competing tools for precisely the reason that it inherits the jobserver environment and can thus make better use of the build hardware. I continue to believe that make is and remains the superior technology. Everyone else does like one or two things better (usually just limited to the developer-facing configuration language) and everything else worse.
1718627440 2y ago
Recursive make is known to be problematic since 1997 (https://petermiller.work/pmiller/books/rmch/ https://petermiller.work/pmiller/books/rmch/).

pornel 2y ago

Rust/Cargo has adopted this protocol: https://lib.rs/jobserver https://lib.rs/jobserver

ajross 2y ago
Frustrating that almost nothing else has. In particular Ninja has been dragging its feet on this for most of a decade (they have an epic GitHub feature request and like six submitted attempts, none merged). All the "build system" components in the ecosystem want to be the Root of all Execution and don't feel like they should have to play in a sandbox with their competitors. Really, the older I get and the more I see of the cmake/ninja/Bazel/whatever world... GNU Make was a better tool even 20 years ago.
- rout39574 2y ago
  Everyone wants to write the build tool in their favorite language, and then stops when the work gets hard. :)
- danudey 2y ago
  There's a lot to hate about make, but looking for good alternatives there are very, very few out there and none of them provided anything extra I was looking for. The one thing I wish were possible (trivially) in make, and I understand why it's not, is the ability to define a dependency not on whether a file exists or not but on the result of executing a process. For example, do I need to rebuild this docker container? The current method to determine that is to create a stamp file; create the docker container then `touch .docker-image-created.stamp`. Unfortunately, those stamps can become out of date if the docker container is removed (or old, etc.) so it leads to confusing situations where make's interpretation of the current state is separate from the reality, and in large complex projects where there are lots of interconnected parts, sometimes your only feasible route is to remove everything and completely rebuild from scratch. There's also an inexplicable lack of debugging output insofar as printing what target make is currently running. It has a ton of debug output, but no option I can find that says "I am running this target right now", "okay I'm done this target now" without manually adding $(info ...) statements to your makefile.

deleted 2y ago

[deleted]

stabbles 2y ago

This is somewhat outdated. GNU make now uses named pipes by default on platforms that support it. https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html https://www.gnu.org/software/make/manual/html_node/POSIX-Job... It's a bit more robust. For example when you have a Makefile like this: target: +python3 script.py $@ where `script.py` itself spawns a sub-process that is jobserver aware (like make, gcc, cargo, ...), you don't have to worry about whether the fds of the python process, needed for communicating with the jobserver, are inherited in the subprocess. Previously you had to run `subprocess.Popen(..., close_fds=False)` and make sure MAKEFLAGS was inherited. Now you only have to ensure MAKEFLAGS is propagated, and that the sub-process can access the fifo by path (which is sometimes problematic, for example when running docker).

xyzzy_plugh 2y ago
Frustratingly there is still value in the old style jobserver because old versions of GNU make are prevalent in the wild, i.e. on macOS. I'm at the point where if you're going to require users to install something, rather than a newer version of make, I'd rather require a more modern build tool and disregard make entirely.
o11c 2y ago
Any program that closes FDs it doesn't know the origin of is fundamentally broken, and should have bugs filed against it. If "close all FDs" is a workaround for other bugs, fix those bugs too.
- int_19h 2y ago
  This particular argument is about closing inherited FDs in a forked child process. This is a perfectly sensible thing to do if the child doesn't need them, and most child processes do not need FDs for various random files opened by the parent.
  - o11c 2y ago
    "Random files opened by the parent" were, as a rule, deliberately given to the child. You would use CLOEXEC if you don't want that. Please don't let your children trash something you gave them just because they don't appreciate it yet.

abbbi 2y ago

so that comes quite right. Im currently working on a codebase that has evolved over 30-ish years. The makefiles are quite a mess and since ages, the software has been built sequentially which results in build times beeing much longer than required. Of course the project consists of multiple module which you should be able to build seperately, thus recursive calls are quite common amongst the makefiles. First test with only a hand full of jobs (-j) already failed in the beginning, i could fix these quite fast (missing makefile targets). Now i have the situation that on some build systems (with faster CPU) i still see races, where the same makfile target for a subproject runs at the same time and overwrites each others target files. On other build systems it works without any issue. However, ive still failed to reproduce the failure manually, it usually happens during automatic build invoked by jenkins or gitlab. Is there a way to make "make" simulate those builds so one could tell where the cause for the races is in detail?

oso2k 2y ago
Is there a way to make "make" simulate those builds so one could tell where the cause for the races is in detail? Have you tried? make —-dry-run