3 ms·

It's interesting that waiting on a signal and a pipe read at the same time is the hard part. Would be interesting to track down how this happens to be implement

It's interesting that waiting on a signal and a pipe read at the same time is the hard part. Would be interesting to track down how this happens to be implemented if you do this from a higher level async framework, like python's asyncio.

awesomerob 2y ago
I'd guess it would be similar to (if not the same as) the internal pipe + select() approach mentioned in the post (although most likely with epoll or whatever instead of select).
nerdponx 2y ago
As far as I know, none of the tricky Unix details here would be handled by Python. The "async" part isn't the problem, it's the coordination around syscalls.
- badmintonbaseba 2y ago
  I wonder how this fails then: import asyncio import os import signal import sys async def get_stdin_reader(): loop = asyncio.get_running_loop() reader = asyncio.StreamReader() protocol = asyncio.StreamReaderProtocol(reader) await loop.connect_read_pipe(lambda: protocol, sys.stdin) return reader async def amain(): print(f'pid: {os.getpid()}') sig_queue: asyncio.Queue[None] = asyncio.Queue() def sig_handler() -> None: sig_queue.put_nowait(None) loop= asyncio.get_running_loop() loop.add_signal_handler(signal.SIGUSR1, sig_handler) reader = await get_stdin_reader() task1 = loop.create_task(reader.read(1)) task2 = loop.create_task(sig_queue.get()) pending = {task1, task2} while True: done, pending = await asyncio.wait(pending, return_when=asyncio.FIRST_COMPLETED) if task1 in done: task1 = loop.create_task(reader.read(1)) pending.add(task1) print('got char on stdin') if task2 in done: task2 = loop.create_task(sig_queue.get()) pending.add(task2) print('got signal') asyncio.run(amain())
aumerle 2y ago
It's not even slightly tricky, just use self pipe. I have no idea why the maintainer of make rejected it. He says select has different signatures, but select is in POSIX, so unless he is porting to a non POSIX platform, it's irrelevant and even if he is, I doubt it is that hard to write a wrapper to abstract the non POSIX compatible implementation of select. Then he complains about needing to do CLOEXEC on the self pipe. This is trivial (one line) on Linux using pipe2 and about 5 lines of code on other platforms. Given that he says make does not use threads its also perfectly robust without pipe2. Opting for a harder algorithm just to avoid a few lines of compatibility shims seems like very much the wrong tradeoff.
- gpderetta 2y ago
  > so unless he is porting to a non POSIX platform Very likely gmake works on non-posix platforms. Whether there is a requirement that the jobserver also works there I don't know. It also needs to work on non-linux platforms. So a portable solution (or multiple non-portable ones) is needed.
  - aumerle 2y ago
    Yes, but this is two well known and understood functions we are talking about, wrapping them is really not that hard.
- badmintonbaseba 2y ago
  I'm not seeing python creating a self-pipe for `loop.add_signal_handler`. edit: Oh, it uses `signal.set_wakeup_fd`, interesting. https://docs.python.org/3/library/signal.html#signal.set_wakeup_fd https://docs.python.org/3/library/signal.html#signal.set_wak... edit2: it looks like the fd comes from a self-socket, but yeah, it's the same approach. The function is even called `_make_self_pipe`. https://github.com/python/cpython/blob/bf21e2160d1dc6869fb230b90a23ab030835395b/Lib/asyncio/selector_events.py#L118-L124 https://github.com/python/cpython/blob/bf21e2160d1dc6869fb23...
zokier 2y ago
On Linux you got signalfd and on BSDs you got kqueue. It is only difficult if you are avoiding the tools made specifically to address the problem.
- crest 2y ago
  On FreeBSD you also have process descriptors (Linux followed suite a while ago) which provide a race free clean interface to supervise processes even if you're not their reaper (so can't use PIDs reliably). You can add them to kqueue using the EVFILT_PROCDESC filter with the NOTE_EXIT flag to get notified when the referenced process exits. The exit status you would normally get from waitpid() is already in the struct kevent. These OS specific APIs are sometimes required and may make certain usecases a lot easier to implement correctly (or even at all), but a job server for a build system isn't one of those. The make jobs can be required to behave and stay in the process group the job server creates for them just like shell job control. Which can be done with just POSIX API. You just have to blow the dust from the relevant tombs the ancients left us (e.g. Advanced Programming in the UNIX Environment). Refuse the temptation and don't make infrastructure tools like gmake depend OS specific APIs. Doing so would make the world a worse place for everyone else.
  - o11c 2y ago
    PIDs are always reliable for your own direct children.
  - danudey 2y ago
    > don't make infrastructure tools like gmake depend OS specific APIs Even if you're going to implement OS-specific APIs you still need a general case for OSes that don't support those APIs or don't support them correctly. That means you still need to solve the original problem regardless, which then means that it doesn't actually save you time and energy to use those APIs but rather creates more development and maintenance work and not less. If those APIs don't provide you a significant benefit (reliability, performance, etc.) then it's likely not worth implementing them at all if the general case works, and if it doesn't work you have to fix it anyway.
  - 2y ago

awesomerob 2y ago

I'd guess it would be similar to (if not the same as) the internal pipe + select() approach mentioned in the post (although most likely with epoll or whatever instead of select).

nerdponx 2y ago

As far as I know, none of the tricky Unix details here would be handled by Python. The "async" part isn't the problem, it's the coordination around syscalls.

badmintonbaseba 2y ago
I wonder how this fails then: import asyncio import os import signal import sys async def get_stdin_reader(): loop = asyncio.get_running_loop() reader = asyncio.StreamReader() protocol = asyncio.StreamReaderProtocol(reader) await loop.connect_read_pipe(lambda: protocol, sys.stdin) return reader async def amain(): print(f'pid: {os.getpid()}') sig_queue: asyncio.Queue[None] = asyncio.Queue() def sig_handler() -> None: sig_queue.put_nowait(None) loop= asyncio.get_running_loop() loop.add_signal_handler(signal.SIGUSR1, sig_handler) reader = await get_stdin_reader() task1 = loop.create_task(reader.read(1)) task2 = loop.create_task(sig_queue.get()) pending = {task1, task2} while True: done, pending = await asyncio.wait(pending, return_when=asyncio.FIRST_COMPLETED) if task1 in done: task1 = loop.create_task(reader.read(1)) pending.add(task1) print('got char on stdin') if task2 in done: task2 = loop.create_task(sig_queue.get()) pending.add(task2) print('got signal') asyncio.run(amain())

aumerle 2y ago

It's not even slightly tricky, just use self pipe. I have no idea why the maintainer of make rejected it. He says select has different signatures, but select is in POSIX, so unless he is porting to a non POSIX platform, it's irrelevant and even if he is, I doubt it is that hard to write a wrapper to abstract the non POSIX compatible implementation of select. Then he complains about needing to do CLOEXEC on the self pipe. This is trivial (one line) on Linux using pipe2 and about 5 lines of code on other platforms. Given that he says make does not use threads its also perfectly robust without pipe2. Opting for a harder algorithm just to avoid a few lines of compatibility shims seems like very much the wrong tradeoff.

gpderetta 2y ago
> so unless he is porting to a non POSIX platform Very likely gmake works on non-posix platforms. Whether there is a requirement that the jobserver also works there I don't know. It also needs to work on non-linux platforms. So a portable solution (or multiple non-portable ones) is needed.
- aumerle 2y ago
  Yes, but this is two well known and understood functions we are talking about, wrapping them is really not that hard.
badmintonbaseba 2y ago
I'm not seeing python creating a self-pipe for `loop.add_signal_handler`. edit: Oh, it uses `signal.set_wakeup_fd`, interesting. https://docs.python.org/3/library/signal.html#signal.set_wakeup_fd https://docs.python.org/3/library/signal.html#signal.set_wak... edit2: it looks like the fd comes from a self-socket, but yeah, it's the same approach. The function is even called `_make_self_pipe`. https://github.com/python/cpython/blob/bf21e2160d1dc6869fb230b90a23ab030835395b/Lib/asyncio/selector_events.py#L118-L124 https://github.com/python/cpython/blob/bf21e2160d1dc6869fb23...

zokier 2y ago

On Linux you got signalfd and on BSDs you got kqueue. It is only difficult if you are avoiding the tools made specifically to address the problem.

crest 2y ago
On FreeBSD you also have process descriptors (Linux followed suite a while ago) which provide a race free clean interface to supervise processes even if you're not their reaper (so can't use PIDs reliably). You can add them to kqueue using the EVFILT_PROCDESC filter with the NOTE_EXIT flag to get notified when the referenced process exits. The exit status you would normally get from waitpid() is already in the struct kevent. These OS specific APIs are sometimes required and may make certain usecases a lot easier to implement correctly (or even at all), but a job server for a build system isn't one of those. The make jobs can be required to behave and stay in the process group the job server creates for them just like shell job control. Which can be done with just POSIX API. You just have to blow the dust from the relevant tombs the ancients left us (e.g. Advanced Programming in the UNIX Environment). Refuse the temptation and don't make infrastructure tools like gmake depend OS specific APIs. Doing so would make the world a worse place for everyone else.
- o11c 2y ago
  PIDs are always reliable for your own direct children.
- danudey 2y ago
  > don't make infrastructure tools like gmake depend OS specific APIs Even if you're going to implement OS-specific APIs you still need a general case for OSes that don't support those APIs or don't support them correctly. That means you still need to solve the original problem regardless, which then means that it doesn't actually save you time and energy to use those APIs but rather creates more development and maintenance work and not less. If those APIs don't provide you a significant benefit (reliability, performance, etc.) then it's likely not worth implementing them at all if the general case works, and if it doesn't work you have to fix it anyway.
- 2y ago