8 ms·

That would not give me a traceback, and it would not catch segfaults or ctrl-C. It's a different style of programming that would work around part of the issue (

That would not give me a traceback, and it would not catch segfaults or ctrl-C. It's a different style of programming that would work around part of the issue (only the exceptions, and only by checking for errors manually), whereas what perplexes me is that multiprocessing and concurrent.futures are not designed to handle these error conditions themselves.

nine_k 13y ago
Why, it would give you a traceback if your worker process collects it, as a reasonable worker would. Since workers are probably non-interactive, they can't exactly receive a Ctrl+C. The dispatcher process, OTOH, will. I suppose a pool should kill all processes on SystemExit. When you think about more details, you grow more and more sophisticated solution that best suits your needs. But the point of the post is different: it's a dead simple way to start. Starting is often the hardest part.
- andreasvc 13y ago
  Collecting tracebacks is the sort of plumbing you would expect a 'batteries included' library to do. I didn't expect the workers to receive the ctrl-C, I'd expect the main process to terminate, along with its children; none of that happens. I think it's clear that these sorts of details belong in the library. Everybody could hack together their own traceback collector, segfault handlers etc., but it's difficult to get right and in a scripting language you should be able to forget about such things.
wc- 13y ago
Check out the combo of sys.exc_info and traceback.format_tb, I've used these two functions to generate stack traces for failed worker jobs in the past with pretty good success. edit: I guess I didn't really answer the segfault issue. Beyond checking the return code of the child process to see if it segfault'ed (so you can log it or retry it) I'm not sure of how you could get more informative failing info.
- andreasvc 13y ago
  Indeed, you can pass the traceback manually as a string. With the segfaults you can use the 'faulthandler' module to dump information about them, and I see that starting with Python 3.3 they correctly detect child processes that have died: http://hg.python.org/cpython/rev/6d6099f7fe89 http://hg.python.org/cpython/rev/6d6099f7fe89

nine_k 13y ago

Why, it would give you a traceback if your worker process collects it, as a reasonable worker would. Since workers are probably non-interactive, they can't exactly receive a Ctrl+C. The dispatcher process, OTOH, will. I suppose a pool should kill all processes on SystemExit. When you think about more details, you grow more and more sophisticated solution that best suits your needs. But the point of the post is different: it's a dead simple way to start. Starting is often the hardest part.

andreasvc 13y ago
Collecting tracebacks is the sort of plumbing you would expect a 'batteries included' library to do. I didn't expect the workers to receive the ctrl-C, I'd expect the main process to terminate, along with its children; none of that happens. I think it's clear that these sorts of details belong in the library. Everybody could hack together their own traceback collector, segfault handlers etc., but it's difficult to get right and in a scripting language you should be able to forget about such things.

wc- 13y ago

Check out the combo of sys.exc_info and traceback.format_tb, I've used these two functions to generate stack traces for failed worker jobs in the past with pretty good success. edit: I guess I didn't really answer the segfault issue. Beyond checking the return code of the child process to see if it segfault'ed (so you can log it or retry it) I'm not sure of how you could get more informative failing info.

andreasvc 13y ago
Indeed, you can pass the traceback manually as a string. With the segfaults you can use the 'faulthandler' module to dump information about them, and I see that starting with Python 3.3 they correctly detect child processes that have died: http://hg.python.org/cpython/rev/6d6099f7fe89 http://hg.python.org/cpython/rev/6d6099f7fe89