4 ms·

Will this finally let me write a parallel Python script that doesn't explode when I press control+C?

by rcthompson 9y ago

metalliqaz 9y ago
Depends. What do you want to happen when you press ctrl+C?
- rcthompson 9y ago
  I want it to not hang forever, not spam the console with pages of exception tracebacks, and not require a ton of boilerplate process-management code to accomplish the above. Ideally it would also allow me to handle other exceptions (i.e. besides KeyboardInterrupt) that occur both in the main process and child processes. I've never figured out how to do this with multiprocessing, despite lots of attempts.
  - mixmastamyk 9y ago
    Don't know what you did wrong, but you should be able to catch exceptions at their appropriate level and shut down gracefully.
    - icegreentea2 9y ago
      multiprocessing was designed to mirror the threading library... so the exceptions are specifically set up not to cross thread/process boundaries. But the same strategies to handle multithreaded exceptions and keyboardinterrupt should apply. If you want the child processes to shutdown when the main process goes down, you should be able to just set .daemon = True on the process objects before you start them. If you want exceptions in the children thread to propagate up to the main thread and then handle, looks like you'd just need to send the exception across a queue or something in multiproccessing. In the new futures library, your future (wrapping the process) has a return value property you can try to access. If the child process ran into an exception, that exception would be raised in the main process when you tried to access that return value.
    - rcthompson 9y ago
      I know it should be possible, but I've never figured out how to do it, nor have I ever seen any example code using multiprocessing that does it. If you know of some example code that properly catches and handles exceptions in any process, I'd love to see it.
  - xapata 9y ago
    The concurrent.futures module handles exceptions well, waiting to raise them until you call ``future.result``. Or did I misunderstand the problem?
    - rcthompson 9y ago
      My complaints are about the multiprocessing module, and my hope is that concurrent.futures will solve some of them.
      - xapata 9y ago
        The latter library has a better design in some respects, but isn't quite as pleasant as ``Pool.imap_unordered`` when you're not worried about exceptions in the subprocesses/threads. However, I've been using ``concurrent.futures`` more in the last year.
jzwinck 9y ago
That's always been easy enough, just a little hidden: import signal signal.signal(signal.SIGINT, signal.SIG_DFL) Now it will just die on Ctrl-C. For text filter programs it's a good idea to do the same for SIGPIPE too.
Rotareti 9y ago
Isn't this what the `with` keyword was made for: with ThreadPool() as pool: fut = pool.submit(foo) print(fut.result()) My idea is that ctrl+c would now cancel child operations cleanly?

Will this finally let me write a parallel Python script that doesn't explode when I press control+C?

metalliqaz 9y ago

Depends. What do you want to happen when you press ctrl+C?

rcthompson 9y ago
I want it to not hang forever, not spam the console with pages of exception tracebacks, and not require a ton of boilerplate process-management code to accomplish the above. Ideally it would also allow me to handle other exceptions (i.e. besides KeyboardInterrupt) that occur both in the main process and child processes. I've never figured out how to do this with multiprocessing, despite lots of attempts.
- mixmastamyk 9y ago
  Don't know what you did wrong, but you should be able to catch exceptions at their appropriate level and shut down gracefully.
  - icegreentea2 9y ago
    multiprocessing was designed to mirror the threading library... so the exceptions are specifically set up not to cross thread/process boundaries. But the same strategies to handle multithreaded exceptions and keyboardinterrupt should apply. If you want the child processes to shutdown when the main process goes down, you should be able to just set .daemon = True on the process objects before you start them. If you want exceptions in the children thread to propagate up to the main thread and then handle, looks like you'd just need to send the exception across a queue or something in multiproccessing. In the new futures library, your future (wrapping the process) has a return value property you can try to access. If the child process ran into an exception, that exception would be raised in the main process when you tried to access that return value.
  - rcthompson 9y ago
    I know it should be possible, but I've never figured out how to do it, nor have I ever seen any example code using multiprocessing that does it. If you know of some example code that properly catches and handles exceptions in any process, I'd love to see it.
- xapata 9y ago
  The concurrent.futures module handles exceptions well, waiting to raise them until you call ``future.result``. Or did I misunderstand the problem?
  - rcthompson 9y ago
    My complaints are about the multiprocessing module, and my hope is that concurrent.futures will solve some of them.
    - xapata 9y ago
      The latter library has a better design in some respects, but isn't quite as pleasant as ``Pool.imap_unordered`` when you're not worried about exceptions in the subprocesses/threads. However, I've been using ``concurrent.futures`` more in the last year.

jzwinck 9y ago

That's always been easy enough, just a little hidden: import signal signal.signal(signal.SIGINT, signal.SIG_DFL) Now it will just die on Ctrl-C. For text filter programs it's a good idea to do the same for SIGPIPE too.

Rotareti 9y ago

Isn't this what the `with` keyword was made for: with ThreadPool() as pool: fut = pool.submit(foo) print(fut.result()) My idea is that ctrl+c would now cancel child operations cleanly?