3 ms·
Lately I've found out that multiprocessing will not help you if your program is multithreaded. There is no sane way of forking a multithreaded program. For one,
by glic3rinu 9y ago
Lately I've found out that multiprocessing will not help you if your program is multithreaded. There is no sane way of forking a multithreaded program. For one, the child process will inherit a copy of all locks in the state they where at forking time, possibly causing random crashes and deadlocks.
- lstyls 9y ago> There is no sane way of forking a multithreaded program The sane way of forking a multithreaded process is to exec immediately after.
- dom0 9y agoIt is possible to do more after a fork (cf. async-signal-safe), but it's hairy enough to just say — don't, always exec (similar to how doing actual work in a signal handler is generally a very bad idea).
- d0mine 9y agothat is why 'forkserver' start method exists. https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods https://docs.python.org/3/library/multiprocessing.html#conte...
- hyperbovine 9y agoIf the child program is multithreaded then it's almost certainly not pure Python in the first place. So, wrap it up in `with nogil:` Cython statements and use the threading module (or concurrent.futures.ThreadPoolExecutor).