3 ms·
The articles' discussion on multiprocessing approaches links to [1] on Pool vs ProcessPoolExecutor. I wish that rather recommendations for non-standard librarie
by RhysU 4y ago
The articles' discussion on multiprocessing approaches links to [1] on Pool vs ProcessPoolExecutor. I wish that rather recommendations for non-standard libraries appeared.
Several years ago, I spent an inordinate amount of time chasing down multiprocessing-usage edge cases. For example: How can I reliably learn that my subprocess died? How can I avoid a fork bomb if my work itself creates new work? Sane serialization semantics? Type hinting? Consistent behavior for spawn, fork, and forkserver? Register callbacks on Futures to run eagerly at completion? Etc.
I ultimately wrote a single-file, drop-it-anywhere “jobserver“ [2] to remove many footcanons. It aims to be the well-tested core inside any more ergonomic API enforcing any additional semantics one might want. No canceling, but that could probably be added.
My "jobserver" is loky-esque [3] as I believe they and I independently were fighting similar challenges but, of course, I prefer mine. Implementation is 560 lines including doc strings. Tests, in the same file, are about another 600 lines.
[1] https://superfastpython.com/multiprocessing-pool-vs-processpoolexecutor/ https://superfastpython.com/multiprocessing-pool-vs-processp...
[2] https://github.com/RhysU/jobserver/blob/master/jobserver.py#L7-L28 https://github.com/RhysU/jobserver/blob/master/jobserver.py#...
[3] https://loky.readthedocs.io/en/stable/ https://loky.readthedocs.io/en/stable/