3 ms·
"You can't just use functions defined in your tool, you need to create a faux-cli interface in order to run each parallel worker." the multiprocessing library
by dec0dedab0de 9y ago
"You can't just use functions defined in your tool, you need to create a faux-cli interface in order to run each parallel worker."
the multiprocessing library allows you to launch multiple processes using your function definitions. It's almost the same as the multithreading library but does not share data.
It seems the real problem, as you pointed out, is the additional memory. I didn't consider situations where each process would need an identical large data set, instead of just a small chunk to work on.
- ant6n 9y agoIt gets more interesting when you have a large data set that's required for the computation, but as you compute, you may discover partial solutions that can be cached and used by other workers. So not only a large read-only data set, but also a read-write cache used by all workers. This sort of thing is relatively easy with threads, but basically impossible with multiprocessing.
- dom0 9y agoDepending on where you want to go and the application, such things may be good idea for a low number of workers but can become a major bottleneck.