3 ms·

> In my case, there was a ~40% speed increase compared to sequential processing. Once a code runs in parallel, the difference in speed performance between the

> In my case, there was a ~40% speed increase compared to sequential processing. Once a code runs in parallel, the difference in speed performance between the parallel methods is very low. But they didn't actually present the total processing time for all of the methods - I assume all of the parallel methods were about 17 seconds? (Compared to the sequential baseline of 29 seconds.) And how were the threaded frameworks configured? How many threads were they told to use (or just the default?), how many threads can they use, and what kind of parallel hardware did they run on? This blog post presents the decision as one-dimensional; it claims all parallelization methods are the same, so the only dimension to choose on is memory efficiency. But I'm skeptical that all parallelization methods are the same, and the experimental design gives me no information on that front.

nickcw 9y ago
I had a squint at the ThreadPoolExecutor code on github and it uses the default parameters. Exactly how many threads you get depends on which version of python you are running: > Changed in version 3.5: If max_workers is None or not given, it will default to the number of processors on the machine, multiplied by 5, assuming that ThreadPoolExecutor is often used to overlap I/O instead of CPU work and the number of workers should be higher than the number of workers for ProcessPoolExecutor. Not knowing how many things are happening in parallel means it is difficult to draw conclusions from it.
- ku3o 9y ago
  Thanks for pointing this out. I updated the article.
ku3o 9y ago
> it claims all parallelization methods are the same, so the only dimension to choose on is memory efficiency From a speed (overall script duration), they are very similar and the differences between them might be as well caused by a different state of a network. My main interest was how the methods perform regarding memory usage. That's why I focused on one dimension only.
- scott_s 9y ago
  And it's fair to care about memory usage, but I think it's unlikely that your circumstances regarding runtime performance (both your testing script and testing environment) will generalize to other scenarios.

nickcw 9y ago

I had a squint at the ThreadPoolExecutor code on github and it uses the default parameters. Exactly how many threads you get depends on which version of python you are running: > Changed in version 3.5: If max_workers is None or not given, it will default to the number of processors on the machine, multiplied by 5, assuming that ThreadPoolExecutor is often used to overlap I/O instead of CPU work and the number of workers should be higher than the number of workers for ProcessPoolExecutor. Not knowing how many things are happening in parallel means it is difficult to draw conclusions from it.

ku3o 9y ago
Thanks for pointing this out. I updated the article.

ku3o 9y ago

> it claims all parallelization methods are the same, so the only dimension to choose on is memory efficiency From a speed (overall script duration), they are very similar and the differences between them might be as well caused by a different state of a network. My main interest was how the methods perform regarding memory usage. That's why I focused on one dimension only.

scott_s 9y ago
And it's fair to care about memory usage, but I think it's unlikely that your circumstances regarding runtime performance (both your testing script and testing environment) will generalize to other scenarios.