4 ms·

The point of the code is not to speedup the execution of summing a list of random number, but rather to speedup the acknowledgement of N random python developer

• kjksf 7y ago
  The major reason people use threads is to speed up compute intensive tasks. Ideally, pure computation like the one in your example should get a speed up linear to number of threads. 2 threads -> 2x faster. 3 threads -> 3x faster, up to a number of cores. If that's not happening then you have excessive locking. In Python excessive locking is caused by GIL - Global Interpreter Lock. As the name implies, there's a single, global lock and execution in a single threads effectively blocks all other threads because it holds GIL captive. What this particular benchmarks is showing is that GIL is as bad as everyone says: instead of getting 16x speedup, you get speedups that are almost within margin of error for such a coarse measure.
• pnako 7y ago
  I'm a casual Python programmer (just bit of scripting) and I had no particular misconception about the GIL; I don't care because I write single-threaded cookie cutter scripts. Your example I think is demonstrating the opposite of what you want to show. Those figures are atrocious.
  • avip 7y ago
    The opposite of what? All I want is the GIL "discussion" to be based on facts rather than emotions, hyperboles and FUD. I'm not even sure what the opposite of that is.
    • deanmoriarty 7y ago
      What were the misconceptions of your interviewer you were trying to prove wrong with your POC of multiple threads summing a list? The way I read your post, it seemed like your interviewer told you that threads in python are not effective for parallel computation because of the GIL, and your example proves exactly that. The performance of your threads are absolutely horrible, if you were to do that in C++/Java/Go you would likely see a speedup in the order of min(16, cores). Your example proves that your threads are exactly serialized during the computation, which I assume was the point of your interviewer (but please clarify my assumption). Perhaps your interviewer was instead proposing that threads in Python work like a charm?
      • avip 7y ago
        2 things: 1. There is a common misconception that any threaded python code is slower than sequential. This is trivially false for IO-bound, and possibly, potentially, in some cases false for CPU-bound. Said interviewer had that very misconception. 2. More importantly - you and another parent are right. The code does not demonstrate what I thought it does. This is proven by removing the ThreadPool and leaving the code intact: import random import math from concurrent.futures import ThreadPoolExecutor as Pool items = [random.random() for _ in range(math.factorial(11))] def run(items, n): step = len(items) // n res = [sum(items[i*step : (i+1)*step]) for i in range(n)] return sum(res) if __name__ == '__main__': import timeit import sys n = sys.argv[1] if len(sys.argv) > 1 else 1 time = timeit.timeit('run(items, %s)' % n, 'from __main__ import run, items', number=10) print("%s\t%.3f" % (n, time / 10)) $ for x in `seq 1 11`; do python3 -m main $x ; done 1 0.799 2 0.749 3 0.671 4 0.715 5 0.730 6 0.704 7 0.649 8 0.631 9 0.689 10 0.613 11 0.616 This is a result I'd have to silently contemplate before making any further comments.
• reitzensteinm 7y ago
  I feel like you're beating a strawman. Using threads to speed up IO in Python is common, and for computation heavy work the fear of the GIL is 100% justified, as your example shows.