3 ms·

The opposite of what? All I want is the GIL "discussion" to be based on facts rather than emotions, hyperboles and FUD. I'm not even sure what the opposite of

by avip 7y ago

The opposite of what? All I want is the GIL "discussion" to be based on facts rather than emotions, hyperboles and FUD. I'm not even sure what the opposite of that is.

deanmoriarty 7y ago
What were the misconceptions of your interviewer you were trying to prove wrong with your POC of multiple threads summing a list? The way I read your post, it seemed like your interviewer told you that threads in python are not effective for parallel computation because of the GIL, and your example proves exactly that. The performance of your threads are absolutely horrible, if you were to do that in C++/Java/Go you would likely see a speedup in the order of min(16, cores). Your example proves that your threads are exactly serialized during the computation, which I assume was the point of your interviewer (but please clarify my assumption). Perhaps your interviewer was instead proposing that threads in Python work like a charm?
- avip 7y ago
  2 things: 1. There is a common misconception that any threaded python code is slower than sequential. This is trivially false for IO-bound, and possibly, potentially, in some cases false for CPU-bound. Said interviewer had that very misconception. 2. More importantly - you and another parent are right. The code does not demonstrate what I thought it does. This is proven by removing the ThreadPool and leaving the code intact: import random import math from concurrent.futures import ThreadPoolExecutor as Pool items = [random.random() for _ in range(math.factorial(11))] def run(items, n): step = len(items) // n res = [sum(items[i*step : (i+1)*step]) for i in range(n)] return sum(res) if __name__ == '__main__': import timeit import sys n = sys.argv[1] if len(sys.argv) > 1 else 1 time = timeit.timeit('run(items, %s)' % n, 'from __main__ import run, items', number=10) print("%s\t%.3f" % (n, time / 10)) $ for x in `seq 1 11`; do python3 -m main $x ; done 1 0.799 2 0.749 3 0.671 4 0.715 5 0.730 6 0.704 7 0.649 8 0.631 9 0.689 10 0.613 11 0.616 This is a result I'd have to silently contemplate before making any further comments.

by avip 7y ago

The opposite of what? All I want is the GIL "discussion" to be based on facts rather than emotions, hyperboles and FUD. I'm not even sure what the opposite of that is.

deanmoriarty 7y ago

What were the misconceptions of your interviewer you were trying to prove wrong with your POC of multiple threads summing a list? The way I read your post, it seemed like your interviewer told you that threads in python are not effective for parallel computation because of the GIL, and your example proves exactly that. The performance of your threads are absolutely horrible, if you were to do that in C++/Java/Go you would likely see a speedup in the order of min(16, cores). Your example proves that your threads are exactly serialized during the computation, which I assume was the point of your interviewer (but please clarify my assumption). Perhaps your interviewer was instead proposing that threads in Python work like a charm?

avip 7y ago
2 things: 1. There is a common misconception that any threaded python code is slower than sequential. This is trivially false for IO-bound, and possibly, potentially, in some cases false for CPU-bound. Said interviewer had that very misconception. 2. More importantly - you and another parent are right. The code does not demonstrate what I thought it does. This is proven by removing the ThreadPool and leaving the code intact: import random import math from concurrent.futures import ThreadPoolExecutor as Pool items = [random.random() for _ in range(math.factorial(11))] def run(items, n): step = len(items) // n res = [sum(items[i*step : (i+1)*step]) for i in range(n)] return sum(res) if __name__ == '__main__': import timeit import sys n = sys.argv[1] if len(sys.argv) > 1 else 1 time = timeit.timeit('run(items, %s)' % n, 'from __main__ import run, items', number=10) print("%s\t%.3f" % (n, time / 10)) $ for x in `seq 1 11`; do python3 -m main $x ; done 1 0.799 2 0.749 3 0.671 4 0.715 5 0.730 6 0.704 7 0.649 8 0.631 9 0.689 10 0.613 11 0.616 This is a result I'd have to silently contemplate before making any further comments.