5 ms·
Ok, I see some comments (rightfully) asking for less talk and more code. # main.py import random from concurrent.futures import ThreadPoolExecutor as
by avip 7y ago
Ok, I see some comments (rightfully) asking for less talk and more code.
# main.py
import random
from concurrent.futures import ThreadPoolExecutor as Pool
items = [random.random() for _ in range(10 ** 7)]
def run(items, n):
step = len(items) // n
with Pool(max_workers=n) as ex:
res = [ex.submit(sum, items[i*step : (i+1)*step]) for i in range(n)]
return sum(r.result() for r in res)
if __name__ == '__main__':
import timeit
import sys
n = sys.argv[1] if len(sys.argv) > 1 else 1
time = timeit.timeit('run(items, %s)' % n, 'from __main__ import run, items', number=10)
print("%s\t%.3f" % (n, time / 10))
$ for x in `seq 1 16` ; do python3 -m main $x ; done
1 0.172
2 0.170
3 0.166
4 0.155
5 0.149
6 0.142
7 0.144
8 0.140
9 0.136
10 0.135
11 0.135
12 0.137
13 0.135
14 0.136
15 0.136
16 0.136
- ginko 7y agoThat’s not exactly what I would call good speedup.
- avip 7y agoThe point of the code is not to speedup the execution of summing a list of random number, but rather to speedup the acknowledgement of N random python developers that they have some misconceptions about the GIL. I think it does that pretty well but, well, that's just like my opinion.
- kjksf 7y agoThe major reason people use threads is to speed up compute intensive tasks. Ideally, pure computation like the one in your example should get a speed up linear to number of threads. 2 threads -> 2x faster. 3 threads -> 3x faster, up to a number of cores. If that's not happening then you have excessive locking. In Python excessive locking is caused by GIL - Global Interpreter Lock. As the name implies, there's a single, global lock and execution in a single threads effectively blocks all other threads because it holds GIL captive. What this particular benchmarks is showing is that GIL is as bad as everyone says: instead of getting 16x speedup, you get speedups that are almost within margin of error for such a coarse measure.
- pnako 7y agoI'm a casual Python programmer (just bit of scripting) and I had no particular misconception about the GIL; I don't care because I write single-threaded cookie cutter scripts. Your example I think is demonstrating the opposite of what you want to show. Those figures are atrocious.
- avip 7y agoThe opposite of what? All I want is the GIL "discussion" to be based on facts rather than emotions, hyperboles and FUD. I'm not even sure what the opposite of that is.
- deanmoriarty 7y agoWhat were the misconceptions of your interviewer you were trying to prove wrong with your POC of multiple threads summing a list? The way I read your post, it seemed like your interviewer told you that threads in python are not effective for parallel computation because of the GIL, and your example proves exactly that. The performance of your threads are absolutely horrible, if you were to do that in C++/Java/Go you would likely see a speedup in the order of min(16, cores). Your example proves that your threads are exactly serialized during the computation, which I assume was the point of your interviewer (but please clarify my assumption). Perhaps your interviewer was instead proposing that threads in Python work like a charm?
- avip 7y ago2 things: 1. There is a common misconception that any threaded python code is slower than sequential. This is trivially false for IO-bound, and possibly, potentially, in some cases false for CPU-bound. Said interviewer had that very misconception. 2. More importantly - you and another parent are right. The code does not demonstrate what I thought it does. This is proven by removing the ThreadPool and leaving the code intact: import random import math from concurrent.futures import ThreadPoolExecutor as Pool items = [random.random() for _ in range(math.factorial(11))] def run(items, n): step = len(items) // n res = [sum(items[i*step : (i+1)*step]) for i in range(n)] return sum(res) if __name__ == '__main__': import timeit import sys n = sys.argv[1] if len(sys.argv) > 1 else 1 time = timeit.timeit('run(items, %s)' % n, 'from __main__ import run, items', number=10) print("%s\t%.3f" % (n, time / 10)) $ for x in `seq 1 11`; do python3 -m main $x ; done 1 0.799 2 0.749 3 0.671 4 0.715 5 0.730 6 0.704 7 0.649 8 0.631 9 0.689 10 0.613 11 0.616 This is a result I'd have to silently contemplate before making any further comments.
- reitzensteinm 7y agoI feel like you're beating a strawman. Using threads to speed up IO in Python is common, and for computation heavy work the fear of the GIL is 100% justified, as your example shows.
- paulddraper 7y ago(1) Even taking a 20% speedup at face value, I'd still describe "single-process Python can only use 1 core" as accurate, with rounding. (2) You're not measuring what you think you are measuring. Take out the Pool altogether: def run(items, n): step = len(items) // n res = [sum(items[i*step : (i+1)*step]) for i in range(n)] return sum(r for r in res) and the results are similar: 1 0.170 2 0.170 3 0.147 4 0.145 5 0.142 6 0.142 7 0.138 8 0.137 9 0.135 10 0.138 I have some guesses why it improves, but I'm not certain. It's definitely nothing to do with multithreading. EDIT: As an aside, you can speed this up 20x by using Numpy: items = numpy.array([random.random() for _ in range(10 ** 7)]) def run(items, n): return numpy.sum(items) 1 0.007 Supporting the original point that running code in low-level languages produces amazing speedups vs fancy parallelism methods.
- orf 7y agoAnd with no threads: 49.6 ms ± 947 µs per loop. I’m pretty sure the time “savings” you are seeing here come from somewhere else. At a first glance, your copying the huge list while submitting it to the thread pool, this has overhead. Maybe lots of smaller copies are faster on your machine.