6 ms·
Many other people "know" about the GIL, to the extent of believing there's no point using threads in python "because of the GIL". I had a funny such experience
by avip 7y ago
Many other people "know" about the GIL, to the extent of believing there's no point using threads in python "because of the GIL".
I had a funny such experience lately in a job interview. I told the interviewer his misconception could be falsified with ~10 LOC summing a list with 2 threads.
- muxator 7y agoCould you expand on this please?
- mlthoughts2018 7y agoHow did you ensure those threads would actually execute concurrently? If any operations were GIL-bound (which is extremely common, even if not intentional, by reliance on bytecode instructions dealing with CPython API under the hood, like attribute lookups or iteration special methods), then execution is probably interleaved serially and constrained by the interpreter’s GIL allocation, and slower than just summing serially. I’ve seen a lot of people who are cocksure they know some contrarian “actually you can use threads” trivia about Python and just naively use the threading module or naively use the newish ThreadPoolExecutor stuff not realizing that no, in fact, it’s not somehow magically always GIL-avoiding to do so.
- rdtsc 7y ago> How did you ensure those threads would actually execute concurrently? By having multiple concurrent IO requests from multiple threads? Can probably log detailed timestamps to see if IO requests were happening in parallel.
- nine_k 7y agoThe only point of using Python threads is to wait for I/O. Which, while limiting, is a huge use case, for network servers in particular.
- ses1984 7y agoCan't you also defer work to c extensions in threads?
- nine_k 7y agoSome extensions do release GIL while doing its own parallel processing, so, to a degree, yes.
- celticmusic 7y agoI don't know the specifics of Python's C interfaces, but I know Ruby by default will cover all 3rd party extensions in the GIL for safety and you can release it upon entry into the extension using the C api. But doing so can be very dangerous if you're calling back into the Ruby core as you could inadvertently cause problems that the GIL was meant to protect you from. I imagine Python works roughly the same.
- paulddraper 7y agoYes. Numpy releases the GIL for its array operations. Parallelism is possible, but only for computations in native code. FWIW Node.js is essentially the same situation.
- nine_k 7y agoNode got background workers recently. Otherwise yes, Node or Ruby play by the same global lock rules.
- paulddraper 7y agoSame in Python, except multiprocessing has actually been around for longer. Launching multiple processes without mutable shared memory is always an option.
- edgyquant 7y agoI'd like to see these 10 lines that supposedly avoid the problems with the GIL.
- aspaceman 7y agoIt still hits the problems, but it can still be better than single-threaded. That's the point of his comment. People say the GIL is bad so they throw the baby out with the bathwater and no longer use threads, which isn't very well-reasoned. Been a while since I've used Python, but as far as I remember the GIL only affects Python objects. So if you use Numpy for operations, you can avoid the GIL.
- numbernine 7y agoCan you give an example for your last sentence?
- aspaceman 7y agoHere's a link: https://scipy-cookbook.readthedocs.io/items/ParallelProgramming.html https://scipy-cookbook.readthedocs.io/items/ParallelProgramm... And a quote in italics below. And, more importantly for us, while numpy is doing an array operation, python also releases the GIL. Thus if you tell one thread to do: print "%s %s %s %s and %s" %( ("spam",) *3 + ("eggs",) + ("spam",) ) A = B + C print A During the print operations and the % formatting operation, no other thread can execute. But during the A = B + C, another thread can run - and if you've written your code in a numpy style, much of the calculation will be done in a few array operations like A = B + C. Thus you can actually get a speedup from using multiple threads.
- paulddraper 7y ago> as far as I remember the GIL only affects Python objects. So if you use Numpy for operations, you can avoid the GIL. Correct. Threads cannot help Python-heavy code; they can help I/O-heavy or native-heavy code (assuming native code releases the GIL). Even without multithreading, numpy programming can give amazing speedups. It's the reason why data science Python isn't dead in the water.
- rdtsc 7y ago> I had a funny such experience lately in a job interview. I told the interviewer his misconception could be falsified with ~10 LOC summing a list with 2 threads. I had multiple of those experiences. Even then people wouldn’t believe me so we had to look at Python C code and read about threads and IO operations in C and such. The other alternative is to write a deliberately slow socket server that takes say 5 seconds to respond then then write a threaded client to issue requests and seeing how they complete in parallel.
- avip 7y agoOk, I see some comments (rightfully) asking for less talk and more code. # main.py import random from concurrent.futures import ThreadPoolExecutor as Pool items = [random.random() for _ in range(10 ** 7)] def run(items, n): step = len(items) // n with Pool(max_workers=n) as ex: res = [ex.submit(sum, items[i*step : (i+1)*step]) for i in range(n)] return sum(r.result() for r in res) if __name__ == '__main__': import timeit import sys n = sys.argv[1] if len(sys.argv) > 1 else 1 time = timeit.timeit('run(items, %s)' % n, 'from __main__ import run, items', number=10) print("%s\t%.3f" % (n, time / 10)) $ for x in `seq 1 16` ; do python3 -m main $x ; done 1 0.172 2 0.170 3 0.166 4 0.155 5 0.149 6 0.142 7 0.144 8 0.140 9 0.136 10 0.135 11 0.135 12 0.137 13 0.135 14 0.136 15 0.136 16 0.136
- ginko 7y agoThat’s not exactly what I would call good speedup.
- avip 7y agoThe point of the code is not to speedup the execution of summing a list of random number, but rather to speedup the acknowledgement of N random python developers that they have some misconceptions about the GIL. I think it does that pretty well but, well, that's just like my opinion.
- kjksf 7y agoThe major reason people use threads is to speed up compute intensive tasks. Ideally, pure computation like the one in your example should get a speed up linear to number of threads. 2 threads -> 2x faster. 3 threads -> 3x faster, up to a number of cores. If that's not happening then you have excessive locking. In Python excessive locking is caused by GIL - Global Interpreter Lock. As the name implies, there's a single, global lock and execution in a single threads effectively blocks all other threads because it holds GIL captive. What this particular benchmarks is showing is that GIL is as bad as everyone says: instead of getting 16x speedup, you get speedups that are almost within margin of error for such a coarse measure.