4 ms·
> My understanding was that threads are subject to GIL and thus cannot run on multiple cores That's exactly the myth I'm trying to address. It's true that in s
by quietbritishjim 6y ago
> My understanding was that threads are subject to GIL and thus cannot run on multiple cores
That's exactly the myth I'm trying to address. It's true that in some circumstances the GIL prevents threads from running in multiple cores, but not all. The GIL would definitely have been released during the file I/O calls in your program. The GIL could well have been released during hashing too, depending on what hashing library you were using, which would have enabled true thread-based parallelism. For example, the built in Python hashlib says [1]:
> Note: For better multithreading performance, the Python GIL is released for data larger than 2047 bytes at object creation or on update.
On the other hand, I'm absolutely not saying that nobody should use multiple processes. For some applications, it's no more complex to use the multiprocessing module than to use threads and not much overhead to pass the data between processes. In that case, it's nice because you just don't even need to worry about whether the GIL is going to play a role, which can be a pain since sometimes there's no documentation saying whether specific functions release the GIL or not. All I'm saying is we should always make clear that to clear to everyone that threads are an option that the GIL doesn't (always) prevent.
> Also, the featured video is from 2015. Maybe it's just outdated information?
Another note on the hashlib page said that the above feature was added in Python 3.1 (released in 2009). That's just about that particular module. I found documentation from Python 1.5 (released in 1998) that describes how C extensions can release the GIL [2]
[1] https://docs.python.org/3/library/hashlib.html https://docs.python.org/3/library/hashlib.html
[2] https://docs.python.org/release/1.5/api/node43.html https://docs.python.org/release/1.5/api/node43.html