6 ms·

I remember using the multiprocessing library to speed up some file I/O. Basically I found out that I could spawn a separate process for each physical storage in

I remember using the multiprocessing library to speed up some file I/O. Basically I found out that I could spawn a separate process for each physical storage interface, and I was able to hash like 7 TiB of data in a little over 7 hours. My understanding was that threads are subject to GIL and thus cannot run on multiple cores, whereas processes don't have that same restriction, so I needed to use the multiprocessing library, NOT the multithreading library to parallelize my I/O. Also, the featured video is from 2015. Maybe it's just outdated information?

quietbritishjim 6y ago
> My understanding was that threads are subject to GIL and thus cannot run on multiple cores That's exactly the myth I'm trying to address. It's true that in some circumstances the GIL prevents threads from running in multiple cores, but not all. The GIL would definitely have been released during the file I/O calls in your program. The GIL could well have been released during hashing too, depending on what hashing library you were using, which would have enabled true thread-based parallelism. For example, the built in Python hashlib says [1]: > Note: For better multithreading performance, the Python GIL is released for data larger than 2047 bytes at object creation or on update. On the other hand, I'm absolutely not saying that nobody should use multiple processes. For some applications, it's no more complex to use the multiprocessing module than to use threads and not much overhead to pass the data between processes. In that case, it's nice because you just don't even need to worry about whether the GIL is going to play a role, which can be a pain since sometimes there's no documentation saying whether specific functions release the GIL or not. All I'm saying is we should always make clear that to clear to everyone that threads are an option that the GIL doesn't (always) prevent. > Also, the featured video is from 2015. Maybe it's just outdated information? Another note on the hashlib page said that the above feature was added in Python 3.1 (released in 2009). That's just about that particular module. I found documentation from Python 1.5 (released in 1998) that describes how C extensions can release the GIL [2] [1] https://docs.python.org/3/library/hashlib.html https://docs.python.org/3/library/hashlib.html [2] https://docs.python.org/release/1.5/api/node43.html https://docs.python.org/release/1.5/api/node43.html
joshgev 6y ago
No, it isn't outdated. The claims are still issues. In the scenario you describe, multiprocessing should work pretty well because each processes can run independently of the others and report one big result at the end (all the hashes that process computed). There are plenty of scenarios, however, where the different processes do have to talk to each other frequently which, in Python, means you introduce a ton of overhead in serializing and deserializing these data (Python transmits pickled data between processes). From personal experience I can tell you that this can be a major problem in terms of performance. Other languages give you more flexibility in how to share data between processes. The fundamental claim that threads running pure Python are limited by the GIL still stands. Others do point out that you can get around this in C and some of the standard libraries in Python (like hashlib) do this for you. That goes a long way to helping the issue, of course, but as yet others point out, it is weird to support Python in this context by saying "Yes Python can use threads effectively; all you have to do is use C."
Geminidog 6y ago
You are completely wrong. GIL is released on I/O. So threads can parallelize during I/O calls. This means if your application is bottle necked by IO threading is better. If it's bottle necked by both IO and compute or only compute then multiple processes are better. For your specific case it sounds like the latter, your application is bound by both, therefore multiple processes are better, but what you are implying with your post of using multiprocessing over multi threading as the more "correct" way to "speed up" IO is categorically wrong. Dealing with the IO bottleneck by allowing IO calls to make progress while switching to other threads is a very common pattern in web development and the pattern has literally been baked into several frameworks and languages. NodeJS, GoLang, Erlang and Elixir can handle around 10k concurrent IO calls on a few threads without skipping a beat.
- quietbritishjim 6y ago
  > If it's bottle necked by both IO and compute or only compute then multiple processes are better. This is why I started this whole thread. The myth that you can't multithread compute in Python is so pervasive that even when we're in the middle of a thread specifically about the topic, and right next to my comment where I show hashlib specifically does release the GIL, there's still a comment saying that multiple processes are needed for compute parallelism.
  - Geminidog 6y ago
    My comment is more addressing the guy trying to use processing OVER threading to improve overall speed of completing multiple IO tasks. Yeah I'm aware of your comment. And I can see how you could be pissed off over someone not mentioning what you're saying but please be aware that forum threads can go off into slight tangents when someone states something that is factually wrong. Yes. I know... Programmers can turn to C or they have to use specific libraries to achieve parallelism with threading. I get it. Though you're describing special cases. Similar to the grammatical special cases in English: https://www.e-education.psu.edu/styleforstudents/c1_p6.html https://www.e-education.psu.edu/styleforstudents/c1_p6.html Special cases make certain grammatical rules technically incorrect but the general notion behind those rules and why those rules are still pervasive and stated by people who are aware of the special cases is still valid. It's the same story with the python GIL. There could be multitudes of special cases but this does not negate the existence of a general grammatical rule: Compute in python threads is not parallel. Do I go into a long winded technical discussion of all the special cases or do I just use the generality to point out his categorical mistake? I guess I should have taken the long winded route because if I don't I'll piss you off. Thank you for voting me down btw, I will be sure to address all stakeholders in HN threads in my future replies.

quietbritishjim 6y ago

> My understanding was that threads are subject to GIL and thus cannot run on multiple cores That's exactly the myth I'm trying to address. It's true that in some circumstances the GIL prevents threads from running in multiple cores, but not all. The GIL would definitely have been released during the file I/O calls in your program. The GIL could well have been released during hashing too, depending on what hashing library you were using, which would have enabled true thread-based parallelism. For example, the built in Python hashlib says [1]: > Note: For better multithreading performance, the Python GIL is released for data larger than 2047 bytes at object creation or on update. On the other hand, I'm absolutely not saying that nobody should use multiple processes. For some applications, it's no more complex to use the multiprocessing module than to use threads and not much overhead to pass the data between processes. In that case, it's nice because you just don't even need to worry about whether the GIL is going to play a role, which can be a pain since sometimes there's no documentation saying whether specific functions release the GIL or not. All I'm saying is we should always make clear that to clear to everyone that threads are an option that the GIL doesn't (always) prevent. > Also, the featured video is from 2015. Maybe it's just outdated information? Another note on the hashlib page said that the above feature was added in Python 3.1 (released in 2009). That's just about that particular module. I found documentation from Python 1.5 (released in 1998) that describes how C extensions can release the GIL [2] [1] https://docs.python.org/3/library/hashlib.html https://docs.python.org/3/library/hashlib.html [2] https://docs.python.org/release/1.5/api/node43.html https://docs.python.org/release/1.5/api/node43.html

joshgev 6y ago

No, it isn't outdated. The claims are still issues. In the scenario you describe, multiprocessing should work pretty well because each processes can run independently of the others and report one big result at the end (all the hashes that process computed). There are plenty of scenarios, however, where the different processes do have to talk to each other frequently which, in Python, means you introduce a ton of overhead in serializing and deserializing these data (Python transmits pickled data between processes). From personal experience I can tell you that this can be a major problem in terms of performance. Other languages give you more flexibility in how to share data between processes. The fundamental claim that threads running pure Python are limited by the GIL still stands. Others do point out that you can get around this in C and some of the standard libraries in Python (like hashlib) do this for you. That goes a long way to helping the issue, of course, but as yet others point out, it is weird to support Python in this context by saying "Yes Python can use threads effectively; all you have to do is use C."

Geminidog 6y ago

You are completely wrong. GIL is released on I/O. So threads can parallelize during I/O calls. This means if your application is bottle necked by IO threading is better. If it's bottle necked by both IO and compute or only compute then multiple processes are better. For your specific case it sounds like the latter, your application is bound by both, therefore multiple processes are better, but what you are implying with your post of using multiprocessing over multi threading as the more "correct" way to "speed up" IO is categorically wrong. Dealing with the IO bottleneck by allowing IO calls to make progress while switching to other threads is a very common pattern in web development and the pattern has literally been baked into several frameworks and languages. NodeJS, GoLang, Erlang and Elixir can handle around 10k concurrent IO calls on a few threads without skipping a beat.

quietbritishjim 6y ago
> If it's bottle necked by both IO and compute or only compute then multiple processes are better. This is why I started this whole thread. The myth that you can't multithread compute in Python is so pervasive that even when we're in the middle of a thread specifically about the topic, and right next to my comment where I show hashlib specifically does release the GIL, there's still a comment saying that multiple processes are needed for compute parallelism.
- Geminidog 6y ago
  My comment is more addressing the guy trying to use processing OVER threading to improve overall speed of completing multiple IO tasks. Yeah I'm aware of your comment. And I can see how you could be pissed off over someone not mentioning what you're saying but please be aware that forum threads can go off into slight tangents when someone states something that is factually wrong. Yes. I know... Programmers can turn to C or they have to use specific libraries to achieve parallelism with threading. I get it. Though you're describing special cases. Similar to the grammatical special cases in English: https://www.e-education.psu.edu/styleforstudents/c1_p6.html https://www.e-education.psu.edu/styleforstudents/c1_p6.html Special cases make certain grammatical rules technically incorrect but the general notion behind those rules and why those rules are still pervasive and stated by people who are aware of the special cases is still valid. It's the same story with the python GIL. There could be multitudes of special cases but this does not negate the existence of a general grammatical rule: Compute in python threads is not parallel. Do I go into a long winded technical discussion of all the special cases or do I just use the generality to point out his categorical mistake? I guess I should have taken the long winded route because if I don't I'll piss you off. Thank you for voting me down btw, I will be sure to address all stakeholders in HN threads in my future replies.