5 ms·

It's very disappointing to see someone who seems authoritative spreading the myth that the GIL prevents you from using threads to achieve concurrency. It didn'

It's very disappointing to see someone who seems authoritative spreading the myth that the GIL prevents you from using threads to achieve concurrency. It didn't work with his toy example, calling the Fibonacci function, because it's pure Python. Typically, if you have pure CPU-bound processing like this then you wouldn't want to use pure Python anyway as it would be too slow. You'd either using C extension library like numpy, scipy or pytorch, or (more rarely) write the code yourself in Cython. In either case the GIL is released whenever you make a call into them (you have to do it manually in Cython but it's straightforward). If his example had been multiplying matrices together with np.dot then threads wouldn't have been a problem. The GIL is also released whenever you do I/O like reading data from a socket or reading from a file. This includes libraries that do this for you, such as those reading from a database (whether a remote DBMS like Postgres or a file-based one like SQLite). Taken together, these cover 99% of cases where you want concurrency. In that 1% where the GIL is not released, fixing that pure Python computation code is often worth doing before adding concurrency anyway.

theelous3 6y ago
It's a python conference demonstrating achieving goals in python. Basically saying "eh use C for cpu bound work" isn't helpful. I think you missed the point a bit, which was purely to show co-operative yielding. The fact that some libs use c and release the gil really doesn't matter. This is another way. The real world usecase, as with basically all python, is io.
- quietbritishjim 6y ago
  I'm not saying write your whole program in C. I'm saying, at worst, write a tiny corner of it in C so that you don't have to restructure the rest of your application - overall it would be simpler, even for most mostly-Python devs. Or, much more likely, use a library like numpy which you were probably going to do anyway. I regularly use numpy and don't find myself thinking "I wish didn't have to write all this C code...". (I must admit, an unfortunate consequence of this is that I've interviewed a few junior candidates that were so used to writing vectorised code that they seemed to be terrified of using a for loop!) I admit I didn't watch the whole of the talk. (Personally I much prefer learning from text than video, and I was put off by the GIL bit anyway.) From the parts I saw, it seemed like the GIL issue was critical for motivating everything he did afterwards, but perhaps there were other reasons that became clear later. In that case, he could have avoided mentioning the GIL altogether (given that he ended up being so misleading about it). > The real world usecase, as with basically all python, is io. That is just straight up not true. Yes, IO is a valid use case of Python but it definitely isn't "basically all Python". I'm sure the vast majority of data scientists use Python, and for most of them their workloads are almost entirely computational (using deep learning libraries with C/C++/CUDA backends).
  - toyg 6y ago
    I think we have to consider that this talk is from 2015, when datascience was a much smaller part of the python ecosystem. I have used python since 2003 for various things and never once I’ve touched numpy, but I bet today 1 in 3 python devs use it probably every day. I understand your point of view and I understand the speaker’s own too - his is just more old-school. We’ve been able to sidestep the GIL in many ways since forever (for example with alternative implementations like Jython, IronPython, PyPy etc), and I think everyone has known that for a long time, but here Dave just wanted to show what the actual problem is, beyond the buzzwords (it had become a bit of a myth in the early ‘10s, when Go suddenly exploded in the sysadmin/backend niche partially because “it can do parallelism better”).
    - quietbritishjim 6y ago
      About the fact this talk is from 2015: The GIL is released by many libraries that aren't about data science, so was and is relevant to many other types of application. In the Python standard library, for example, zipfile (which can operate on files or in-memory buffers) and hashlib come to mind. As I mentioned in another comment, hashlib calls have released the GIL at least as far back as 2009, well before this talk, and releasing the GIL was possible all the way back in 1998. Many (most?) Python libraries are implemented in C, and almost all of those that are will release the GIL. About appreciating both points of view: I'm sorry if I'm making it sound like I think Python developers should always use threads and never multiprocessing. I don't believe that. And I don't believe the speaker should have reformulated their whole talk to use threads instead of processes. Just that it's such a pervasive myth that the GIL prevents all thread-based parallelism that they should have been careful to avoid reinforcing that. The speaker showed an example where the GIL actually did prevent parallelism; all I wanted was for them to add verbally that, by the way, if you were calling almost any CPU-bound library function rather than one you wrote yourself then this wouldn't be a problem.
znpy 6y ago
> It didn't work with his toy example, calling the Fibonacci function, because it's pure Python. maybe I do want to write pure python AND have multi-threaded applications ?
- Geminidog 6y ago
  One use case is IO bound applications. Similar to NodeJS which can handle 10K concurrent IO calls on a single thread, python releases the GIL on IO. So you can write pure python and have multi threading and have all the benefits of "parallelism" depending on whether or not python releases the GIL which it does with certain libraries and on IO. For all else use multiprocessing.
nurettin 6y ago
> GIL prevents you from using threads to achieve concurrency I didn't watch, but nothing in python prevents concurrency. In Python, a lot of things prevent parallelism. Threads/async are ways of achieving concurrency in python. Parallelism is achieved using multiprocessing. It is inefficient, but it works.
- quietbritishjim 6y ago
  Oops, you're right, I was talking about parallelism, not just concurrency. Thanks for the correction. > Parallelism is achieved using multiprocessing. As I said in my previous comment (but used the wrong term), threads are a perfectly reasonable way of getting parallelism in Python, so long so long your application uses C-based libraries like numpy (or even some built-in modules like zipfile). In my experience, the vast majority of Python programs that might need parallelism fall into that category (or are IO bound) anyway.
- Geminidog 6y ago
  Also from certain perspectives parallelism is achieved with threads during IO. So if 20 threads are making blocking IO calls, all 20 threads can make progress on those IO calls in parallel while having an additional parallel thread doing compute operations yielding a total of 21 threads executing in parallel.
  - nurettin 6y ago
    We use "parallel" explicitly to mean "running on multiple cores scheduled by the kernel".
    - Geminidog 6y ago
      Yes, but "parallel" is also used in CS to specify multiple things happening at the same time including IO. https://www.wikiwand.com/en/Parallel_computing https://www.wikiwand.com/en/Parallel_computing Quote: "Parallel computing is a type of computation where many calculations or the execution of processes are carried out simultaneously." https://www.wikiwand.com/en/Parallel_communication https://www.wikiwand.com/en/Parallel_communication Quote: "In data transmission, parallel communication is a method of conveying multiple binary digits (bits) simultaneously. It contrasts with serial communication, which conveys only a single bit at a time; this distinction is one way of characterizing a communications link."
huseyinkeles 6y ago
For the curious, GIL = Global Interpreter Lock
jacobwilliamroy 6y ago
I remember using the multiprocessing library to speed up some file I/O. Basically I found out that I could spawn a separate process for each physical storage interface, and I was able to hash like 7 TiB of data in a little over 7 hours. My understanding was that threads are subject to GIL and thus cannot run on multiple cores, whereas processes don't have that same restriction, so I needed to use the multiprocessing library, NOT the multithreading library to parallelize my I/O. Also, the featured video is from 2015. Maybe it's just outdated information?
- quietbritishjim 6y ago
  > My understanding was that threads are subject to GIL and thus cannot run on multiple cores That's exactly the myth I'm trying to address. It's true that in some circumstances the GIL prevents threads from running in multiple cores, but not all. The GIL would definitely have been released during the file I/O calls in your program. The GIL could well have been released during hashing too, depending on what hashing library you were using, which would have enabled true thread-based parallelism. For example, the built in Python hashlib says [1]: > Note: For better multithreading performance, the Python GIL is released for data larger than 2047 bytes at object creation or on update. On the other hand, I'm absolutely not saying that nobody should use multiple processes. For some applications, it's no more complex to use the multiprocessing module than to use threads and not much overhead to pass the data between processes. In that case, it's nice because you just don't even need to worry about whether the GIL is going to play a role, which can be a pain since sometimes there's no documentation saying whether specific functions release the GIL or not. All I'm saying is we should always make clear that to clear to everyone that threads are an option that the GIL doesn't (always) prevent. > Also, the featured video is from 2015. Maybe it's just outdated information? Another note on the hashlib page said that the above feature was added in Python 3.1 (released in 2009). That's just about that particular module. I found documentation from Python 1.5 (released in 1998) that describes how C extensions can release the GIL [2] [1] https://docs.python.org/3/library/hashlib.html https://docs.python.org/3/library/hashlib.html [2] https://docs.python.org/release/1.5/api/node43.html https://docs.python.org/release/1.5/api/node43.html
- joshgev 6y ago

theelous3 6y ago

It's a python conference demonstrating achieving goals in python. Basically saying "eh use C for cpu bound work" isn't helpful. I think you missed the point a bit, which was purely to show co-operative yielding. The fact that some libs use c and release the gil really doesn't matter. This is another way. The real world usecase, as with basically all python, is io.

quietbritishjim 6y ago
I'm not saying write your whole program in C. I'm saying, at worst, write a tiny corner of it in C so that you don't have to restructure the rest of your application - overall it would be simpler, even for most mostly-Python devs. Or, much more likely, use a library like numpy which you were probably going to do anyway. I regularly use numpy and don't find myself thinking "I wish didn't have to write all this C code...". (I must admit, an unfortunate consequence of this is that I've interviewed a few junior candidates that were so used to writing vectorised code that they seemed to be terrified of using a for loop!) I admit I didn't watch the whole of the talk. (Personally I much prefer learning from text than video, and I was put off by the GIL bit anyway.) From the parts I saw, it seemed like the GIL issue was critical for motivating everything he did afterwards, but perhaps there were other reasons that became clear later. In that case, he could have avoided mentioning the GIL altogether (given that he ended up being so misleading about it). > The real world usecase, as with basically all python, is io. That is just straight up not true. Yes, IO is a valid use case of Python but it definitely isn't "basically all Python". I'm sure the vast majority of data scientists use Python, and for most of them their workloads are almost entirely computational (using deep learning libraries with C/C++/CUDA backends).
- toyg 6y ago
  I think we have to consider that this talk is from 2015, when datascience was a much smaller part of the python ecosystem. I have used python since 2003 for various things and never once I’ve touched numpy, but I bet today 1 in 3 python devs use it probably every day. I understand your point of view and I understand the speaker’s own too - his is just more old-school. We’ve been able to sidestep the GIL in many ways since forever (for example with alternative implementations like Jython, IronPython, PyPy etc), and I think everyone has known that for a long time, but here Dave just wanted to show what the actual problem is, beyond the buzzwords (it had become a bit of a myth in the early ‘10s, when Go suddenly exploded in the sysadmin/backend niche partially because “it can do parallelism better”).
  - quietbritishjim 6y ago
    About the fact this talk is from 2015: The GIL is released by many libraries that aren't about data science, so was and is relevant to many other types of application. In the Python standard library, for example, zipfile (which can operate on files or in-memory buffers) and hashlib come to mind. As I mentioned in another comment, hashlib calls have released the GIL at least as far back as 2009, well before this talk, and releasing the GIL was possible all the way back in 1998. Many (most?) Python libraries are implemented in C, and almost all of those that are will release the GIL. About appreciating both points of view: I'm sorry if I'm making it sound like I think Python developers should always use threads and never multiprocessing. I don't believe that. And I don't believe the speaker should have reformulated their whole talk to use threads instead of processes. Just that it's such a pervasive myth that the GIL prevents all thread-based parallelism that they should have been careful to avoid reinforcing that. The speaker showed an example where the GIL actually did prevent parallelism; all I wanted was for them to add verbally that, by the way, if you were calling almost any CPU-bound library function rather than one you wrote yourself then this wouldn't be a problem.

znpy 6y ago

> It didn't work with his toy example, calling the Fibonacci function, because it's pure Python. maybe I do want to write pure python AND have multi-threaded applications ?

Geminidog 6y ago
One use case is IO bound applications. Similar to NodeJS which can handle 10K concurrent IO calls on a single thread, python releases the GIL on IO. So you can write pure python and have multi threading and have all the benefits of "parallelism" depending on whether or not python releases the GIL which it does with certain libraries and on IO. For all else use multiprocessing.

nurettin 6y ago

> GIL prevents you from using threads to achieve concurrency I didn't watch, but nothing in python prevents concurrency. In Python, a lot of things prevent parallelism. Threads/async are ways of achieving concurrency in python. Parallelism is achieved using multiprocessing. It is inefficient, but it works.

quietbritishjim 6y ago
Oops, you're right, I was talking about parallelism, not just concurrency. Thanks for the correction. > Parallelism is achieved using multiprocessing. As I said in my previous comment (but used the wrong term), threads are a perfectly reasonable way of getting parallelism in Python, so long so long your application uses C-based libraries like numpy (or even some built-in modules like zipfile). In my experience, the vast majority of Python programs that might need parallelism fall into that category (or are IO bound) anyway.
Geminidog 6y ago
Also from certain perspectives parallelism is achieved with threads during IO. So if 20 threads are making blocking IO calls, all 20 threads can make progress on those IO calls in parallel while having an additional parallel thread doing compute operations yielding a total of 21 threads executing in parallel.
- nurettin 6y ago
  We use "parallel" explicitly to mean "running on multiple cores scheduled by the kernel".
  - Geminidog 6y ago
    Yes, but "parallel" is also used in CS to specify multiple things happening at the same time including IO. https://www.wikiwand.com/en/Parallel_computing https://www.wikiwand.com/en/Parallel_computing Quote: "Parallel computing is a type of computation where many calculations or the execution of processes are carried out simultaneously." https://www.wikiwand.com/en/Parallel_communication https://www.wikiwand.com/en/Parallel_communication Quote: "In data transmission, parallel communication is a method of conveying multiple binary digits (bits) simultaneously. It contrasts with serial communication, which conveys only a single bit at a time; this distinction is one way of characterizing a communications link."

huseyinkeles 6y ago

For the curious, GIL = Global Interpreter Lock

jacobwilliamroy 6y ago

I remember using the multiprocessing library to speed up some file I/O. Basically I found out that I could spawn a separate process for each physical storage interface, and I was able to hash like 7 TiB of data in a little over 7 hours. My understanding was that threads are subject to GIL and thus cannot run on multiple cores, whereas processes don't have that same restriction, so I needed to use the multiprocessing library, NOT the multithreading library to parallelize my I/O. Also, the featured video is from 2015. Maybe it's just outdated information?

quietbritishjim 6y ago
> My understanding was that threads are subject to GIL and thus cannot run on multiple cores That's exactly the myth I'm trying to address. It's true that in some circumstances the GIL prevents threads from running in multiple cores, but not all. The GIL would definitely have been released during the file I/O calls in your program. The GIL could well have been released during hashing too, depending on what hashing library you were using, which would have enabled true thread-based parallelism. For example, the built in Python hashlib says [1]: > Note: For better multithreading performance, the Python GIL is released for data larger than 2047 bytes at object creation or on update. On the other hand, I'm absolutely not saying that nobody should use multiple processes. For some applications, it's no more complex to use the multiprocessing module than to use threads and not much overhead to pass the data between processes. In that case, it's nice because you just don't even need to worry about whether the GIL is going to play a role, which can be a pain since sometimes there's no documentation saying whether specific functions release the GIL or not. All I'm saying is we should always make clear that to clear to everyone that threads are an option that the GIL doesn't (always) prevent. > Also, the featured video is from 2015. Maybe it's just outdated information? Another note on the hashlib page said that the above feature was added in Python 3.1 (released in 2009). That's just about that particular module. I found documentation from Python 1.5 (released in 1998) that describes how C extensions can release the GIL [2] [1] https://docs.python.org/3/library/hashlib.html https://docs.python.org/3/library/hashlib.html [2] https://docs.python.org/release/1.5/api/node43.html https://docs.python.org/release/1.5/api/node43.html
joshgev 6y ago