3 ms·

It's a python conference demonstrating achieving goals in python. Basically saying "eh use C for cpu bound work" isn't helpful. I think you missed the point a

It's a python conference demonstrating achieving goals in python. Basically saying "eh use C for cpu bound work" isn't helpful. I think you missed the point a bit, which was purely to show co-operative yielding. The fact that some libs use c and release the gil really doesn't matter. This is another way. The real world usecase, as with basically all python, is io.

quietbritishjim 6y ago
I'm not saying write your whole program in C. I'm saying, at worst, write a tiny corner of it in C so that you don't have to restructure the rest of your application - overall it would be simpler, even for most mostly-Python devs. Or, much more likely, use a library like numpy which you were probably going to do anyway. I regularly use numpy and don't find myself thinking "I wish didn't have to write all this C code...". (I must admit, an unfortunate consequence of this is that I've interviewed a few junior candidates that were so used to writing vectorised code that they seemed to be terrified of using a for loop!) I admit I didn't watch the whole of the talk. (Personally I much prefer learning from text than video, and I was put off by the GIL bit anyway.) From the parts I saw, it seemed like the GIL issue was critical for motivating everything he did afterwards, but perhaps there were other reasons that became clear later. In that case, he could have avoided mentioning the GIL altogether (given that he ended up being so misleading about it). > The real world usecase, as with basically all python, is io. That is just straight up not true. Yes, IO is a valid use case of Python but it definitely isn't "basically all Python". I'm sure the vast majority of data scientists use Python, and for most of them their workloads are almost entirely computational (using deep learning libraries with C/C++/CUDA backends).
- toyg 6y ago
  I think we have to consider that this talk is from 2015, when datascience was a much smaller part of the python ecosystem. I have used python since 2003 for various things and never once I’ve touched numpy, but I bet today 1 in 3 python devs use it probably every day. I understand your point of view and I understand the speaker’s own too - his is just more old-school. We’ve been able to sidestep the GIL in many ways since forever (for example with alternative implementations like Jython, IronPython, PyPy etc), and I think everyone has known that for a long time, but here Dave just wanted to show what the actual problem is, beyond the buzzwords (it had become a bit of a myth in the early ‘10s, when Go suddenly exploded in the sysadmin/backend niche partially because “it can do parallelism better”).
  - quietbritishjim 6y ago
    About the fact this talk is from 2015: The GIL is released by many libraries that aren't about data science, so was and is relevant to many other types of application. In the Python standard library, for example, zipfile (which can operate on files or in-memory buffers) and hashlib come to mind. As I mentioned in another comment, hashlib calls have released the GIL at least as far back as 2009, well before this talk, and releasing the GIL was possible all the way back in 1998. Many (most?) Python libraries are implemented in C, and almost all of those that are will release the GIL. About appreciating both points of view: I'm sorry if I'm making it sound like I think Python developers should always use threads and never multiprocessing. I don't believe that. And I don't believe the speaker should have reformulated their whole talk to use threads instead of processes. Just that it's such a pervasive myth that the GIL prevents all thread-based parallelism that they should have been careful to avoid reinforcing that. The speaker showed an example where the GIL actually did prevent parallelism; all I wanted was for them to add verbally that, by the way, if you were calling almost any CPU-bound library function rather than one you wrote yourself then this wouldn't be a problem.

quietbritishjim 6y ago

I'm not saying write your whole program in C. I'm saying, at worst, write a tiny corner of it in C so that you don't have to restructure the rest of your application - overall it would be simpler, even for most mostly-Python devs. Or, much more likely, use a library like numpy which you were probably going to do anyway. I regularly use numpy and don't find myself thinking "I wish didn't have to write all this C code...". (I must admit, an unfortunate consequence of this is that I've interviewed a few junior candidates that were so used to writing vectorised code that they seemed to be terrified of using a for loop!) I admit I didn't watch the whole of the talk. (Personally I much prefer learning from text than video, and I was put off by the GIL bit anyway.) From the parts I saw, it seemed like the GIL issue was critical for motivating everything he did afterwards, but perhaps there were other reasons that became clear later. In that case, he could have avoided mentioning the GIL altogether (given that he ended up being so misleading about it). > The real world usecase, as with basically all python, is io. That is just straight up not true. Yes, IO is a valid use case of Python but it definitely isn't "basically all Python". I'm sure the vast majority of data scientists use Python, and for most of them their workloads are almost entirely computational (using deep learning libraries with C/C++/CUDA backends).

toyg 6y ago
I think we have to consider that this talk is from 2015, when datascience was a much smaller part of the python ecosystem. I have used python since 2003 for various things and never once I’ve touched numpy, but I bet today 1 in 3 python devs use it probably every day. I understand your point of view and I understand the speaker’s own too - his is just more old-school. We’ve been able to sidestep the GIL in many ways since forever (for example with alternative implementations like Jython, IronPython, PyPy etc), and I think everyone has known that for a long time, but here Dave just wanted to show what the actual problem is, beyond the buzzwords (it had become a bit of a myth in the early ‘10s, when Go suddenly exploded in the sysadmin/backend niche partially because “it can do parallelism better”).
- quietbritishjim 6y ago
  About the fact this talk is from 2015: The GIL is released by many libraries that aren't about data science, so was and is relevant to many other types of application. In the Python standard library, for example, zipfile (which can operate on files or in-memory buffers) and hashlib come to mind. As I mentioned in another comment, hashlib calls have released the GIL at least as far back as 2009, well before this talk, and releasing the GIL was possible all the way back in 1998. Many (most?) Python libraries are implemented in C, and almost all of those that are will release the GIL. About appreciating both points of view: I'm sorry if I'm making it sound like I think Python developers should always use threads and never multiprocessing. I don't believe that. And I don't believe the speaker should have reformulated their whole talk to use threads instead of processes. Just that it's such a pervasive myth that the GIL prevents all thread-based parallelism that they should have been careful to avoid reinforcing that. The speaker showed an example where the GIL actually did prevent parallelism; all I wanted was for them to add verbally that, by the way, if you were calling almost any CPU-bound library function rather than one you wrote yourself then this wouldn't be a problem.