8 ms·

I dont know about others, but I really enjoy content about the Python GIL. Its a fascinatingly complex problem.

lucb1e 5y ago
For a minute I thought I finally found someone else who likes the GIL, but then you said content about. Programs that just divide up work across processes are much easier to write without introducing obscure bugs due to the lack of atomicity. I'm definitely excited for a GIL-less python, even if it's a rare scenario where it makes sense to try to do performant code in python in the first place rather than offloading a few lines to another language to be fast, but I am a bit afraid that people (particularly beginner programmers) will grab this with too many hands. Having also seen recommendations for this-or-that threading method going around in other languages, threads are recommended really much more often than where it makes sense and beginners won't have a comparative experience yet of writing multi-process code instead. That said, I am also always interested in GIL-related content like this! Loved the article.
- globular-toast 5y ago
  > Programs that just divide up work across processes are much easier to write without introducing obscure bugs due to the lack of atomicity. You often don't even need to do this yourself. GNU parallel is the way to go for dividing work up amongst CPU cores. Why reinvent the wheel? I agree with you that threads are talked about way more than they should be. It's like all programmers learn this one simple rule: to be fast you have to be multi-threaded. It's really not the case. There is also massive confusion amongst programmers on the difference between concurrency and parallelism. I sometimes ask applicants to describe the difference and few can. Python is fine at concurrency if that's all you want to do.
  - adwn 5y ago
    > GNU parallel is the way to go for dividing work up amongst CPU cores. Why reinvent the wheel? Because most problems are not the embarrassingly parallel kind suitable for use with GNU parallel. For example, any problems that require some communication between the individual tasks.
  - sigotirandolas 5y ago
    I don't think the parent was proposing reinventing the wheel, Python has straightforward process parallelism support in the 'multiprocessing' library and for Python that's generally a better idea than GNU Parallel, IMO.
    - globular-toast 5y ago
      The advantage of GNU parallel is it's a standard tool that works for any non-parallel process. This has all the usual advantages of following the Unix principle.
      - adwn 5y ago
        > that works for any non-parallel process No, it doesn't. Only for those processes, where you can trivially split the input and concatenate the outputs. Try using GNU parallel to sort a list of numbers, or to compute their prefix sum – it's not possible, and those are even simpler use cases than most of what you'll encounter in practice.
        globular-toast 5y ago
        Oh come on. It should be obvious that I'm talking about the processes that can be split up in that way. Those problems are so common that someone literally wrote GNU parallel to solve them.
        adwn 5y ago
        > I'm talking about the processes that can be split up in that way No, you weren't. You said: "[...] GNU parallel [...] works for any non-parallel process" (emphasis mine) > Those problems are so common that someone literally wrote GNU parallel to solve them. As part of my job I'm writing multi-threaded, parallel programs all the time, and in those years only a single problem would have been feasible to parallelize with GNU parallel; but since I was using Rust, it was trivial to do the parallelization right there in my code without having to resort an outer script/binary that calls GNU parallel on my program.
  - lucb1e 5y ago
    That's actually what I'm doing a lot of the time. Or even just bash: for i in {1..threadcount}; do pypy my.py $i/$threadcount & done;
    - da39a3ee 5y ago
      We’re not talking about how to write scripts to run on your laptop, we’re talking about production systems.
  - da39a3ee 5y ago
    > GNU parallel is the way to go for dividing work up amongst CPU cores. Why reinvent the wheel? We’re not talking about writing scripts to run on your laptop. We’re talking about code written for production applications. Deploying GNU parallel to production nodes / containers would be a major change to production systems that may not be feasible and even if it is would come with a high cost in terms of added complexity, maintenance, and production troubleshooting.
    - globular-toast 5y ago
      I used to use GNU parallel to run big data tasks on supercomputers. There's nothing special about "production". It's all just computers.
- omnicognate 5y ago
  I like the GIL and would prefer it not be removed, regardless of any impact on performance. It's a powerful assumption for both python code and extensions to be able to make that only one thread will be executing in the interpreter at a time. Knowing it, you can do a lot of things with a much lower cognitive burden. I tend to think through a problem initially in a non-concurrent way and then think "ok, but there's concurrency to think of too, so how is this all affected?". With the GIL you usually don't need to go far into that if at all, and that's useful. That doesn't mean I don't approve of concurrency. I use lots of languages, and a need for proper concurrency is one of the major reasons I might avoid python as a tool for a particular task. But there are lots of languages, and python became the popular tool it is with the GIL. If python were to embrace concurrency in the interpreter its nature would change, and I think my toolbox would lose something valuable. That said, I'm not worried. CPython exists in a cloud of extensions developed against its C API, and these are heavily reliant on the GIL. Refcounting isn't even the start of the problems you would need to solve in order to remove it and not have everyone just move to a fork that still has it. I'll be astonished if anyone manages to pull it off.
  - arc-in-space 5y ago
    I'm confused. The article mentions that Python's data structures would be made thread-safe, which sounds like there is no intention to change concurrency semantics. How does the presence of the GIL help? Can you elaborate what would be lost by removing it?
    - masklinn 5y ago
      A C extension which does not release the gil does not have to worry about concurrent execution and can trivially work in global space.
    - nitred 5y ago
      Can't speak for the OP but here's what I think they may have meant and my intuitions agree with them as well. In python I mostly use threads for non-CPU intensive tasks like IO or timers or event handling and I rarely implement these myself. However for CPU intensive tasks, there's rarely any point to having 10 threads in Python because they all have to run on one core because of the GIL. Therefore when working with python the GIL indirectly lets me never have to think about writing concurrent CPU intensive code AND I know others wouldn't do it too because it wouldn't make sense most of the time. This is leads to a significant reduction in cognitive load when thinking about solving a problem in Python. And it isn't about thread-safety but that in concurrent code I have to worry how other threads might change the values of shared variables and global variables of which there are many. Instead the GIL lets me think synchronously, write synchronous code and when I needed to parallelize my work I use multiprocessing where my synchronous code and assumptions work perfectly because each process has its own GIL and I explicitly pass shared variables to each of the processes. I don't know enough to comment on the good or the bad of GIL. Just my two cents about how I think about code with Python in it's current state.
solarmist 5y ago
Yup. Me too, but I'm not sad to see it go.

For a minute I thought I finally found someone else who likes the GIL, but then you said content about. Programs that just divide up work across processes are much easier to write without introducing obscure bugs due to the lack of atomicity. I'm definitely excited for a GIL-less python, even if it's a rare scenario where it makes sense to try to do performant code in python in the first place rather than offloading a few lines to another language to be fast, but I am a bit afraid that people (particularly beginner programmers) will grab this with too many hands. Having also seen recommendations for this-or-that threading method going around in other languages, threads are recommended really much more often than where it makes sense and beginners won't have a comparative experience yet of writing multi-process code instead. That said, I am also always interested in GIL-related content like this! Loved the article.

globular-toast 5y ago
> Programs that just divide up work across processes are much easier to write without introducing obscure bugs due to the lack of atomicity. You often don't even need to do this yourself. GNU parallel is the way to go for dividing work up amongst CPU cores. Why reinvent the wheel? I agree with you that threads are talked about way more than they should be. It's like all programmers learn this one simple rule: to be fast you have to be multi-threaded. It's really not the case. There is also massive confusion amongst programmers on the difference between concurrency and parallelism. I sometimes ask applicants to describe the difference and few can. Python is fine at concurrency if that's all you want to do.
- adwn 5y ago
  > GNU parallel is the way to go for dividing work up amongst CPU cores. Why reinvent the wheel? Because most problems are not the embarrassingly parallel kind suitable for use with GNU parallel. For example, any problems that require some communication between the individual tasks.
- sigotirandolas 5y ago
  I don't think the parent was proposing reinventing the wheel, Python has straightforward process parallelism support in the 'multiprocessing' library and for Python that's generally a better idea than GNU Parallel, IMO.
  - globular-toast 5y ago
    The advantage of GNU parallel is it's a standard tool that works for any non-parallel process. This has all the usual advantages of following the Unix principle.
    - adwn 5y ago
      > that works for any non-parallel process No, it doesn't. Only for those processes, where you can trivially split the input and concatenate the outputs. Try using GNU parallel to sort a list of numbers, or to compute their prefix sum – it's not possible, and those are even simpler use cases than most of what you'll encounter in practice.
      - globular-toast 5y ago
        Oh come on. It should be obvious that I'm talking about the processes that can be split up in that way. Those problems are so common that someone literally wrote GNU parallel to solve them.
        adwn 5y ago
        > I'm talking about the processes that can be split up in that way No, you weren't. You said: "[...] GNU parallel [...] works for any non-parallel process" (emphasis mine) > Those problems are so common that someone literally wrote GNU parallel to solve them. As part of my job I'm writing multi-threaded, parallel programs all the time, and in those years only a single problem would have been feasible to parallelize with GNU parallel; but since I was using Rust, it was trivial to do the parallelization right there in my code without having to resort an outer script/binary that calls GNU parallel on my program.
- lucb1e 5y ago
  That's actually what I'm doing a lot of the time. Or even just bash: for i in {1..threadcount}; do pypy my.py $i/$threadcount & done;
  - da39a3ee 5y ago
    We’re not talking about how to write scripts to run on your laptop, we’re talking about production systems.
- da39a3ee 5y ago
  > GNU parallel is the way to go for dividing work up amongst CPU cores. Why reinvent the wheel? We’re not talking about writing scripts to run on your laptop. We’re talking about code written for production applications. Deploying GNU parallel to production nodes / containers would be a major change to production systems that may not be feasible and even if it is would come with a high cost in terms of added complexity, maintenance, and production troubleshooting.
  - globular-toast 5y ago
    I used to use GNU parallel to run big data tasks on supercomputers. There's nothing special about "production". It's all just computers.
omnicognate 5y ago
I like the GIL and would prefer it not be removed, regardless of any impact on performance. It's a powerful assumption for both python code and extensions to be able to make that only one thread will be executing in the interpreter at a time. Knowing it, you can do a lot of things with a much lower cognitive burden. I tend to think through a problem initially in a non-concurrent way and then think "ok, but there's concurrency to think of too, so how is this all affected?". With the GIL you usually don't need to go far into that if at all, and that's useful. That doesn't mean I don't approve of concurrency. I use lots of languages, and a need for proper concurrency is one of the major reasons I might avoid python as a tool for a particular task. But there are lots of languages, and python became the popular tool it is with the GIL. If python were to embrace concurrency in the interpreter its nature would change, and I think my toolbox would lose something valuable. That said, I'm not worried. CPython exists in a cloud of extensions developed against its C API, and these are heavily reliant on the GIL. Refcounting isn't even the start of the problems you would need to solve in order to remove it and not have everyone just move to a fork that still has it. I'll be astonished if anyone manages to pull it off.
- arc-in-space 5y ago
  I'm confused. The article mentions that Python's data structures would be made thread-safe, which sounds like there is no intention to change concurrency semantics. How does the presence of the GIL help? Can you elaborate what would be lost by removing it?
  - masklinn 5y ago
    A C extension which does not release the gil does not have to worry about concurrent execution and can trivially work in global space.
  - nitred 5y ago
    Can't speak for the OP but here's what I think they may have meant and my intuitions agree with them as well. In python I mostly use threads for non-CPU intensive tasks like IO or timers or event handling and I rarely implement these myself. However for CPU intensive tasks, there's rarely any point to having 10 threads in Python because they all have to run on one core because of the GIL. Therefore when working with python the GIL indirectly lets me never have to think about writing concurrent CPU intensive code AND I know others wouldn't do it too because it wouldn't make sense most of the time. This is leads to a significant reduction in cognitive load when thinking about solving a problem in Python. And it isn't about thread-safety but that in concurrent code I have to worry how other threads might change the values of shared variables and global variables of which there are many. Instead the GIL lets me think synchronously, write synchronous code and when I needed to parallelize my work I use multiprocessing where my synchronous code and assumptions work perfectly because each process has its own GIL and I explicitly pass shared variables to each of the processes. I don't know enough to comment on the good or the bad of GIL. Just my two cents about how I think about code with Python in it's current state.

solarmist 5y ago

Yup. Me too, but I'm not sad to see it go.