3 ms·

It seems like in every python discussion I hear people complain about the GIL. I’m happy people are working on removing the GIL, but As a professional python d

It seems like in every python discussion I hear people complain about the GIL. I’m happy people are working on removing the GIL, but As a professional python dev for about 5 years now I have literally never had a problem where the GIL was a limiter. Although I just make web apps so maybe I’m not the target audience.

BorgHunter 3y ago
Conversely, I've worked on backend, data processing-type applications for most of my career, much of it in Java but some (especially recently) in Python, and the GIL is a huge limiting factor for writing efficient, readable Python code. I've had to write very annoying Python code using the multiprocessing library to get around the GIL, and ultimately it works, but it's ugly and clunky and overall just a pain. And remember, I've written a lot of Java code, so I have a high tolerance for pain! But the JVM's concurrency abstractions are actually kind of a joy to use, even if Java the language isn't. Python is the opposite, so if they can shed the GIL and make multithreading viable in Python without forking new processes, that would be a huge win.
- Gabrys1 3y ago
  I think multiprocessing is quite sensible in Python (comparing to async for an example)
  - toxik 3y ago
    multiprocessing is almost never a good idea in my experience.
svara 3y ago
It's just about what kind of code you write. If you write a lot of code that parallelizes over data you will hurt all the time because of the GIL. If you have worker processes that do something on the CPU, and the results need to be collected and processed further in some other process, you now need to pickle the data to copy it around. That can get really slow. I understand a lot of python users don't do that kind of thing, but it's a real problem. I'm happy that the python community seems to be slowly beginning to take this seriously after decades of just claiming the GIL isn't a real issue.
- quietbritishjim 3y ago
  > If you write a lot of code that parallelizes over data [and that parallelization can't happen by vectorizing your operations in a C extension module that releases the GIL, like numpy] you will hurt all the time because of the GIL. I've added some text for clarity
klyrs 3y ago
In my rapidly-approaching-20 years of python development, I have butted heads with the GIL countless times. Are your web apps single-threaded? Have you looked at scaling your service? Or do you use a web framework that handles that for you behind the scenes?
bvirb 3y ago
I’ve had the same experience mostly writing web apps in Ruby, which locks similarly to Python. Multi-threading is always needed to saturate IO, network, etc, never CPU, so GIL isn’t an issue. I always wondered what people were doing that ran into issues. With Python being the language of choice for ML workloads I guess it’s more common to have the CPU be a bottleneck. It seems cool they’re making an option to turn it off for those use cases. It seems like Python could maintain a GIL compatibility option to preserve the current/old behavior for legacy code.

BorgHunter 3y ago

Conversely, I've worked on backend, data processing-type applications for most of my career, much of it in Java but some (especially recently) in Python, and the GIL is a huge limiting factor for writing efficient, readable Python code. I've had to write very annoying Python code using the multiprocessing library to get around the GIL, and ultimately it works, but it's ugly and clunky and overall just a pain. And remember, I've written a lot of Java code, so I have a high tolerance for pain! But the JVM's concurrency abstractions are actually kind of a joy to use, even if Java the language isn't. Python is the opposite, so if they can shed the GIL and make multithreading viable in Python without forking new processes, that would be a huge win.

Gabrys1 3y ago
I think multiprocessing is quite sensible in Python (comparing to async for an example)
- toxik 3y ago
  multiprocessing is almost never a good idea in my experience.

svara 3y ago

It's just about what kind of code you write. If you write a lot of code that parallelizes over data you will hurt all the time because of the GIL. If you have worker processes that do something on the CPU, and the results need to be collected and processed further in some other process, you now need to pickle the data to copy it around. That can get really slow. I understand a lot of python users don't do that kind of thing, but it's a real problem. I'm happy that the python community seems to be slowly beginning to take this seriously after decades of just claiming the GIL isn't a real issue.

quietbritishjim 3y ago
> If you write a lot of code that parallelizes over data [and that parallelization can't happen by vectorizing your operations in a C extension module that releases the GIL, like numpy] you will hurt all the time because of the GIL. I've added some text for clarity

klyrs 3y ago

In my rapidly-approaching-20 years of python development, I have butted heads with the GIL countless times. Are your web apps single-threaded? Have you looked at scaling your service? Or do you use a web framework that handles that for you behind the scenes?

bvirb 3y ago

I’ve had the same experience mostly writing web apps in Ruby, which locks similarly to Python. Multi-threading is always needed to saturate IO, network, etc, never CPU, so GIL isn’t an issue. I always wondered what people were doing that ran into issues. With Python being the language of choice for ML workloads I guess it’s more common to have the CPU be a bottleneck. It seems cool they’re making an option to turn it off for those use cases. It seems like Python could maintain a GIL compatibility option to preserve the current/old behavior for legacy code.