2 ms·
It's just about what kind of code you write. If you write a lot of code that parallelizes over data you will hurt all the time because of the GIL. If you have
by svara 3y ago
It's just about what kind of code you write.
If you write a lot of code that parallelizes over data you will hurt all the time because of the GIL.
If you have worker processes that do something on the CPU, and the results need to be collected and processed further in some other process, you now need to pickle the data to copy it around. That can get really slow.
I understand a lot of python users don't do that kind of thing, but it's a real problem. I'm happy that the python community seems to be slowly beginning to take this seriously after decades of just claiming the GIL isn't a real issue.
- quietbritishjim 3y ago> If you write a lot of code that parallelizes over data [and that parallelization can't happen by vectorizing your operations in a C extension module that releases the GIL, like numpy] you will hurt all the time because of the GIL. I've added some text for clarity