7 ms·

Then why didn't they remove GIL? Given the end of Moore's law and Python's performance, this alone would have justified the discontinuity.

by cft 7y ago

fiedzia 7y ago
Because GIL is useful, preventing you from shooting yourself terribly in the foot among other benefits. Languages with better concurrency support (main reason for removing GIL) were designed from scratch for that purpose. You can't remove GIL and keep Python being Python everyone knows. And if you don't mind using another language, there are many choices.
- laurencerowe 7y ago
  I believe Jython offers the same level of protection against shooting yourself in the foot as CPython, yet it allows allows for parallel execution by relying on finer grained locking. The concurrency issues with threads are still present in CPython with the GIL as the interpreter may switch threads between any instruction.
kstrauser 7y ago
There've been various attempts at this over the years, but it's really orthogonal to changes to the language itself. Also, check out the SharedMemory objects[1] in 3.8. For my own use cases, this goes a long way toward making me no longer care about the GIL. I'd rather work on multiprocessing code than multithreaded code any time, and if you can easily share objects across processes, then the GIL isn't relevant to anything I'm working on. [1] https://docs.python.org/3/library/multiprocessing.shared_memory.html#multiprocessing.shared_memory.SharedMemory https://docs.python.org/3/library/multiprocessing.shared_mem...
dickeytk 7y ago
I would argue the GIL is a feature put in place to make it easier to write parallel code. If you want to do this in a more performant way you should be looking at a different language. People talk about the GIL as if it’s just some drawback that could be removed and everything would be faster. There is a very good reason why it’s there.
- masklinn 7y ago
  > I would argue the GIL is a feature put in place to make it easier to write parallel code. It doesn't do that though. The GIL protects interpreter internals. It doesn't make Python code thread-safe, although it does make various operations atomic: * native calls (which doesn't release the GIL) * individual bytecode instructions (which don't call into more python code) The latter can make it seem like it magically protects your code, but even a simple method call is at least 3 instructions (LOAD_FAST, LOAD_METHOD and CALL_METHOD). Each instruction is atomic and the method call will be if it's a native call (e.g. dict.get) but that's it, the entire call itself is not atomic and you could thread-switch between LOAD_FAST and LOAD_METHOD, or LOAD_METHOD and CALL_METHOD.
  - dickeytk 7y ago
    You just explained a bunch of implementation details but at the end of the day my point remains: having the GIL makes it easier to write parallel code. I didn't say it magically makes everything threadsafe.
    - masklinn 7y ago
      > having the GIL makes it easier to write parallel code. It does not. What it does is create a bigger trap, because the code looks to be working in more testing situations, and will be more difficult to debug when it starts failing.
- pavanky 7y ago
  Isnt the GIL part of the implementation rather than the language?
  - dickeytk 7y ago
    In the same exact way the garbage collector is. You never touch it but the fact it's there makes it easier to write code.
- laurencerowe 7y ago
dashwav 7y ago
Because the GIL is only a problem in specific sets of problems that the majority of python users won't run into. With the addition of asyncio etc, the negative of GIL is usually pretty restricted to CPU-heavy, blocking services which the core team decided was a very small subset of users. So removing the GIL would require an extremely heavy rewrite of the way python works at a fundamental basis, and it would break a lot of the "promises" that python affords in regards to memory management and safety. As an aside, there is currently a PEP that will allow for better concurrency __without__ removing the GIL, [PEP554](https://www.python.org/dev/peps/pep-0554/ https://www.python.org/dev/peps/pep-0554/) and a larger project overall for multi-core support, which is (hopefully) being included in 3.9 [Multi-core](https://github.com/ericsnowcurrently/multi-core-python https://github.com/ericsnowcurrently/multi-core-python)
deleted 7y ago
[deleted]
Ralfp 7y ago
Removing GIL will break ecosystem in worse way than Python 2->3 ever did, as both interpreter and compiled libraries rely on it for thread safety. There are other paths for improving performance that Python is embrancing: asyncio provides considerable performance gains to IO bound apps, and memory-sharing between processes allows for relatively cheap sidestep for those who find threads too slow for their use-case.
the_why_of_y 7y ago
In addition to the points other people raised, "simply" removing the GIL by changing the current reference-counting GC to do atomic reference updates will also substantially reduce single threaded performance; there was a PyCon talk about that. https://www.youtube.com/watch?v=P3AyI_u66Bw https://www.youtube.com/watch?v=P3AyI_u66Bw See also Swift for another language whose implementation has performance issues caused by ubiquitous atomic reference counting.

Then why didn't they remove GIL? Given the end of Moore's law and Python's performance, this alone would have justified the discontinuity.

by cft 7y ago

Then why didn't they remove GIL? Given the end of Moore's law and Python's performance, this alone would have justified the discontinuity.

fiedzia 7y ago

Because GIL is useful, preventing you from shooting yourself terribly in the foot among other benefits. Languages with better concurrency support (main reason for removing GIL) were designed from scratch for that purpose. You can't remove GIL and keep Python being Python everyone knows. And if you don't mind using another language, there are many choices.

laurencerowe 7y ago
I believe Jython offers the same level of protection against shooting yourself in the foot as CPython, yet it allows allows for parallel execution by relying on finer grained locking. The concurrency issues with threads are still present in CPython with the GIL as the interpreter may switch threads between any instruction.

kstrauser 7y ago

There've been various attempts at this over the years, but it's really orthogonal to changes to the language itself. Also, check out the SharedMemory objects[1] in 3.8. For my own use cases, this goes a long way toward making me no longer care about the GIL. I'd rather work on multiprocessing code than multithreaded code any time, and if you can easily share objects across processes, then the GIL isn't relevant to anything I'm working on. [1] https://docs.python.org/3/library/multiprocessing.shared_memory.html#multiprocessing.shared_memory.SharedMemory https://docs.python.org/3/library/multiprocessing.shared_mem...

dickeytk 7y ago

I would argue the GIL is a feature put in place to make it easier to write parallel code. If you want to do this in a more performant way you should be looking at a different language. People talk about the GIL as if it’s just some drawback that could be removed and everything would be faster. There is a very good reason why it’s there.

masklinn 7y ago
> I would argue the GIL is a feature put in place to make it easier to write parallel code. It doesn't do that though. The GIL protects interpreter internals. It doesn't make Python code thread-safe, although it does make various operations atomic: * native calls (which doesn't release the GIL) * individual bytecode instructions (which don't call into more python code) The latter can make it seem like it magically protects your code, but even a simple method call is at least 3 instructions (LOAD_FAST, LOAD_METHOD and CALL_METHOD). Each instruction is atomic and the method call will be if it's a native call (e.g. dict.get) but that's it, the entire call itself is not atomic and you could thread-switch between LOAD_FAST and LOAD_METHOD, or LOAD_METHOD and CALL_METHOD.
- dickeytk 7y ago
  You just explained a bunch of implementation details but at the end of the day my point remains: having the GIL makes it easier to write parallel code. I didn't say it magically makes everything threadsafe.
  - masklinn 7y ago
    > having the GIL makes it easier to write parallel code. It does not. What it does is create a bigger trap, because the code looks to be working in more testing situations, and will be more difficult to debug when it starts failing.
pavanky 7y ago
Isnt the GIL part of the implementation rather than the language?
- dickeytk 7y ago
  In the same exact way the garbage collector is. You never touch it but the fact it's there makes it easier to write code.
laurencerowe 7y ago

dashwav 7y ago

Because the GIL is only a problem in specific sets of problems that the majority of python users won't run into. With the addition of asyncio etc, the negative of GIL is usually pretty restricted to CPU-heavy, blocking services which the core team decided was a very small subset of users. So removing the GIL would require an extremely heavy rewrite of the way python works at a fundamental basis, and it would break a lot of the "promises" that python affords in regards to memory management and safety. As an aside, there is currently a PEP that will allow for better concurrency __without__ removing the GIL, [PEP554](https://www.python.org/dev/peps/pep-0554/ https://www.python.org/dev/peps/pep-0554/) and a larger project overall for multi-core support, which is (hopefully) being included in 3.9 [Multi-core](https://github.com/ericsnowcurrently/multi-core-python https://github.com/ericsnowcurrently/multi-core-python)

deleted 7y ago

[deleted]

Ralfp 7y ago

Removing GIL will break ecosystem in worse way than Python 2->3 ever did, as both interpreter and compiled libraries rely on it for thread safety. There are other paths for improving performance that Python is embrancing: asyncio provides considerable performance gains to IO bound apps, and memory-sharing between processes allows for relatively cheap sidestep for those who find threads too slow for their use-case.

the_why_of_y 7y ago

In addition to the points other people raised, "simply" removing the GIL by changing the current reference-counting GC to do atomic reference updates will also substantially reduce single threaded performance; there was a PyCon talk about that. https://www.youtube.com/watch?v=P3AyI_u66Bw https://www.youtube.com/watch?v=P3AyI_u66Bw See also Swift for another language whose implementation has performance issues caused by ubiquitous atomic reference counting.