3 ms·

How is the GIL different from atomics in other languages? There are many cases where atomics are useful. One example would be incrementing a counter for statis

How is the GIL different from atomics in other languages? There are many cases where atomics are useful. One example would be incrementing a counter for statistics purposes. If the counter is atomic, and the reader of the value is ok with a slightly out of date value, it's fine. If code is doing this in GIL Python, it's working now, and will break after the GIL is removed.

dragonwriter 5y ago
> How is the GIL different from atomics in other languages? Atomics are isolated to where they are needed, the GIL is global though C (but not Python) code can release it.
- Thorrez 5y ago
  My question was about from a program correctness standpoint, not an efficiency standpoint. But it does look there is a difference from a correctness standpoint, as other comments address.
detaro 5y ago
What atomic counter increment exists currently and would be broken by this proposal?
- Thorrez 5y ago
  Hmm, so even though an increment on a loaded int is atomic, loading it and storing it aren't. https://stackoverflow.com/a/1717514 https://stackoverflow.com/a/1717514 The Python documentation seems misleading to me on this: > In theory, this means an exact accounting requires an exact understanding of the PVM bytecode implementation. In practice, it means that operations on shared variables of built-in data types (ints, lists, dicts, etc) that “look atomic” really are. count = 0 def inc(): count += 1 sure "looks atomic" to me, so according to the documentation should be, but isn't. On the other hand, I think you could build a horribly inefficient actual atomic counter with count = [] def inc(): count.append(None) def get_count(): return len(count) I think it's quite likely there's correct and non-horrible code relying on list append and length being atomic. Although it sounds like this might continue working without the GIL: > A lot of work has gone into the list and dict implementations to make them thread-safe. And so on. So removing the GIL might not be a problem.
  - detaro 5y ago
    Indeed, the proposal explicitly covers maintaining existing guarantees, which is why I was confused that so many people just assumed it would break them.
fulafel 5y ago
Numbers are immutable and incrementing one eg in an attribute is actuay many byte code ops. So doesn't work even currently unless you are fine with losing updates. But a version of this question using another example (eg using a list as a queue) is interesting.
- Thorrez 5y ago
  Hah, we both came to the same conclusion at the same time. Regarding list, it sounds like it might actually keep working atomically without the GIL: > A lot of work has gone into the list and dict implementations to make them thread-safe. And so on.
  - fulafel 5y ago
    Yep. So in a final answer to the original question of backwards bug-compatibility (https://news.ycombinator.com/item?id=28897534 https://news.ycombinator.com/item?id=28897534), it seems that it will be retained under the current proposal.
dkersten 5y ago
> If the counter is atomic if I know you came to the same conclusion in another comments, but here's a look at it using the `dis` module: a += 1 turns into LOAD_FAST 1 (loads b) LOAD_CONST 1 (loads the constant 1) INPLACE_ADD (perform the addition) STORE_FAST 1 (store back into b) So if the interpreter switches threads between LOAD_CONST and STORE_FAST (so either before or after the INPLACE_ADD), you could clobber the value another thread wrote to `b`. From your other comment: > so even though an increment on a loaded int is atomic, loading it and storing it aren't Its always the loads and stores that are the problem. But that's the problem with relying on the GIL: it has lock in the name, but it does not protect you, unless you understand the internals and know what you're doing. It protects the interpreter. This isn't much different from programming in other languages without a GIL: if you understand the internals what you're doing you may or may not need locks, because you will know what is and isn't atomic (and even when things are atomic, its still difficult to write thread-safe code! lock-free algorithms are much harder than using mutexes). Thread safe code requires thinking hard about your code, the GIL does not protect you from that.
- dkersten 5y ago
  Too late to edit correction: between LOAD_FAST (not LOAD_CONST) and STORE_FAST

dragonwriter 5y ago

> How is the GIL different from atomics in other languages? Atomics are isolated to where they are needed, the GIL is global though C (but not Python) code can release it.

Thorrez 5y ago
My question was about from a program correctness standpoint, not an efficiency standpoint. But it does look there is a difference from a correctness standpoint, as other comments address.

detaro 5y ago

What atomic counter increment exists currently and would be broken by this proposal?

Thorrez 5y ago
Hmm, so even though an increment on a loaded int is atomic, loading it and storing it aren't. https://stackoverflow.com/a/1717514 https://stackoverflow.com/a/1717514 The Python documentation seems misleading to me on this: > In theory, this means an exact accounting requires an exact understanding of the PVM bytecode implementation. In practice, it means that operations on shared variables of built-in data types (ints, lists, dicts, etc) that “look atomic” really are. count = 0 def inc(): count += 1 sure "looks atomic" to me, so according to the documentation should be, but isn't. On the other hand, I think you could build a horribly inefficient actual atomic counter with count = [] def inc(): count.append(None) def get_count(): return len(count) I think it's quite likely there's correct and non-horrible code relying on list append and length being atomic. Although it sounds like this might continue working without the GIL: > A lot of work has gone into the list and dict implementations to make them thread-safe. And so on. So removing the GIL might not be a problem.
- detaro 5y ago
  Indeed, the proposal explicitly covers maintaining existing guarantees, which is why I was confused that so many people just assumed it would break them.

fulafel 5y ago

Numbers are immutable and incrementing one eg in an attribute is actuay many byte code ops. So doesn't work even currently unless you are fine with losing updates. But a version of this question using another example (eg using a list as a queue) is interesting.

Thorrez 5y ago
Hah, we both came to the same conclusion at the same time. Regarding list, it sounds like it might actually keep working atomically without the GIL: > A lot of work has gone into the list and dict implementations to make them thread-safe. And so on.
- fulafel 5y ago
  Yep. So in a final answer to the original question of backwards bug-compatibility (https://news.ycombinator.com/item?id=28897534 https://news.ycombinator.com/item?id=28897534), it seems that it will be retained under the current proposal.

dkersten 5y ago

> If the counter is atomic if I know you came to the same conclusion in another comments, but here's a look at it using the `dis` module: a += 1 turns into LOAD_FAST 1 (loads b) LOAD_CONST 1 (loads the constant 1) INPLACE_ADD (perform the addition) STORE_FAST 1 (store back into b) So if the interpreter switches threads between LOAD_CONST and STORE_FAST (so either before or after the INPLACE_ADD), you could clobber the value another thread wrote to `b`. From your other comment: > so even though an increment on a loaded int is atomic, loading it and storing it aren't Its always the loads and stores that are the problem. But that's the problem with relying on the GIL: it has lock in the name, but it does not protect you, unless you understand the internals and know what you're doing. It protects the interpreter. This isn't much different from programming in other languages without a GIL: if you understand the internals what you're doing you may or may not need locks, because you will know what is and isn't atomic (and even when things are atomic, its still difficult to write thread-safe code! lock-free algorithms are much harder than using mutexes). Thread safe code requires thinking hard about your code, the GIL does not protect you from that.

dkersten 5y ago
Too late to edit correction: between LOAD_FAST (not LOAD_CONST) and STORE_FAST