8 ms·

Code that assumes that something is going to be atomic because of the GIL (or any other implementation detail) is simply broken. If you need something to be ato

by dfox 3y ago

Code that assumes that something is going to be atomic because of the GIL (or any other implementation detail) is simply broken. If you need something to be atomic you should be explicit about that and use mutex or something.

bheadmaster 3y ago
True, but when a majority of ecosystem is relying on an implementation detail, that implementation detail becomes de-facto standard.
- darklion 3y ago
  Just because everybody does something the wrong way doesn’t somehow make it magically correct. One study [1] in the US released in 2020 found almost 90% of people admitted to speeding, but I don’t think anyone would say that speeding is now approved by the authorities and consequence-free. [1] https://www.thezebra.com/resources/research/speeding-car-insurance-rates/ https://www.thezebra.com/resources/research/speeding-car-ins...
  - bheadmaster 3y ago
    > Just because everybody does something the wrong way doesn’t somehow make it magically correct. Depends on the definition of "correct". One way to define "correct" is "as defined by the standard". However, standard serves a purpose - to maintain interoperability between components, so the other, deeper, way to define "correct" is "interoperable with the ecosystem". The official standard can argue that relying on the implementation detail is incorrect because they didn't specify it explicitly, but they would go against the grain of the rest of the ecosystem. Which reminds me of an old joke: An old woman is watching the news. She sees a news report saying there is a car driving in the wrong direction on the highway. So the old woman calls up her husband. Old woman: be careful on the highway dear, there is a crazy driver on the highway driving the wrong way! Husband: It’s not just one car, it’s hundreds of them!
    - nerdponx 3y ago
      Lest we forget that Python doesn't even have a language standard. The "standard" is a combination of PEPs, the Python docs, and the CPython implementation. Implementation details are language features, because implementation details are the standard. Python programs that rely on the GIL are not "wrong". Such programs are relying on clearly-documented features of the system that they use. Is it "wrong" to use GCC-specific features in a C program, if you know that you only intend to target GCC?
      - usrbinbash 3y ago
        > Implementation details are language features, because implementation details are the standard. Python programs that rely on the GIL are not "wrong". The problem here is: Implementation details are not guaranteed to be stable. They can change with every release. So if I write code that relies on any particular implementation detail, it may be correct today, and may be wrong two weeks from now, even though I didn't change anything.
      - samus 3y ago
        GCC-specific features are explicitly documented in their manual, and they can and should be used if one is fine with one's program being tied to that compiler. Best example: the Linux kernel. On the other hand, Linux kernel developers have also been burned by GCC changing what it does regarding undefined behavior (the infamous not-so-corner cases of C/C++). They expect GCC to be sort of like an assembler, which does the most straightforward thing in case of ambiguities. Instead, GCC is an optimizing compiler that is expected to deliver high-performance code for general-purpose programs. It treats undefined behavior as opportunities to enact optimizations. What it actually does is also usually documented, but also here different people have different expectations about what that implies. The GIL is an implementation choice that was put in place by Python's developers for simplicity's sake. It is important to be aware of it because it has huge performance implications, but it is unwise to rely on it for semantics. Especially since it has not exactly been a secret that various parties would eventually like it to be removed. Anyways, the GIL has very little to do with Python's semantics. User code is racy with or without the GIL, which is actually the point of TA.
  - hardware2win 3y ago
    Nobody says it is correct Just hard as hell to change
  - gjvc 3y ago
    Just because everybody does something the wrong way doesn’t somehow make it magically correct. Hyrum's Law says the opposite: https://www.hyrumslaw.com/ https://www.hyrumslaw.com/ "With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody." Named after some guy called Hyrum who worked / works at Google.
    - verteu 3y ago
      Agreed. It's also the "the first rule of kernel maintenance," according to Linus: https://linuxreviews.org/WE_DO_NOT_BREAK_USERSPACE https://linuxreviews.org/WE_DO_NOT_BREAK_USERSPACE
  - marcinzm 3y ago
    > One study [1] in the US released in 2020 found almost 90% of people admitted to speeding, but I don’t think anyone would say that speeding is now approved by the authorities and consequence-free. I would say jaywalking in NYC is a much better example because breaking the law doesn't kill others except in rare cases. Jaywalking is illegal. It's also not enforced for all intents and purposes. Every attempt to enforce it has led to such loud public outcry that it was stopped.
  - HDThoreaun 3y ago
    Speeding absolutely is approved by the authorities and consequence free where I live.
  - citrin_ru 3y ago
    If 90% are speeding on a given highway / road then the speed limit is likely unreasonably low. Traffic accident which could be prevented by a speed limit typically caused by fastest 10% of the speed distribution, not by remaining 90%.
- usrbinbash 3y ago
  Maybe, but "de-facto" and "actual" are still not the same, and if I write code that relies on them being the same, then I have only myself to blame when my code breaks. e.g.; there are C-compilers that usually zero most allocated structures. That's an implementation detail however, not a feature of C. C doesn't guarantee you zeroed memory anywhere, and code that assumes otherwise is just one compilation away from a desaster.
- samus 3y ago
  The vendor might not agree with that and change the implementation detail at their convenience. In absence of a prescriptive standard, a reference could be reasonably trusted, but relying on only sort-of-documented implementation details is a kind of tech debt.
BiteCode_dev 3y ago
The gil is a feature. Not using a feature that makes sense in our context is counter productive.
- nicolaslem 3y ago
  The GIL in an implementation detail of CPython, it's not part of Python the language.
  - colanderman 3y ago
    It's been a while since I've followed Python closely but last I heard Python-the-language is de facto defined as "whatever CPython does". Has Python since grown a proper language specification?
    - formerly_proven 3y ago
      No.
    - dragonwriter 3y ago
      Python doesn't have a full specification, but it does have first party documentation that distinguishes between CPython implementation details and language guarantees, in part to support alternative implementations.
      - pdonis 3y ago
        > Python doesn't have a full specification This doesn't count? https://docs.python.org/3/reference/index.html https://docs.python.org/3/reference/index.html
        squeaky-clean 3y ago
        The introduction section states that it's not a complete/exact specification. If you're using a Python implementation and have a question, this should answer it. If you're writing a Python implementation and have a question, this may not answer it. > Consequently, if you were coming from Mars and tried to re-implement Python from this document alone, you might have to guess things and in fact you would probably end up implementing quite a different language. On the other hand, if you are using Python and wonder what the precise rules about a particular area of the language are, you should definitely be able to find them here. If you would like to see a more formal definition of the language, maybe you could volunteer your time — or invent a cloning machine :-).
o11c 3y ago
It depends on what kind of "atomic". Assuming sequential consistency is pretty broken. Assuming acquire/release atomicity is much more reasonable. Assuming "at least relaxed" is outright mandatory.
blamestross 3y ago
Most people and scripts using the GIL to ensure safety likely don't even realize it is necessary. That is the hard part of this type of migration. Decades of code written with the implicit assumption of the GIL for all situations.
- Gabrys1 3y ago
  I don't think people generally write code to use this or another guarantee of the implementation. They write code, they run it, it seems to work (on my machine) and that's it. Later, the implementation changes in a subtle way (or someone tries to run the code in an incompatible version) and the code stops working. Nothing to do with "relying on GIL". The same may apply to relying on dict key order, timing of things, or anything else. That's why we have race conditions, Docker images, freezing dependencies, "only works in IE", etc. Writing to spec is very rare in my experience. You usually learn the spec once someone tells you your code doesn't work on XYZ.
mikepurvis 3y ago
I dunno about that. There's a lot of Python out there with worker threads dumping their output into a shared "results" dict on the assumption that those insert operations are atomic.
- formerly_proven 3y ago
  Relying on implementation behavior, even if it’s very visible and hasn’t changed for a few decades, is morally wrong and reprehensible. Only the letter of the standard has any bearing on correct reality.
  - HDThoreaun 3y ago
    > morally wrong That's a wild one
    - smegsicle 3y ago
      woe is man who loses sight of god and sees only himself
    - cozzyd 3y ago
      clearly a user of https://github.com/munificent/vigil https://github.com/munificent/vigil
      - m463 3y ago
        lol from FAQ: Q: But isn't a language that deletes code crazy?
  - deleted 3y ago
    [deleted]
  - Kranar 3y ago
    Python's semantics do not have a standards document or specification. It's based on its reference implementation (CPython) and different variations of Python even have different semantics (Jython, IronPython, PyPy). The library is documented, and the syntax is also documented, but the semantics themselves are not.
lelanthran 3y ago
Python doesn't have a standards document. It has a reference implementation that defines the language. IOW, the python implementation is the standard, and therefore any code that relies on an implementation detail in the reference implementation is, by definition, correct.
- oblvious-earth 3y ago
  > Python doesn't have a standards document. It has a reference implementation that defines the language. That's not quite strictly true, Python does have a documented "Language Reference": https://docs.python.org/3/reference/index.html https://docs.python.org/3/reference/index.html If there is a contradiction between the Language Reference and CPython then one, or both, of them needs to be updated and it's treated on a case by case basis. If an alternative Python implementation follows the Language Reference but chooses different details outside it, that doesn't stop it from being "Python". Of course practically speaking most alternative implementations are incentivized to closely follow CPython.
  - solarkraft 3y ago
    And what about PEPs, which describe (in pretty good detail) what changes will be made?
    - samus 3y ago
      They describe large-scale changes that usually also lead to changes of implementation details. But their biggest disadvantage is that they are not kept up to date when things change due to later PEPs or smaller-scale changes interfering. After landing, they are of historical value only.
citrin_ru 3y ago
A related xkcd: https://xkcd.com/1172/ https://xkcd.com/1172/ For a sufficiently popular project there are often cases where users (other projects) rely on existing observed behavior even if it not guaranteed by documentation/specification.
hyperpape 3y ago
You can say it's broken until you're blue in the face, and honestly, you're not wrong, but it's irrelevant. If major libraries and user code are constantly incorrect, but work because of an implementation detail, then removing that implementation detail becomes extraordinarily difficult, verging on impossible. It would be like retrofitting your language to distinguish valid unicode text from arbitrary byte strings, when you'd previously treated them as equivalent.

by dfox 3y ago

bheadmaster 3y ago

True, but when a majority of ecosystem is relying on an implementation detail, that implementation detail becomes de-facto standard.

darklion 3y ago
Just because everybody does something the wrong way doesn’t somehow make it magically correct. One study [1] in the US released in 2020 found almost 90% of people admitted to speeding, but I don’t think anyone would say that speeding is now approved by the authorities and consequence-free. [1] https://www.thezebra.com/resources/research/speeding-car-insurance-rates/ https://www.thezebra.com/resources/research/speeding-car-ins...
- bheadmaster 3y ago
  > Just because everybody does something the wrong way doesn’t somehow make it magically correct. Depends on the definition of "correct". One way to define "correct" is "as defined by the standard". However, standard serves a purpose - to maintain interoperability between components, so the other, deeper, way to define "correct" is "interoperable with the ecosystem". The official standard can argue that relying on the implementation detail is incorrect because they didn't specify it explicitly, but they would go against the grain of the rest of the ecosystem. Which reminds me of an old joke: An old woman is watching the news. She sees a news report saying there is a car driving in the wrong direction on the highway. So the old woman calls up her husband. Old woman: be careful on the highway dear, there is a crazy driver on the highway driving the wrong way! Husband: It’s not just one car, it’s hundreds of them!
  - nerdponx 3y ago
    Lest we forget that Python doesn't even have a language standard. The "standard" is a combination of PEPs, the Python docs, and the CPython implementation. Implementation details are language features, because implementation details are the standard. Python programs that rely on the GIL are not "wrong". Such programs are relying on clearly-documented features of the system that they use. Is it "wrong" to use GCC-specific features in a C program, if you know that you only intend to target GCC?
    - usrbinbash 3y ago
      > Implementation details are language features, because implementation details are the standard. Python programs that rely on the GIL are not "wrong". The problem here is: Implementation details are not guaranteed to be stable. They can change with every release. So if I write code that relies on any particular implementation detail, it may be correct today, and may be wrong two weeks from now, even though I didn't change anything.
    - samus 3y ago
      GCC-specific features are explicitly documented in their manual, and they can and should be used if one is fine with one's program being tied to that compiler. Best example: the Linux kernel. On the other hand, Linux kernel developers have also been burned by GCC changing what it does regarding undefined behavior (the infamous not-so-corner cases of C/C++). They expect GCC to be sort of like an assembler, which does the most straightforward thing in case of ambiguities. Instead, GCC is an optimizing compiler that is expected to deliver high-performance code for general-purpose programs. It treats undefined behavior as opportunities to enact optimizations. What it actually does is also usually documented, but also here different people have different expectations about what that implies. The GIL is an implementation choice that was put in place by Python's developers for simplicity's sake. It is important to be aware of it because it has huge performance implications, but it is unwise to rely on it for semantics. Especially since it has not exactly been a secret that various parties would eventually like it to be removed. Anyways, the GIL has very little to do with Python's semantics. User code is racy with or without the GIL, which is actually the point of TA.
- hardware2win 3y ago
  Nobody says it is correct Just hard as hell to change
- gjvc 3y ago
  Just because everybody does something the wrong way doesn’t somehow make it magically correct. Hyrum's Law says the opposite: https://www.hyrumslaw.com/ https://www.hyrumslaw.com/ "With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody." Named after some guy called Hyrum who worked / works at Google.
  - verteu 3y ago
    Agreed. It's also the "the first rule of kernel maintenance," according to Linus: https://linuxreviews.org/WE_DO_NOT_BREAK_USERSPACE https://linuxreviews.org/WE_DO_NOT_BREAK_USERSPACE
- marcinzm 3y ago
  > One study [1] in the US released in 2020 found almost 90% of people admitted to speeding, but I don’t think anyone would say that speeding is now approved by the authorities and consequence-free. I would say jaywalking in NYC is a much better example because breaking the law doesn't kill others except in rare cases. Jaywalking is illegal. It's also not enforced for all intents and purposes. Every attempt to enforce it has led to such loud public outcry that it was stopped.
- HDThoreaun 3y ago
  Speeding absolutely is approved by the authorities and consequence free where I live.
- citrin_ru 3y ago
  If 90% are speeding on a given highway / road then the speed limit is likely unreasonably low. Traffic accident which could be prevented by a speed limit typically caused by fastest 10% of the speed distribution, not by remaining 90%.
usrbinbash 3y ago
Maybe, but "de-facto" and "actual" are still not the same, and if I write code that relies on them being the same, then I have only myself to blame when my code breaks. e.g.; there are C-compilers that usually zero most allocated structures. That's an implementation detail however, not a feature of C. C doesn't guarantee you zeroed memory anywhere, and code that assumes otherwise is just one compilation away from a desaster.
samus 3y ago
The vendor might not agree with that and change the implementation detail at their convenience. In absence of a prescriptive standard, a reference could be reasonably trusted, but relying on only sort-of-documented implementation details is a kind of tech debt.

BiteCode_dev 3y ago

The gil is a feature. Not using a feature that makes sense in our context is counter productive.

nicolaslem 3y ago
The GIL in an implementation detail of CPython, it's not part of Python the language.
- colanderman 3y ago
  It's been a while since I've followed Python closely but last I heard Python-the-language is de facto defined as "whatever CPython does". Has Python since grown a proper language specification?
  - formerly_proven 3y ago
    No.
  - dragonwriter 3y ago
    Python doesn't have a full specification, but it does have first party documentation that distinguishes between CPython implementation details and language guarantees, in part to support alternative implementations.
    - pdonis 3y ago
      > Python doesn't have a full specification This doesn't count? https://docs.python.org/3/reference/index.html https://docs.python.org/3/reference/index.html
      - squeaky-clean 3y ago
        The introduction section states that it's not a complete/exact specification. If you're using a Python implementation and have a question, this should answer it. If you're writing a Python implementation and have a question, this may not answer it. > Consequently, if you were coming from Mars and tried to re-implement Python from this document alone, you might have to guess things and in fact you would probably end up implementing quite a different language. On the other hand, if you are using Python and wonder what the precise rules about a particular area of the language are, you should definitely be able to find them here. If you would like to see a more formal definition of the language, maybe you could volunteer your time — or invent a cloning machine :-).

o11c 3y ago

It depends on what kind of "atomic". Assuming sequential consistency is pretty broken. Assuming acquire/release atomicity is much more reasonable. Assuming "at least relaxed" is outright mandatory.

blamestross 3y ago

Most people and scripts using the GIL to ensure safety likely don't even realize it is necessary. That is the hard part of this type of migration. Decades of code written with the implicit assumption of the GIL for all situations.

Gabrys1 3y ago
I don't think people generally write code to use this or another guarantee of the implementation. They write code, they run it, it seems to work (on my machine) and that's it. Later, the implementation changes in a subtle way (or someone tries to run the code in an incompatible version) and the code stops working. Nothing to do with "relying on GIL". The same may apply to relying on dict key order, timing of things, or anything else. That's why we have race conditions, Docker images, freezing dependencies, "only works in IE", etc. Writing to spec is very rare in my experience. You usually learn the spec once someone tells you your code doesn't work on XYZ.

mikepurvis 3y ago

I dunno about that. There's a lot of Python out there with worker threads dumping their output into a shared "results" dict on the assumption that those insert operations are atomic.

formerly_proven 3y ago
Relying on implementation behavior, even if it’s very visible and hasn’t changed for a few decades, is morally wrong and reprehensible. Only the letter of the standard has any bearing on correct reality.
- HDThoreaun 3y ago
  > morally wrong That's a wild one
  - smegsicle 3y ago
    woe is man who loses sight of god and sees only himself
  - cozzyd 3y ago
    clearly a user of https://github.com/munificent/vigil https://github.com/munificent/vigil
    - m463 3y ago
      lol from FAQ: Q: But isn't a language that deletes code crazy?
- deleted 3y ago
  [deleted]
- Kranar 3y ago
  Python's semantics do not have a standards document or specification. It's based on its reference implementation (CPython) and different variations of Python even have different semantics (Jython, IronPython, PyPy). The library is documented, and the syntax is also documented, but the semantics themselves are not.

lelanthran 3y ago

Python doesn't have a standards document. It has a reference implementation that defines the language. IOW, the python implementation is the standard, and therefore any code that relies on an implementation detail in the reference implementation is, by definition, correct.

oblvious-earth 3y ago
> Python doesn't have a standards document. It has a reference implementation that defines the language. That's not quite strictly true, Python does have a documented "Language Reference": https://docs.python.org/3/reference/index.html https://docs.python.org/3/reference/index.html If there is a contradiction between the Language Reference and CPython then one, or both, of them needs to be updated and it's treated on a case by case basis. If an alternative Python implementation follows the Language Reference but chooses different details outside it, that doesn't stop it from being "Python". Of course practically speaking most alternative implementations are incentivized to closely follow CPython.
- solarkraft 3y ago
  And what about PEPs, which describe (in pretty good detail) what changes will be made?
  - samus 3y ago
    They describe large-scale changes that usually also lead to changes of implementation details. But their biggest disadvantage is that they are not kept up to date when things change due to later PEPs or smaller-scale changes interfering. After landing, they are of historical value only.

citrin_ru 3y ago

A related xkcd: https://xkcd.com/1172/ https://xkcd.com/1172/ For a sufficiently popular project there are often cases where users (other projects) rely on existing observed behavior even if it not guaranteed by documentation/specification.

hyperpape 3y ago

You can say it's broken until you're blue in the face, and honestly, you're not wrong, but it's irrelevant. If major libraries and user code are constantly incorrect, but work because of an implementation detail, then removing that implementation detail becomes extraordinarily difficult, verging on impossible. It would be like retrofitting your language to distinguish valid unicode text from arbitrary byte strings, when you'd previously treated them as equivalent.