5 ms·

I can't help but read all the interjections from the core developers as a strong indication of why all these sorts of things tend to fail on the vine. From Unla

by ctoth 5y ago

I can't help but read all the interjections from the core developers as a strong indication of why all these sorts of things tend to fail on the vine. From Unladen Swallow onward there have been these groups off doing interesting and awesome experiments to try and make Python faster, and they never actually make it into something that Python end-users can use (yes, I know US had shortcomings). This is the work of one (admittedly super smart) guy over two years. Just like Nuitka, something else that was supposed to be impossible but just keeps making steady progress. Maybe I'm just seeing design by committee? The lone geniuses can only go so far. Maybe the problem is that Python can't quite decide what it wants to be because it's too many things for too many people already. Is the concept of Python the language, as opposed to Python the ecosystem, valuable enough so that a Python that broke backwards compatibility with all the C extensions would be useful as its own multicore-capable runtime? PyPy seemed to think so for a while and now has gone hard in the other direction, reimplementing (faking?) a bunch of the CPython extension API so maybe this approach would never work. I don't know, but seeing things like: > there is a large number of “dark matter” Python (and C extension) code out there that isn’t open-source. We need to be careful not to break it since it might not be feasible for its users to make required changes, or to report problems back upstream to us. In particular, some C extensions protect their own internal state with the GIL. This is a big worry, and might be a big hindrance to adoption of a GIL-free Python. really make me wonder if the community as a whole would conclude the same on these critical sorts of decisions which shape the future of the language if they were put forward and not just made by a couple people in a closed meeting. Would you prefer to support some weird arbitrary nameless closed source extensions, or have a multicore Python? This obviously depends on who you are and what you're doing, which leads us back to Python being too much for too many, but even here we can get a feeling for how many people do what with the language.

shepardrtc 5y ago
> Would you prefer to support some weird arbitrary nameless closed source extensions, or have a multicore Python? There's nothing wrong with staying on an older LTS version of Python. Let the people with the nameless closed-source stuff stick with that. The beauty of open source is that they can fork the older, GIL-ful version of Python and maintain it, if they like. Multicore would be a tremendous boon to the language.
- ldng 5y ago
  The problem is those nameless usually want the cake and eat it too but will complain rudely when their toys breaks.
- rbanffy 5y ago
  > Let the people with the nameless closed-source stuff stick with that. Let's hope it becomes a teachable moment.
  - vlovich123 5y ago
    Is there some particular reason to believe the Python team is even able to get good insight into regressions in the long tail of python packages? Also it's important to remember that a lot of of material contributions to the community (either to the foundation, via jobs, or even open-sourcing part of their internal stack) might be coming from closed source in some way. It's not wise to ignore that & I think the core team is rightfully cognizant of needing to balance that (balance - not tip to one extreme or the other).
    - rectang 5y ago
      Concurrency is impossible to prove sound without language-level guarantees. https://www2.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-1.pdf https://www2.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-... I'm astounded that a change which will release untold heisenbugs into the wild is being considered. It changes my view of Python. In terms of inducing subtle, silent breakage in existing code, it reminds me of this horrifying change from PHP 8: https://www.php.net/manual/en/migration80.incompatible.php#migration80.incompatible.core.string-number-comparision https://www.php.net/manual/en/migration80.incompatible.php#m...
      - da39a3ee 5y ago
        However it, err, looks like some things did need fixing in PHP v7.
      - drothlis 5y ago
rbanffy 5y ago
> really make me wonder if the community as a whole would conclude the same on these critical sorts of decisions which shape the future of the language if they were put forward and not just made by a couple people in a closed meeting. I for one couldn't care less if some proprietary binaries fail on Python 3.11 or so. That's why we keep multiple versions around (at last company, I could only use up to 3.6 because that was the version in the Sacred CentOS AMI) And, of course, a very critical piece of code was depending on a bug in regex that was fixed in 3.8 or so, and decided to break during a demo (where I was using 3.9 instead of 3.6).
- unityByFreedom 5y ago
  I agree. If you're doing something so special in your code that nobody is publicly doing it and you're unable to update the code to make it compatible, then don't upgrade. That's the risk you took when coding it. We shouldn't go out on a limb for hypotheticals that cannot be confirmed. Force them to demonstrate an example publicly so all the cards are on the table. Proprietary code is fine and you can still share concepts by extracting out the issues. Sorry to hear about your demo. Sounds not fun! At least that's in the past :)
formerly_proven 5y ago
> there is a large number of “dark matter” Python (and C extension) code out there that isn’t open-source. We need to be careful not to break it since it might not be feasible for its users to make required changes, or to report problems back upstream to us. In particular, some C extensions protect their own internal state with the GIL. This is a big worry, and might be a big hindrance to adoption of a GIL-free Python. Probably doesn't work across minor versions anyway, most stuff isn't built against the limited API.
bigdict 5y ago
If this proof of concept is accepted, Facebook will put more resources to move the project forward.
simonh 5y ago
I don't think it's very coherent to criticise the team because "Python can't quite decide what it wants to be", but also criticise them for not adopting all these crazy cool changes that would fundamentally change Python. Taking a highly conservative approach to breaking changes is absolutely not the same thing as being indecisive. The Python team has learned from experience how disruptive breaking changes can be.
pganssle 5y ago
> I can't help but read all the interjections from the core developers as a strong indication of why all these sorts of things tend to fail on the vine. I think this might be a misunderstanding of the nature of the event that these notes are generated from, unless I'm misunderstanding your objection. The point of this Q&A as I saw it was to explore the feasibility of the idea and fully flesh out the costs and benefits so that we can make informed decisions about how to proceed. The "random interjections" are notes of caution about what trade-offs need to be made. For example, it is very easy to overlook "dark matter" code because we don't have access to it, but it's almost certainly the majority of Python code out there. It is also not a complete deal-breaker to say that some change could break unknown proprietary extensions — otherwise we'd never be able to change anything; the key is that the changes have to be worth it. A lot of that depends on details — if it's easy to update C extensions for nogil mode (even if they were designed without parallelism in mind), then making breaking changes to remove the GIL might not be so bad. If nogil mode requires that most C extensions totally overhaul their reference counting and C API usage and the changes require restructuring code rather than something that can be done with automated search and replace, that's a much bigger cost and will probably come with a long term fork of the ecosystem (which is a huge pain to deal with) and it might not be worth it. Avoiding this sort of criticism will not make the underlying problems go away, and I think everyone involved understood that this meeting was intended to bring to light any objections that might guide the work towards ultimate resolution.

by ctoth 5y ago

shepardrtc 5y ago

> Would you prefer to support some weird arbitrary nameless closed source extensions, or have a multicore Python? There's nothing wrong with staying on an older LTS version of Python. Let the people with the nameless closed-source stuff stick with that. The beauty of open source is that they can fork the older, GIL-ful version of Python and maintain it, if they like. Multicore would be a tremendous boon to the language.

ldng 5y ago
The problem is those nameless usually want the cake and eat it too but will complain rudely when their toys breaks.
rbanffy 5y ago
> Let the people with the nameless closed-source stuff stick with that. Let's hope it becomes a teachable moment.
- vlovich123 5y ago
  Is there some particular reason to believe the Python team is even able to get good insight into regressions in the long tail of python packages? Also it's important to remember that a lot of of material contributions to the community (either to the foundation, via jobs, or even open-sourcing part of their internal stack) might be coming from closed source in some way. It's not wise to ignore that & I think the core team is rightfully cognizant of needing to balance that (balance - not tip to one extreme or the other).
  - rectang 5y ago
    Concurrency is impossible to prove sound without language-level guarantees. https://www2.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-1.pdf https://www2.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-... I'm astounded that a change which will release untold heisenbugs into the wild is being considered. It changes my view of Python. In terms of inducing subtle, silent breakage in existing code, it reminds me of this horrifying change from PHP 8: https://www.php.net/manual/en/migration80.incompatible.php#migration80.incompatible.core.string-number-comparision https://www.php.net/manual/en/migration80.incompatible.php#m...
    - da39a3ee 5y ago
      However it, err, looks like some things did need fixing in PHP v7.
    - drothlis 5y ago

rbanffy 5y ago

> really make me wonder if the community as a whole would conclude the same on these critical sorts of decisions which shape the future of the language if they were put forward and not just made by a couple people in a closed meeting. I for one couldn't care less if some proprietary binaries fail on Python 3.11 or so. That's why we keep multiple versions around (at last company, I could only use up to 3.6 because that was the version in the Sacred CentOS AMI) And, of course, a very critical piece of code was depending on a bug in regex that was fixed in 3.8 or so, and decided to break during a demo (where I was using 3.9 instead of 3.6).

unityByFreedom 5y ago
I agree. If you're doing something so special in your code that nobody is publicly doing it and you're unable to update the code to make it compatible, then don't upgrade. That's the risk you took when coding it. We shouldn't go out on a limb for hypotheticals that cannot be confirmed. Force them to demonstrate an example publicly so all the cards are on the table. Proprietary code is fine and you can still share concepts by extracting out the issues. Sorry to hear about your demo. Sounds not fun! At least that's in the past :)

formerly_proven 5y ago

> there is a large number of “dark matter” Python (and C extension) code out there that isn’t open-source. We need to be careful not to break it since it might not be feasible for its users to make required changes, or to report problems back upstream to us. In particular, some C extensions protect their own internal state with the GIL. This is a big worry, and might be a big hindrance to adoption of a GIL-free Python. Probably doesn't work across minor versions anyway, most stuff isn't built against the limited API.

bigdict 5y ago

If this proof of concept is accepted, Facebook will put more resources to move the project forward.

simonh 5y ago

I don't think it's very coherent to criticise the team because "Python can't quite decide what it wants to be", but also criticise them for not adopting all these crazy cool changes that would fundamentally change Python. Taking a highly conservative approach to breaking changes is absolutely not the same thing as being indecisive. The Python team has learned from experience how disruptive breaking changes can be.

pganssle 5y ago

> I can't help but read all the interjections from the core developers as a strong indication of why all these sorts of things tend to fail on the vine. I think this might be a misunderstanding of the nature of the event that these notes are generated from, unless I'm misunderstanding your objection. The point of this Q&A as I saw it was to explore the feasibility of the idea and fully flesh out the costs and benefits so that we can make informed decisions about how to proceed. The "random interjections" are notes of caution about what trade-offs need to be made. For example, it is very easy to overlook "dark matter" code because we don't have access to it, but it's almost certainly the majority of Python code out there. It is also not a complete deal-breaker to say that some change could break unknown proprietary extensions — otherwise we'd never be able to change anything; the key is that the changes have to be worth it. A lot of that depends on details — if it's easy to update C extensions for nogil mode (even if they were designed without parallelism in mind), then making breaking changes to remove the GIL might not be so bad. If nogil mode requires that most C extensions totally overhaul their reference counting and C API usage and the changes require restructuring code rather than something that can be done with automated search and replace, that's a much bigger cost and will probably come with a long term fork of the ecosystem (which is a huge pain to deal with) and it might not be worth it. Avoiding this sort of criticism will not make the underlying problems go away, and I think everyone involved understood that this meeting was intended to bring to light any objections that might guide the work towards ultimate resolution.