6 ms·

I don't have strong opinions about these PEPs, but there's some background on how python is governed that the pydantic maintainer omits (as does most of the dis

I don't have strong opinions about these PEPs, but there's some background on how python is governed that the pydantic maintainer omits (as does most of the discussion here). I know people don't like the specific decisions python has made recently (me too!), but the process for deprecation and implementation once those decisions are made is at least clear and unambiguous, thanks to backwards compatibility policy adopted in PEP 387, i.e. way back in 2009. "Wait for the warning to appear in at least two major Python versions. It's fine to wait more than two releases." PEP 563, the one that apparently breaks pydantic, was accepted during Python 3.7. This was before pydantic reached 1.0 and before fastapi even existed. It was eligible for full implementation in 3.9 according to the backward compatibility policy, but the plan right now is for it to be implemented in Python 3.10. Python 3.10 is due to code freeze in three weeks. The maintainers of these projects have had over three years to raise these issues. pydantic and fastapi both have outstanding documentation that is a credit to their projects and to the python community. One side-effect of this, however, is they have some inexperienced and frankly excitable users, who perhaps aren't aware how python itself is developed. These are the users their maintainers (see also https://twitter.com/tiangolo/status/1382800928982642692 https://twitter.com/tiangolo/status/1382800928982642692) are unleashing on the python core maintainers and release managers literally days before a release that will implement a change they've known about for over three years. I hope this can be worked out. PEP 649 looks promising. Maybe PEP 563 should wait for python 3.11. And pydantic and fastapi are great! But this is, simply put, a shitty way to treat volunteer open source maintainers at the absolute busiest and most stressful time of the release cycle.

BiteCode_dev 6y ago
The fact you are warned in advance that something bad is going to happen doesn't make it less bad. Just more manageable. The elephant in the room is that the core devs, after Python 2.7, have taken the very bad habit of breaking compat. For such a popular and core tech a Python, this is not responsible. It erodes trust. It also raises the cost of producing good software tremendously. There are so many moving parts in tech now a day you can't spend you time, as an open source dev, tracking everything that break all the time. Nor fixing it. We have limited resources, energy and spare time. And what if I miss it? It's human after all. A project of the scale of Python, that is so central, so essential, should not break on me more than once in a decade, at least willingly. The way the issue is dealt with currently is very short sighted. It's not about the short term cost of having some teams having to pay the price of a rewrite. It's about how bad it is for the Python community, ecosystem and project on the long run.
- wirrbel 6y ago
  > The elephant in the room is that the core devs, after Python 2.7, have taken the very bad habit of breaking compat. For such a popular and core tech a Python, this is not responsible. It erodes trust. It also raises the cost of producing good software tremendously. You know, the core devs cannot make it right for everyone. For each its-a-breaking-change-complaint there is a similar amount of we-want-shiny-new-things-and-my-teammates-want-to-move-on-to-shiny-new-language-X complaint. So its clear to me that Python in order to stay relevant needs to evolve, add features, break occasionally with new releases. There was once the idea of combining a couple of breaking changes in one release (I would call this the 'once-in-a-decade' breaking change you seem to find acceptable), that was Py2->Py3 and it was a shitshow, in part, because too many changes happened at once. So I am grateful that we now get the occasional breaking change, one change at a time. I think it is the right way if Python aims to be still relevant in 10 years from now.
  - BiteCode_dev 6y ago
    See my answer to your sibling. It's not an "or" proposition. The 2 to 3 transition was hard because we did an abrupt cut. We could have had few transitioning versions that supported both worlds, then disabled the old ways but with a switch to put them back on if needed for a while, and allow for a more peaceful transition.
    - uranusjr 6y ago
      It’s very much an “or” situation from a practical standpoint. Maybe not exclusively or, but it’s unrealistic to say one does not significantly affect the other. Python core dev (CPython or otherwise) is already thin on resource for a long time, and you can’t just keep adding things without cleaning up old cruft. Parties most involved in development of the Python language (CPython devs, the Steering Council, etc.) value sustainability most, so it makes sense they decide to drop old stuff when they see no need of it. On the other side, parties that value stability more (e.g. libraries authors) need to actively lobby their goals to the core devs so they can assess the situation correctly and find the right balance between the two. You can’t just sit there and expect people to understand your needs; you have to tell them, which is what pydantic failed to do in this situation.
oflannabhra 6y ago
Both authors saw PEP-649 as a solution, with no need to raise alarm bells. If you follow the discussion of the last proposal, you’ll see that it’s acceptance is now in doubt, and it’s inclusion in 3.10 seems unlikely. Both authors have been active in that discussion. The issue is not that they had no warning, it’s that the proposed solution is being pulled out from under them.
fabioz 6y ago
I must say I've just spent quite some time looking at both PEPs and discussions and I'm strongly in favor of PEP 563 right now... it does improve the performance of typed modules substantially (which is a big downside for me in typed codebases) -- PEP 649 improves the situation but still adds a considerable overhead to the import time. I think that the numbers given from Inada Naoki https://mail.python.org/archives/list/python-dev@python.org/message/K5PFGS6DQUHUG63UWRXYNLLMXAVELP32/ https://mail.python.org/archives/list/python-dev@python.org/... do sum it well. Also, I don't see what will be made impossible at the pydantic side... from the bug list that's shown at https://github.com/samuelcolvin/pydantic/issues/2678 https://github.com/samuelcolvin/pydantic/issues/2678, yes, the code needs to be adapted, but I don't see what's made impossible (maybe at some point the pydantic devs will actually say what the real problems are as so far no real concrete info on that was given). Heck, they could even probably not even use `typing.get_type_hints()` (which currently apparently does an internal `eval()` and treat the annotation string however they'd like -- maybe more work, but then, absolute freedom to treat it as they see fit). So, my plead goes the other way around: please don't make all projects which don't require runtime info from annotations pay the price for the few use cases which do use them (but don't make those impossible either -- but my take so far as that this is not the case).
williamsmj 6y ago
> I hope this can be worked out. PEP 649 looks promising. Maybe PEP 563 should wait for python 3.11. And pydantic and fastapi are great! But this is, simply put, a shitty way to treat volunteer open source maintainers at the absolute busiest and most stressful time of the release cycle. Updates from the fastapi maintainer (https://twitter.com/tiangolo/status/1383041351160369154 https://twitter.com/tiangolo/status/1383041351160369154): "Quick update, the Python steering council, and core devs are fully aware of this and supportively helping figure it out. You can read all the conversations here and in the issue, but please restrain from commenting, tagging, or writing to them directly about this, it's already being handled with care. And [my] Tweet is already causing more extra work for them than helping." And the pydantic maintainer (https://github.com/samuelcolvin/pydantic/issues/2678#issuecomment-821099258 https://github.com/samuelcolvin/pydantic/issues/2678#issueco...) "I only heard about the debate about PEP 649 on Wednesday night, and given the extremely narrow window to get my point heard, I banged the drum in every way I could think of. The lesson here is "careful what you wish for", my message has been amplified more than I expected. I'm sorry if that has wasted anyone's time, even more so if it reduces the chance my request gets a positive reception. You're also right that i should have engaged with python-dev long ago about this, I'm sorry I did not. Lesson (hopefully) learnt." (More detail and explanation in his comment, but that's the bit that's salient to the point I was making in this thread.)
jhayward 6y ago
> I know people don't like the specific decisions python has made recently (me too!), but the process for deprecation and implementation once those decisions are made is at least clear and unambiguous, thanks to backwards compatibility policy adopted in PEP 387, i.e. way back in 2009. > "Wait for the warning to appear in at least two major Python versions. It's fine to wait more than two releases." > PEP 563, the one that apparently breaks pydantic, was accepted during Python 3.7. [..] It was eligible for full implementation in 3.9 according to the backward compatibility policy Can you point me to the warning that was raised in Python 3.7, 3.8, and 3.9? I don't find it on a quick look.
- williamsmj 6y ago
  If you're looking for runtime warnings, me neither. I think the definition of "appear in" is vague. My understanding (not a python core dev, or even someone who follows PEPs closely) is that a PEP has a corresponding python version, but it does not necessarily result in changes to cpython (or other implementations) that issue runtime warnings.
  - jhayward 6y ago
    Ah, we disagree on that reading of PEP 387 then. I read it to say the API or behavior being altered must have a deprecation warning for two major releases.
    - williamsmj 6y ago
      No. I agree with your reading, but the reality is that the PSF doesn't appear to interpret it that way and it's sufficiently vague to allow them to get away with that :( I can't remember the last time (if ever) I saw a runtime deprecation warning from cpython itself (I see plenty from libraries). I don't understand PEP 563 (or the python type system and ecosystem of tooling) well enough to know whether it would even be possible for cpython itself to issue a deprecation warning in this case. In any case, it's clear that the pydantic and fastapi authors were aware of this issue.
williamsmj 6y ago
Ha! PEP 387 was edited yesterday in a way that does not make any the above wrong, but does mean you'll be confused if you go looking for my quote: https://github.com/python/peps/commit/307b9cdf8897cbade627733ad6c84fe3fc838d25 https://github.com/python/peps/commit/307b9cdf8897cbade62773....
williamsmj 6y ago
Postponed until 3.11: https://mail.python.org/archives/list/python-dev@python.org/thread/CLVXXPQ2T2LQ5MP2Y53VVQFCXYWQJHKZ/ https://mail.python.org/archives/list/python-dev@python.org/...

BiteCode_dev 6y ago

The fact you are warned in advance that something bad is going to happen doesn't make it less bad. Just more manageable. The elephant in the room is that the core devs, after Python 2.7, have taken the very bad habit of breaking compat. For such a popular and core tech a Python, this is not responsible. It erodes trust. It also raises the cost of producing good software tremendously. There are so many moving parts in tech now a day you can't spend you time, as an open source dev, tracking everything that break all the time. Nor fixing it. We have limited resources, energy and spare time. And what if I miss it? It's human after all. A project of the scale of Python, that is so central, so essential, should not break on me more than once in a decade, at least willingly. The way the issue is dealt with currently is very short sighted. It's not about the short term cost of having some teams having to pay the price of a rewrite. It's about how bad it is for the Python community, ecosystem and project on the long run.

wirrbel 6y ago
> The elephant in the room is that the core devs, after Python 2.7, have taken the very bad habit of breaking compat. For such a popular and core tech a Python, this is not responsible. It erodes trust. It also raises the cost of producing good software tremendously. You know, the core devs cannot make it right for everyone. For each its-a-breaking-change-complaint there is a similar amount of we-want-shiny-new-things-and-my-teammates-want-to-move-on-to-shiny-new-language-X complaint. So its clear to me that Python in order to stay relevant needs to evolve, add features, break occasionally with new releases. There was once the idea of combining a couple of breaking changes in one release (I would call this the 'once-in-a-decade' breaking change you seem to find acceptable), that was Py2->Py3 and it was a shitshow, in part, because too many changes happened at once. So I am grateful that we now get the occasional breaking change, one change at a time. I think it is the right way if Python aims to be still relevant in 10 years from now.
- BiteCode_dev 6y ago
  See my answer to your sibling. It's not an "or" proposition. The 2 to 3 transition was hard because we did an abrupt cut. We could have had few transitioning versions that supported both worlds, then disabled the old ways but with a switch to put them back on if needed for a while, and allow for a more peaceful transition.
  - uranusjr 6y ago
    It’s very much an “or” situation from a practical standpoint. Maybe not exclusively or, but it’s unrealistic to say one does not significantly affect the other. Python core dev (CPython or otherwise) is already thin on resource for a long time, and you can’t just keep adding things without cleaning up old cruft. Parties most involved in development of the Python language (CPython devs, the Steering Council, etc.) value sustainability most, so it makes sense they decide to drop old stuff when they see no need of it. On the other side, parties that value stability more (e.g. libraries authors) need to actively lobby their goals to the core devs so they can assess the situation correctly and find the right balance between the two. You can’t just sit there and expect people to understand your needs; you have to tell them, which is what pydantic failed to do in this situation.

oflannabhra 6y ago

Both authors saw PEP-649 as a solution, with no need to raise alarm bells. If you follow the discussion of the last proposal, you’ll see that it’s acceptance is now in doubt, and it’s inclusion in 3.10 seems unlikely. Both authors have been active in that discussion. The issue is not that they had no warning, it’s that the proposed solution is being pulled out from under them.

fabioz 6y ago

I must say I've just spent quite some time looking at both PEPs and discussions and I'm strongly in favor of PEP 563 right now... it does improve the performance of typed modules substantially (which is a big downside for me in typed codebases) -- PEP 649 improves the situation but still adds a considerable overhead to the import time. I think that the numbers given from Inada Naoki https://mail.python.org/archives/list/python-dev@python.org/message/K5PFGS6DQUHUG63UWRXYNLLMXAVELP32/ https://mail.python.org/archives/list/python-dev@python.org/... do sum it well. Also, I don't see what will be made impossible at the pydantic side... from the bug list that's shown at https://github.com/samuelcolvin/pydantic/issues/2678 https://github.com/samuelcolvin/pydantic/issues/2678, yes, the code needs to be adapted, but I don't see what's made impossible (maybe at some point the pydantic devs will actually say what the real problems are as so far no real concrete info on that was given). Heck, they could even probably not even use `typing.get_type_hints()` (which currently apparently does an internal `eval()` and treat the annotation string however they'd like -- maybe more work, but then, absolute freedom to treat it as they see fit). So, my plead goes the other way around: please don't make all projects which don't require runtime info from annotations pay the price for the few use cases which do use them (but don't make those impossible either -- but my take so far as that this is not the case).

williamsmj 6y ago

> I hope this can be worked out. PEP 649 looks promising. Maybe PEP 563 should wait for python 3.11. And pydantic and fastapi are great! But this is, simply put, a shitty way to treat volunteer open source maintainers at the absolute busiest and most stressful time of the release cycle. Updates from the fastapi maintainer (https://twitter.com/tiangolo/status/1383041351160369154 https://twitter.com/tiangolo/status/1383041351160369154): "Quick update, the Python steering council, and core devs are fully aware of this and supportively helping figure it out. You can read all the conversations here and in the issue, but please restrain from commenting, tagging, or writing to them directly about this, it's already being handled with care. And [my] Tweet is already causing more extra work for them than helping." And the pydantic maintainer (https://github.com/samuelcolvin/pydantic/issues/2678#issuecomment-821099258 https://github.com/samuelcolvin/pydantic/issues/2678#issueco...) "I only heard about the debate about PEP 649 on Wednesday night, and given the extremely narrow window to get my point heard, I banged the drum in every way I could think of. The lesson here is "careful what you wish for", my message has been amplified more than I expected. I'm sorry if that has wasted anyone's time, even more so if it reduces the chance my request gets a positive reception. You're also right that i should have engaged with python-dev long ago about this, I'm sorry I did not. Lesson (hopefully) learnt." (More detail and explanation in his comment, but that's the bit that's salient to the point I was making in this thread.)

jhayward 6y ago

> I know people don't like the specific decisions python has made recently (me too!), but the process for deprecation and implementation once those decisions are made is at least clear and unambiguous, thanks to backwards compatibility policy adopted in PEP 387, i.e. way back in 2009. > "Wait for the warning to appear in at least two major Python versions. It's fine to wait more than two releases." > PEP 563, the one that apparently breaks pydantic, was accepted during Python 3.7. [..] It was eligible for full implementation in 3.9 according to the backward compatibility policy Can you point me to the warning that was raised in Python 3.7, 3.8, and 3.9? I don't find it on a quick look.

williamsmj 6y ago
If you're looking for runtime warnings, me neither. I think the definition of "appear in" is vague. My understanding (not a python core dev, or even someone who follows PEPs closely) is that a PEP has a corresponding python version, but it does not necessarily result in changes to cpython (or other implementations) that issue runtime warnings.
- jhayward 6y ago
  Ah, we disagree on that reading of PEP 387 then. I read it to say the API or behavior being altered must have a deprecation warning for two major releases.
  - williamsmj 6y ago
    No. I agree with your reading, but the reality is that the PSF doesn't appear to interpret it that way and it's sufficiently vague to allow them to get away with that :( I can't remember the last time (if ever) I saw a runtime deprecation warning from cpython itself (I see plenty from libraries). I don't understand PEP 563 (or the python type system and ecosystem of tooling) well enough to know whether it would even be possible for cpython itself to issue a deprecation warning in this case. In any case, it's clear that the pydantic and fastapi authors were aware of this issue.

williamsmj 6y ago

Ha! PEP 387 was edited yesterday in a way that does not make any the above wrong, but does mean you'll be confused if you go looking for my quote: https://github.com/python/peps/commit/307b9cdf8897cbade627733ad6c84fe3fc838d25 https://github.com/python/peps/commit/307b9cdf8897cbade62773....

williamsmj 6y ago

Postponed until 3.11: https://mail.python.org/archives/list/python-dev@python.org/thread/CLVXXPQ2T2LQ5MP2Y53VVQFCXYWQJHKZ/ https://mail.python.org/archives/list/python-dev@python.org/...