16 ms·

PEP 563, PEP 649 and the future of pydantic and FastAPI

282 points by amrrs 6y ago

polyrand 6y ago
From a personal point of view, FastAPI and Pydantic are some of the greatest additions to the Python ecosystem lately. Apart from that, FastAPI is currently the top 3 web framework according to the Python developer survey[0]. I think having such a breaking change is a serious issue. [0] https://www.jetbrains.com/lp/python-developers-survey-2020/ https://www.jetbrains.com/lp/python-developers-survey-2020/
- darkteflon 6y ago
  What do you like about FastAPI and Pydantic?
  - SirSourdough 6y ago
    From the perspective of someone who mostly dabbles in creating APIs for my own use: - Auto-generated docs and simple front end for your API allow you to examine and interact with your endpoints with essentially no added work or tools - Feature rich and performant - Tight pairing with Pydantic makes specifying and validating data models easy Basically, FastAPI (and pydantic under the hood) just abstracts away a lot of the work you would need to do to build a relatively fully-featured API on your own, and adds quality of life features that one generally couldn’t justify building/setting up for themselves, especially for smaller projects. Plus, the docs are solid and it’s a pretty simple library to get started with. It’s very much been an “it just works” library for me in most cases.
  - qbasic_forever 6y ago
    I like that FastAPI has documentation meant to be read from beginning to end. It walks you through not only the how but the why of core design decisions, and even dives into ancillary things like pydantic. This is in sharp contrast to most python packages of modest size where IMHO the bare minimum effort is put into technical docs. Usually you get something using a 10+ year old Sphinx template that hasn't changed for modern devices, and no description of usage is more than a trite paragraph (i.e. the attention span of a developer writing a docstring).
    - KptMarchewa 6y ago
      Agreed. This is an issue not only in Python. How many Java packages documentation is just generated Javadoc?
  - ggregoire 6y ago
    Among other things, having the query and body parameters automatically validated from the type hints defined on the endpoint and having the corresponding HTTP error and error message automatically added to the response is a huge time saving. Also, the doc is great and the maintainers are very helpful on github.
  - polyrand 6y ago
    Other people have already mentioned great things about FastAPI (documentation, data validation, good performance, easy to use, etc). There's also a lot to love about Pydantic alone. For me, the best thing about Pydantic is that I get performant, strongly validated objects. When I create Python apps, I try to make sure functions receive and return Pydantic models. That way, if the function does not fail, I'm 100% sure about the data coming in and out. It also saves me from writing many if statements. Those Pydantic objects can also be dynamically created without losing the data validation parts. By using it, I feel very "secure" about my types and objects, but keeping the simplicity of Python for my whole app.
    - travisjungroth 6y ago
      Have you written it with an ORM backed app? I’m curious if you put the business logic on the Pydantic models and keep the ORM small, or put the logic on the ORM and treat Pydantic like a serializer. Personally, I’d rather work with Pydantic models than ORM.
      - sandGorgon 6y ago
        there's a way to autogenerate pydantic and ORM (sqlalchemy) models from the same python dataclasses. here's a fully functional gist - https://gist.github.com/sandys/671b8b86ba913e6436d4cb22d04b135f https://gist.github.com/sandys/671b8b86ba913e6436d4cb22d04b1...
      - madkat 6y ago
        We use SQLAlchemy as the ORM layer, but it's very small and most of the work is done with the Pydantic models. It works quite nicely, despite that we loose change tracking, but SQLAlchemy got us covered with merging models.
      - polyrand 6y ago
        I don’t use ORMs. For bigger apps, I use asyncpg and map the results to Pydantic objects. I write raw SQL and keep Pydantic objects as the “source of truth” throughout my app.
s_Hogg 6y ago
I read that and I'm still no clearer about how to do anything about this, though as a user of both pydantic and fastapi (and having mandated their adoption in my org), I'm extremely motivated to do something about this. Can anyone spell out to me precisely what i need to do to be heard?
- SomaticPirate 6y ago
  I think +1 on this post would be a good first step https://mail.python.org/archives/list/python-dev@python.org/thread/7VMJWFGHVXDSRQFHMXVJKDDOVT47B54T/ https://mail.python.org/archives/list/python-dev@python.org/...
- x1798DE 6y ago
  I'm pretty sure the answer is, "You don't need to do anything about this." Note that Brett Cannon (Python Steering Council member) said that more people +1-ing or liking the issue is probably going to be disregarded (or possibly be taken negatively), in the issue: https://github.com/samuelcolvin/pydantic/issues/2678#issuecomment-820784182 https://github.com/samuelcolvin/pydantic/issues/2678#issueco... and on twitter: https://twitter.com/brettsky/status/1382829606084829186?s=20 https://twitter.com/brettsky/status/1382829606084829186?s=20 It seems that this post and the attempts to draw a lot of attention outside of core dev is likely to hurt pydantic's position rather than help it (though probably it will do neither because the core devs / steering council aren't going to ignore an issue out of spite or something).
josalhor 6y ago
I have developed a quite powerful tool building on top of Python type hints. In a way, I have almost started loving their flexibility as an API designer. I hope PEP 649 or a similar proposal is approved before 3.10.
- Xavdidtheshadow 6y ago
  What tool is it?
  - josalhor 6y ago
    The tool is in process of getting published. Not much I can say yet. I didn't start this project and I am not the sole contributor. The project is medium-sized.
dec0dedab0de 6y ago
I've read all the arguments about static typing, but I still can't comprehend how people get themselves into a situation where using the wrong type is a problem. I totally understand it as an optimization if your language supports it, and even for generating docs. I just don't know why people insist on writing code to stop themselves and other developers from doing things. That said, I really really can't stand breaking backwards compatibility, and it sounds like that is what is happening here.
- progval 6y ago
  > I just don't know why people insist on writing code to stop themselves and other developers from doing things. In the hope of stopping yourself from doing the wrong things more often than the right things. It also forces some of the documentation (the signature) to be up to date.
- vore 6y ago
  In a larger codebase, there's two major kinds of value you can get out of types: - Types act as a contract between component surfaces - if you need to integrate multiple components and have different people working on it, having the compiler enforce the contract is super useful. - Types act as a refactoring aid - if you need to alter a component, the compiler can assist you in telling you what needs to be changed and you can offload more of that work at compile time rather than at run time. As pieces move and change, tracking down what else needs to get moved and changed gets more and more difficult and this is where the "passing the wrong type in" issue can occur: having the compiler help you out with that is a huge boon.
  - Varriount 6y ago
    Similar to your first point, types also allow an IDE to inform user about what values can be used for a function or class argument. I can't say that typing bugs occurred very often in released code I wrote before type annotations, but it was not uncommon to encounter them when testing code I had just written (where they usually just caused an exception early on in the program's lifecycle).
  - 6y ago
SomaticPirate 6y ago
I don't use Python nearly as much as I used to (mostly Go now) but changes like this make me realize how much I appreciate the Go 1 compatibility guarantee. I have Go libraries from 1.6 that I have upgraded without a second thought. These sort of PEPs (remember the walrus operator?) seem like fundamental language changes that serve to divide the community without much gained. Although I will say, Go has made it much easier to be involved in the language design process.
- qbasic_forever 6y ago
  Nothing is immune, these are fundamentally people and person problems which can and will affect any language. It's ironic that just this week one of the core maintainers of Perl stepped down because of backlash and harassment from other maintainers over proposed changes (https://news.ycombinator.com/item?id=26779152 https://news.ycombinator.com/item?id=26779152). Python is a little younger than Perl but also faced similar problems--Guido had so much backlash against proposals that he stepped down too (https://news.ycombinator.com/item?id=17515492 https://news.ycombinator.com/item?id=17515492). Go is younger than either of those but not immune to the same fate--just look at the total divide and amount of controversy created over things like generics or modules in the community. There's no magic fix for this--if an organization is going to thrive and grow (or even just survive) for 30+ years it has to address those issues sooner rather than later. Burnout, at all levels, is real.
ben509 6y ago
I had tried to make json-syntax work with __annotations__, glad to know it's not just me having trouble with it.
deleted 6y ago
[deleted]
keithasaurus 6y ago
I'll shamelessly piggy-back here, and note that I'm working on a different type of type-safe validation library for Python here: https://github.com/keithasaurus/koda https://github.com/keithasaurus/koda. I'm nearing an initial release, but am still working on some of the docs, naming, and things like that. If anyone would like to give feedback, I'd be grateful!
- remram 6y ago
  What's different about it?
shuringai 6y ago
I do use and like both fastapi and pydantic but this situiation is something they brought to themselves by adopting experimenral features into their mainline code and sell it as production ready. Python designers are not responsible for time-to-market decisions of other projects. It comes with the "no liability" part of opensource.
- travisjungroth 6y ago
  I don’t think calling type hints experimental is accurate. They’ve been in the stable release for 4 years.
  - folkrav 6y ago
    We've had type hints since 3.5 - therefore 2015, so 6 years.
    - travisjungroth 6y ago
      I was remembering 3.6 but you’re right. September 2015.
    - heavyset_go 6y ago
      We've had function annotations since Python 3.0, so for over a decade now.
  - nerdponx 6y ago
    It's definitely experimental. No other language has an established system like this, although Ruby seems to have acquired one in 3.0. We still aren't close to uncovering all the unsolved problems yet, let alone solving said problems.
    - travisjungroth 6y ago
      You and I seem to have different definitions of "experimental feature" then. By your definition, I don't think they did anything wrong. dataclasses in the standard library also rely on runtime type annotations. (how they don't have the same issue with 3.10 I don't know). I would hope people would start relying on new language features years after their release.
staticassertion 6y ago
Python is like a 40 year case study of how not to build a programming language.
- Blackthorn 6y ago
  If only the rest of us could fail this successfully.
  - staticassertion 6y ago
    Agreed, must be nice. A testament to the space, I think.
Grimm1 6y ago
"In the end I don't think python's core developers and steering council want to make your experience of python worse. They just haven't realised how important this is. (Even Larry hadn't heard of pydantic or FastAPI until yesterday when he emailed me, I guess they don't read the python developer survey wink)" Yikes should a steering council that doesn't even understand the needs of the general community through common sources like that survey really be the steering council? That's a really strong signal that they're out of touch and can do more harm than good. Personally given the drama over the last few years regarding BDFL and then some of the peps that have made it in in recent time, as a heavy user I've had various concerns over the direction of the language and this further drives those.
- sicromoft 6y ago
  Yep. Sadly, they are very out of touch. As recently as a year ago, Guido hadn't even heard of Poetry: https://twitter.com/gvanrossum/status/1227126706089021440 https://twitter.com/gvanrossum/status/1227126706089021440
  - earthboundkid 6y ago
    Ouch. I knew PyPI and pip were unloved, but wow, what brutal neglect.
    - guitarbill 6y ago
      See also mypy and pyproject.toml support (https://github.com/python/mypy/issues/5205 https://github.com/python/mypy/issues/5205), although there's currently a PR in progress that'll hopefully put that to rest.
  - IAmNotAFix 6y ago
    I find it reassuring in some way. The day Python turns to an ecosystem as dynamic and community-driven as JavaScript is the day it turns to shit.
- nerdponx 6y ago
  On the contrary, I think they are more concerned with supporting the core language and interpreter than with anything else. Also why is this a Github issue and not being discussed somewhere like the PEP subforum of discuss.python.org?
fiddlerwoaroof 6y ago
This probably would have upset me five years ago when I was still invested in the Python ecosystem (I had been using it since 2.1 in various capacities): the decision to break backwards compatibility in Python 3 was a huge factor in my decision to look around for another language to focus on: my experience with Common Lisp and Clojure (see Rich Hickey’s Speculation talk) has really made me value ecosystems where the people managing the ecosystem don’t waste everyone’s time with trivial breaking changes.
davesque 6y ago
Honestly, I have mixed feelings about this. Pydantic is pretty cool but also kind of a strange bird. It seems to have been originally conceived to do the job of forms in a web app (serialization/deserialization). But then it also has a double life as a runtime type validation framework. And I don't think it really does either one quite that well. It definitely offers a level of conciseness that is nice compared with traditional form or serializer frameworks. But then at times it seems underpowered or awkward to customize. I think that's because it doesn't seem to know exactly what it wants to be. I've worked on projects where we used pydantic and it eventually became clear that we were either overusing or misusing it. It always seemed like the abuse of pydantic came out of developers on the team not really understanding what its purpose was. Maybe it's just my personal bias, but I think it's worth bringing a bit of skepticism to this debate and not just accepting the description of things in the issue ticket at face value. In other words, is it possible that pydantic is colliding with the steering council because it's misusing the annotations feature? I don't think I have a clear answer but it's worth asking the question anyway.
- nerdponx 6y ago
  I agree, I don't love Pydantic. I much prefer the "stack" of Attrs + Marshmallow + Desert + Apispec. More tools, yes, but they compose nicely and they each do one thing well. Now where have I heard that one before... I wish FastAPI had chosen that path instead of Pydantic, but here we are. FastAPI itself is a wonderful tool, with wonderful docs. is it possible that pydantic is colliding with the steering council because it's misusing the annotations feature? Yes. There are lots of good reasons why types aren't necessarily a great match with runtime validation. It gets the whole thing backwards IMO; I want to be erasing types at runtime and verifying them ahead-of-time. If I consume a piece of data that might be the wrong type, I want to explicitly validate the data myself, and return the desired type if and only if my validation passes. This is the Attrs approach and it works much better, even if you have to put in a few extra keystrokes in the handful of situations when you want explicit validation upon every class instantiation. Moreover, making annotations lazy is not mindless adherence to purity or some kind of intended "fuck you" to the Pydantic ecosystem. PEP 649 clearly states its motivations. Type annotations have always been a static analysis tool first and foremost, and I think it's absolutely wrong to even suggest that CPython should suffer a higher maintenance burden and suboptimal performance to support what is effectively an off-label usage.
hagy 6y ago
I've read through PEP 563 several times now and I just don't find the motivation particularly strong. The PEP itself outline two high level goals: 1. simply forward refs in types 2. reduce the runtime costs of type defs While I can respect both goals, I haven't found either problematic enough to break backwards compatibility, including functionality used in several mainstream libraries. Further, I have yet to see the runtime costs quantified. So I'm not even convinced this will have a tangible benefit. This PEP rejects the alternative option to just disable typedefs with opt-in __future__ imports or command line optimization args. Yet, they rejected this because it would break existing functionality for code that relies on runtime typedefs. And the proposed solution also breaks existing functionality. In terms of forward refs, the deferred evaluation just feels like a kludge. Every Python programmer is already familiar with how Python evaluates module code at runtime and understands how this prevents forward refs. If anything, this kludge adds a special case for typedefs. I really feel its time for Python 3 to freeze language features. This PEP and the previous walrus operator fiasco feel like people trying to morph Python into something that it isn't with one kludge after another. This will not lead to a robust language, but instead another Perl 5 monstrosity of complexity. Instead, I feel programmers looking for more robust, performant, and expressive languages should start exploring new languages. Let Python be the best possible Python and other language features can be explored in languages that incorporate those features in a holistic design.
- monster2control 6y ago
  I agree with this pretty much 100%. Stop adding new features for features sake. I can’t actually think of anything since 3.6 that I yearn to use. If you can’t add something without breaking backwards comparability, time to release Python 4 and wait another 15 years for adoption.
  - matanrubin 6y ago
    Well, dataclasses are pretty darn useful and were added in 3.7. I can't imangine writing Python code without them now...
- heavyset_go 6y ago
atxbcp 6y ago
FastAPI is just a rip off of Starlette, it's obvious when you look at the code (https://github.com/tiangolo/fastapi/blob/master/fastapi/templating.py https://github.com/tiangolo/fastapi/blob/master/fastapi/temp..., https://github.com/tiangolo/fastapi/blob/master/fastapi/testclient.py https://github.com/tiangolo/fastapi/blob/master/fastapi/test..., etc). Hopefully this change will kill FastAPI and force people to use the original project that is Starlette (https://www.starlette.io/ https://www.starlette.io/) As to why people used FastAPI instead of Starlette in the first place, it comes down to hype and great marketing.
- narism 6y ago
  Rip off? They explicitly say it’s based on starlette (along with why you might want to use FastAPI instead): https://fastapi.tiangolo.com/alternatives/#starlette https://fastapi.tiangolo.com/alternatives/#starlette
  - atxbcp 6y ago
    Not sure what else they could say, when 95% of FastAPI features come directly from Starlette. Again, the code on github speak for itself. As for the remaining 5%, that's basically pydantic and the swagger / openAPI stuff: questionably useful and very easily reimplemented if need be. The way I see it, FastAPI stole much of the recognition Starlette deserved. I find it a bit sad. Starlette is where all the async magic happens.
- maccam912 6y ago
  I used starlette and switched to fastapi. For a one person project the bonus free swagger docs in fastapi were what got my attention. If that's also built in to starlette... I guess then the marketing worked on me.
  - BerislavLopac 6y ago
    You might want to take a look at (shameless plug ;) ) https://pyotr.readthedocs.io https://pyotr.readthedocs.io
- 6y ago
williamsmj 6y ago
I don't have strong opinions about these PEPs, but there's some background on how python is governed that the pydantic maintainer omits (as does most of the discussion here). I know people don't like the specific decisions python has made recently (me too!), but the process for deprecation and implementation once those decisions are made is at least clear and unambiguous, thanks to backwards compatibility policy adopted in PEP 387, i.e. way back in 2009. "Wait for the warning to appear in at least two major Python versions. It's fine to wait more than two releases." PEP 563, the one that apparently breaks pydantic, was accepted during Python 3.7. This was before pydantic reached 1.0 and before fastapi even existed. It was eligible for full implementation in 3.9 according to the backward compatibility policy, but the plan right now is for it to be implemented in Python 3.10. Python 3.10 is due to code freeze in three weeks. The maintainers of these projects have had over three years to raise these issues. pydantic and fastapi both have outstanding documentation that is a credit to their projects and to the python community. One side-effect of this, however, is they have some inexperienced and frankly excitable users, who perhaps aren't aware how python itself is developed. These are the users their maintainers (see also https://twitter.com/tiangolo/status/1382800928982642692 https://twitter.com/tiangolo/status/1382800928982642692) are unleashing on the python core maintainers and release managers literally days before a release that will implement a change they've known about for over three years. I hope this can be worked out. PEP 649 looks promising. Maybe PEP 563 should wait for python 3.11. And pydantic and fastapi are great! But this is, simply put, a shitty way to treat volunteer open source maintainers at the absolute busiest and most stressful time of the release cycle.
- BiteCode_dev 6y ago
  The fact you are warned in advance that something bad is going to happen doesn't make it less bad. Just more manageable. The elephant in the room is that the core devs, after Python 2.7, have taken the very bad habit of breaking compat. For such a popular and core tech a Python, this is not responsible. It erodes trust. It also raises the cost of producing good software tremendously. There are so many moving parts in tech now a day you can't spend you time, as an open source dev, tracking everything that break all the time. Nor fixing it. We have limited resources, energy and spare time. And what if I miss it? It's human after all. A project of the scale of Python, that is so central, so essential, should not break on me more than once in a decade, at least willingly. The way the issue is dealt with currently is very short sighted. It's not about the short term cost of having some teams having to pay the price of a rewrite. It's about how bad it is for the Python community, ecosystem and project on the long run.
Helmut10001 6y ago
I've been using FastAPI and type annotations for over 2 years now and they have improved even small scripts. I love FastAPI, it is the best Framework I've come across in years. I wish there was a way to vote for PEP 649 instead of letting a small group decide about this alone. Maybe we can create a fork of Python 3.9 and call it different from hereon?
- codethief 6y ago
  > instead of letting a small group decide about this alone. "Small group". It's a group that consists of all people willing to spend their spare time (or money) on improving Python and its ecosystem. In fact, since you asked: You can become a member and get voting rights, too -> https://www.python.org/psf/membership/ https://www.python.org/psf/membership/
gigatexal 6y ago
It would be a shame if FastAPI is crippled because of PEPs. It truly is a wonderful API and a very nice API to work with. Here’s hoping a solution can be had and that the Python core maintainers “see the light” so to speak.
azinman2 6y ago
Flask does the same thing as FastAPI with @app… what will it do? Or am I not understanding correctly?
- fleetfox 6y ago
  Decorators and type hints are not the same thing.
  - azinman2 6y ago
    Ok so then I’m really confused. I thought it said annotations were having to be strings versus objects. If this isn’t the what’s called annotations in other languages, then what is the actual issue that pydantic has? The python type hints are I guess objects, but they’re standalone non-instantiated.. unless pydantic does some crazy stuff. Their main home page doesn’t show anything out of the ordinary here? What’s the actual issue?
muktabh 6y ago
I absolutely love working in FastAPI and pydantic. So much so that the backend of a new app I just launched in private beta on is written in FastAPI. My startup has traditionally been a flask workshop but FastAPI is just so much more productive due to the documentation and pydantic type checks. I seriously hope Python developers understand that apart from the superb programming language it is, package ecosystem also is what makes people prefer Python as the language to invest their hard labor into. Small teams need productive libraries like FastAPI to develop systems. Please don't kill FastAPI and pydantic. I don't want to go back to writing flask again after working in FastAPI.
rtpg 6y ago
Doesn’t data classes have the same stuff going on in terms of parsing type hints? Do their problems go away if they were using decorators I wonder...
WhyNotHugo 6y ago
Reminds me for https://xkcd.com/1172/ https://xkcd.com/1172/ Type annotations were meant to be used as linting, and other forms of type-checking done NOT AT RUNTIME. Gradually, some support for using them at runtime popped up, and them annotations starting being [ab]used for all sorts of things. PEP 563 (the one breaking pydantic here) is actually super important: it allows referencing things defined later. Like, a function on line 1 can take a type on line 10 as parameter. It is super necessary to make type annotations usable for their original purpose. It breaks pydantic, but also makes annotations usable for their intended purpose. Previously you had to manually opt it on each file, and now it's the default, as planned. I feel like pydantic and tools built on top of it were built ignoring the direction on which Python was heading. Much like the user in xkcd#1172.

282 points by amrrs 6y ago

From a personal point of view, FastAPI and Pydantic are some of the greatest additions to the Python ecosystem lately. Apart from that, FastAPI is currently the top 3 web framework according to the Python developer survey[0]. I think having such a breaking change is a serious issue. [0] https://www.jetbrains.com/lp/python-developers-survey-2020/ https://www.jetbrains.com/lp/python-developers-survey-2020/

darkteflon 6y ago
What do you like about FastAPI and Pydantic?
- SirSourdough 6y ago
  From the perspective of someone who mostly dabbles in creating APIs for my own use: - Auto-generated docs and simple front end for your API allow you to examine and interact with your endpoints with essentially no added work or tools - Feature rich and performant - Tight pairing with Pydantic makes specifying and validating data models easy Basically, FastAPI (and pydantic under the hood) just abstracts away a lot of the work you would need to do to build a relatively fully-featured API on your own, and adds quality of life features that one generally couldn’t justify building/setting up for themselves, especially for smaller projects. Plus, the docs are solid and it’s a pretty simple library to get started with. It’s very much been an “it just works” library for me in most cases.
- qbasic_forever 6y ago
  I like that FastAPI has documentation meant to be read from beginning to end. It walks you through not only the how but the why of core design decisions, and even dives into ancillary things like pydantic. This is in sharp contrast to most python packages of modest size where IMHO the bare minimum effort is put into technical docs. Usually you get something using a 10+ year old Sphinx template that hasn't changed for modern devices, and no description of usage is more than a trite paragraph (i.e. the attention span of a developer writing a docstring).
  - KptMarchewa 6y ago
    Agreed. This is an issue not only in Python. How many Java packages documentation is just generated Javadoc?
- ggregoire 6y ago
  Among other things, having the query and body parameters automatically validated from the type hints defined on the endpoint and having the corresponding HTTP error and error message automatically added to the response is a huge time saving. Also, the doc is great and the maintainers are very helpful on github.
- polyrand 6y ago
  Other people have already mentioned great things about FastAPI (documentation, data validation, good performance, easy to use, etc). There's also a lot to love about Pydantic alone. For me, the best thing about Pydantic is that I get performant, strongly validated objects. When I create Python apps, I try to make sure functions receive and return Pydantic models. That way, if the function does not fail, I'm 100% sure about the data coming in and out. It also saves me from writing many if statements. Those Pydantic objects can also be dynamically created without losing the data validation parts. By using it, I feel very "secure" about my types and objects, but keeping the simplicity of Python for my whole app.
  - travisjungroth 6y ago
    Have you written it with an ORM backed app? I’m curious if you put the business logic on the Pydantic models and keep the ORM small, or put the logic on the ORM and treat Pydantic like a serializer. Personally, I’d rather work with Pydantic models than ORM.
    - sandGorgon 6y ago
      there's a way to autogenerate pydantic and ORM (sqlalchemy) models from the same python dataclasses. here's a fully functional gist - https://gist.github.com/sandys/671b8b86ba913e6436d4cb22d04b135f https://gist.github.com/sandys/671b8b86ba913e6436d4cb22d04b1...
    - madkat 6y ago
      We use SQLAlchemy as the ORM layer, but it's very small and most of the work is done with the Pydantic models. It works quite nicely, despite that we loose change tracking, but SQLAlchemy got us covered with merging models.
    - polyrand 6y ago
      I don’t use ORMs. For bigger apps, I use asyncpg and map the results to Pydantic objects. I write raw SQL and keep Pydantic objects as the “source of truth” throughout my app.

s_Hogg 6y ago

I read that and I'm still no clearer about how to do anything about this, though as a user of both pydantic and fastapi (and having mandated their adoption in my org), I'm extremely motivated to do something about this. Can anyone spell out to me precisely what i need to do to be heard?

SomaticPirate 6y ago
I think +1 on this post would be a good first step https://mail.python.org/archives/list/python-dev@python.org/thread/7VMJWFGHVXDSRQFHMXVJKDDOVT47B54T/ https://mail.python.org/archives/list/python-dev@python.org/...
x1798DE 6y ago
I'm pretty sure the answer is, "You don't need to do anything about this." Note that Brett Cannon (Python Steering Council member) said that more people +1-ing or liking the issue is probably going to be disregarded (or possibly be taken negatively), in the issue: https://github.com/samuelcolvin/pydantic/issues/2678#issuecomment-820784182 https://github.com/samuelcolvin/pydantic/issues/2678#issueco... and on twitter: https://twitter.com/brettsky/status/1382829606084829186?s=20 https://twitter.com/brettsky/status/1382829606084829186?s=20 It seems that this post and the attempts to draw a lot of attention outside of core dev is likely to hurt pydantic's position rather than help it (though probably it will do neither because the core devs / steering council aren't going to ignore an issue out of spite or something).

josalhor 6y ago

I have developed a quite powerful tool building on top of Python type hints. In a way, I have almost started loving their flexibility as an API designer. I hope PEP 649 or a similar proposal is approved before 3.10.

Xavdidtheshadow 6y ago
What tool is it?
- josalhor 6y ago
  The tool is in process of getting published. Not much I can say yet. I didn't start this project and I am not the sole contributor. The project is medium-sized.

dec0dedab0de 6y ago

I've read all the arguments about static typing, but I still can't comprehend how people get themselves into a situation where using the wrong type is a problem. I totally understand it as an optimization if your language supports it, and even for generating docs. I just don't know why people insist on writing code to stop themselves and other developers from doing things. That said, I really really can't stand breaking backwards compatibility, and it sounds like that is what is happening here.

progval 6y ago
> I just don't know why people insist on writing code to stop themselves and other developers from doing things. In the hope of stopping yourself from doing the wrong things more often than the right things. It also forces some of the documentation (the signature) to be up to date.
vore 6y ago
In a larger codebase, there's two major kinds of value you can get out of types: - Types act as a contract between component surfaces - if you need to integrate multiple components and have different people working on it, having the compiler enforce the contract is super useful. - Types act as a refactoring aid - if you need to alter a component, the compiler can assist you in telling you what needs to be changed and you can offload more of that work at compile time rather than at run time. As pieces move and change, tracking down what else needs to get moved and changed gets more and more difficult and this is where the "passing the wrong type in" issue can occur: having the compiler help you out with that is a huge boon.
- Varriount 6y ago
  Similar to your first point, types also allow an IDE to inform user about what values can be used for a function or class argument. I can't say that typing bugs occurred very often in released code I wrote before type annotations, but it was not uncommon to encounter them when testing code I had just written (where they usually just caused an exception early on in the program's lifecycle).
- 6y ago

SomaticPirate 6y ago

I don't use Python nearly as much as I used to (mostly Go now) but changes like this make me realize how much I appreciate the Go 1 compatibility guarantee. I have Go libraries from 1.6 that I have upgraded without a second thought. These sort of PEPs (remember the walrus operator?) seem like fundamental language changes that serve to divide the community without much gained. Although I will say, Go has made it much easier to be involved in the language design process.

qbasic_forever 6y ago
Nothing is immune, these are fundamentally people and person problems which can and will affect any language. It's ironic that just this week one of the core maintainers of Perl stepped down because of backlash and harassment from other maintainers over proposed changes (https://news.ycombinator.com/item?id=26779152 https://news.ycombinator.com/item?id=26779152). Python is a little younger than Perl but also faced similar problems--Guido had so much backlash against proposals that he stepped down too (https://news.ycombinator.com/item?id=17515492 https://news.ycombinator.com/item?id=17515492). Go is younger than either of those but not immune to the same fate--just look at the total divide and amount of controversy created over things like generics or modules in the community. There's no magic fix for this--if an organization is going to thrive and grow (or even just survive) for 30+ years it has to address those issues sooner rather than later. Burnout, at all levels, is real.

ben509 6y ago

I had tried to make json-syntax work with __annotations__, glad to know it's not just me having trouble with it.

deleted 6y ago

[deleted]

keithasaurus 6y ago

I'll shamelessly piggy-back here, and note that I'm working on a different type of type-safe validation library for Python here: https://github.com/keithasaurus/koda https://github.com/keithasaurus/koda. I'm nearing an initial release, but am still working on some of the docs, naming, and things like that. If anyone would like to give feedback, I'd be grateful!

remram 6y ago
What's different about it?

shuringai 6y ago

I do use and like both fastapi and pydantic but this situiation is something they brought to themselves by adopting experimenral features into their mainline code and sell it as production ready. Python designers are not responsible for time-to-market decisions of other projects. It comes with the "no liability" part of opensource.

travisjungroth 6y ago
I don’t think calling type hints experimental is accurate. They’ve been in the stable release for 4 years.
- folkrav 6y ago
  We've had type hints since 3.5 - therefore 2015, so 6 years.
  - travisjungroth 6y ago
    I was remembering 3.6 but you’re right. September 2015.
  - heavyset_go 6y ago
    We've had function annotations since Python 3.0, so for over a decade now.
- nerdponx 6y ago
  It's definitely experimental. No other language has an established system like this, although Ruby seems to have acquired one in 3.0. We still aren't close to uncovering all the unsolved problems yet, let alone solving said problems.
  - travisjungroth 6y ago
    You and I seem to have different definitions of "experimental feature" then. By your definition, I don't think they did anything wrong. dataclasses in the standard library also rely on runtime type annotations. (how they don't have the same issue with 3.10 I don't know). I would hope people would start relying on new language features years after their release.

staticassertion 6y ago

Python is like a 40 year case study of how not to build a programming language.

Blackthorn 6y ago
If only the rest of us could fail this successfully.
- staticassertion 6y ago
  Agreed, must be nice. A testament to the space, I think.

Grimm1 6y ago

"In the end I don't think python's core developers and steering council want to make your experience of python worse. They just haven't realised how important this is. (Even Larry hadn't heard of pydantic or FastAPI until yesterday when he emailed me, I guess they don't read the python developer survey wink)" Yikes should a steering council that doesn't even understand the needs of the general community through common sources like that survey really be the steering council? That's a really strong signal that they're out of touch and can do more harm than good. Personally given the drama over the last few years regarding BDFL and then some of the peps that have made it in in recent time, as a heavy user I've had various concerns over the direction of the language and this further drives those.

sicromoft 6y ago
Yep. Sadly, they are very out of touch. As recently as a year ago, Guido hadn't even heard of Poetry: https://twitter.com/gvanrossum/status/1227126706089021440 https://twitter.com/gvanrossum/status/1227126706089021440
- earthboundkid 6y ago
  Ouch. I knew PyPI and pip were unloved, but wow, what brutal neglect.
  - guitarbill 6y ago
    See also mypy and pyproject.toml support (https://github.com/python/mypy/issues/5205 https://github.com/python/mypy/issues/5205), although there's currently a PR in progress that'll hopefully put that to rest.
- IAmNotAFix 6y ago
  I find it reassuring in some way. The day Python turns to an ecosystem as dynamic and community-driven as JavaScript is the day it turns to shit.
nerdponx 6y ago
On the contrary, I think they are more concerned with supporting the core language and interpreter than with anything else. Also why is this a Github issue and not being discussed somewhere like the PEP subforum of discuss.python.org?

fiddlerwoaroof 6y ago

This probably would have upset me five years ago when I was still invested in the Python ecosystem (I had been using it since 2.1 in various capacities): the decision to break backwards compatibility in Python 3 was a huge factor in my decision to look around for another language to focus on: my experience with Common Lisp and Clojure (see Rich Hickey’s Speculation talk) has really made me value ecosystems where the people managing the ecosystem don’t waste everyone’s time with trivial breaking changes.

davesque 6y ago

Honestly, I have mixed feelings about this. Pydantic is pretty cool but also kind of a strange bird. It seems to have been originally conceived to do the job of forms in a web app (serialization/deserialization). But then it also has a double life as a runtime type validation framework. And I don't think it really does either one quite that well. It definitely offers a level of conciseness that is nice compared with traditional form or serializer frameworks. But then at times it seems underpowered or awkward to customize. I think that's because it doesn't seem to know exactly what it wants to be. I've worked on projects where we used pydantic and it eventually became clear that we were either overusing or misusing it. It always seemed like the abuse of pydantic came out of developers on the team not really understanding what its purpose was. Maybe it's just my personal bias, but I think it's worth bringing a bit of skepticism to this debate and not just accepting the description of things in the issue ticket at face value. In other words, is it possible that pydantic is colliding with the steering council because it's misusing the annotations feature? I don't think I have a clear answer but it's worth asking the question anyway.

nerdponx 6y ago
I agree, I don't love Pydantic. I much prefer the "stack" of Attrs + Marshmallow + Desert + Apispec. More tools, yes, but they compose nicely and they each do one thing well. Now where have I heard that one before... I wish FastAPI had chosen that path instead of Pydantic, but here we are. FastAPI itself is a wonderful tool, with wonderful docs. is it possible that pydantic is colliding with the steering council because it's misusing the annotations feature? Yes. There are lots of good reasons why types aren't necessarily a great match with runtime validation. It gets the whole thing backwards IMO; I want to be erasing types at runtime and verifying them ahead-of-time. If I consume a piece of data that might be the wrong type, I want to explicitly validate the data myself, and return the desired type if and only if my validation passes. This is the Attrs approach and it works much better, even if you have to put in a few extra keystrokes in the handful of situations when you want explicit validation upon every class instantiation. Moreover, making annotations lazy is not mindless adherence to purity or some kind of intended "fuck you" to the Pydantic ecosystem. PEP 649 clearly states its motivations. Type annotations have always been a static analysis tool first and foremost, and I think it's absolutely wrong to even suggest that CPython should suffer a higher maintenance burden and suboptimal performance to support what is effectively an off-label usage.

hagy 6y ago

I've read through PEP 563 several times now and I just don't find the motivation particularly strong. The PEP itself outline two high level goals: 1. simply forward refs in types 2. reduce the runtime costs of type defs While I can respect both goals, I haven't found either problematic enough to break backwards compatibility, including functionality used in several mainstream libraries. Further, I have yet to see the runtime costs quantified. So I'm not even convinced this will have a tangible benefit. This PEP rejects the alternative option to just disable typedefs with opt-in __future__ imports or command line optimization args. Yet, they rejected this because it would break existing functionality for code that relies on runtime typedefs. And the proposed solution also breaks existing functionality. In terms of forward refs, the deferred evaluation just feels like a kludge. Every Python programmer is already familiar with how Python evaluates module code at runtime and understands how this prevents forward refs. If anything, this kludge adds a special case for typedefs. I really feel its time for Python 3 to freeze language features. This PEP and the previous walrus operator fiasco feel like people trying to morph Python into something that it isn't with one kludge after another. This will not lead to a robust language, but instead another Perl 5 monstrosity of complexity. Instead, I feel programmers looking for more robust, performant, and expressive languages should start exploring new languages. Let Python be the best possible Python and other language features can be explored in languages that incorporate those features in a holistic design.

monster2control 6y ago
I agree with this pretty much 100%. Stop adding new features for features sake. I can’t actually think of anything since 3.6 that I yearn to use. If you can’t add something without breaking backwards comparability, time to release Python 4 and wait another 15 years for adoption.
- matanrubin 6y ago
  Well, dataclasses are pretty darn useful and were added in 3.7. I can't imangine writing Python code without them now...
heavyset_go 6y ago

atxbcp 6y ago

FastAPI is just a rip off of Starlette, it's obvious when you look at the code (https://github.com/tiangolo/fastapi/blob/master/fastapi/templating.py https://github.com/tiangolo/fastapi/blob/master/fastapi/temp..., https://github.com/tiangolo/fastapi/blob/master/fastapi/testclient.py https://github.com/tiangolo/fastapi/blob/master/fastapi/test..., etc). Hopefully this change will kill FastAPI and force people to use the original project that is Starlette (https://www.starlette.io/ https://www.starlette.io/) As to why people used FastAPI instead of Starlette in the first place, it comes down to hype and great marketing.

narism 6y ago
Rip off? They explicitly say it’s based on starlette (along with why you might want to use FastAPI instead): https://fastapi.tiangolo.com/alternatives/#starlette https://fastapi.tiangolo.com/alternatives/#starlette
- atxbcp 6y ago
  Not sure what else they could say, when 95% of FastAPI features come directly from Starlette. Again, the code on github speak for itself. As for the remaining 5%, that's basically pydantic and the swagger / openAPI stuff: questionably useful and very easily reimplemented if need be. The way I see it, FastAPI stole much of the recognition Starlette deserved. I find it a bit sad. Starlette is where all the async magic happens.
maccam912 6y ago
I used starlette and switched to fastapi. For a one person project the bonus free swagger docs in fastapi were what got my attention. If that's also built in to starlette... I guess then the marketing worked on me.
- BerislavLopac 6y ago
  You might want to take a look at (shameless plug ;) ) https://pyotr.readthedocs.io https://pyotr.readthedocs.io
6y ago

williamsmj 6y ago

I don't have strong opinions about these PEPs, but there's some background on how python is governed that the pydantic maintainer omits (as does most of the discussion here). I know people don't like the specific decisions python has made recently (me too!), but the process for deprecation and implementation once those decisions are made is at least clear and unambiguous, thanks to backwards compatibility policy adopted in PEP 387, i.e. way back in 2009. "Wait for the warning to appear in at least two major Python versions. It's fine to wait more than two releases." PEP 563, the one that apparently breaks pydantic, was accepted during Python 3.7. This was before pydantic reached 1.0 and before fastapi even existed. It was eligible for full implementation in 3.9 according to the backward compatibility policy, but the plan right now is for it to be implemented in Python 3.10. Python 3.10 is due to code freeze in three weeks. The maintainers of these projects have had over three years to raise these issues. pydantic and fastapi both have outstanding documentation that is a credit to their projects and to the python community. One side-effect of this, however, is they have some inexperienced and frankly excitable users, who perhaps aren't aware how python itself is developed. These are the users their maintainers (see also https://twitter.com/tiangolo/status/1382800928982642692 https://twitter.com/tiangolo/status/1382800928982642692) are unleashing on the python core maintainers and release managers literally days before a release that will implement a change they've known about for over three years. I hope this can be worked out. PEP 649 looks promising. Maybe PEP 563 should wait for python 3.11. And pydantic and fastapi are great! But this is, simply put, a shitty way to treat volunteer open source maintainers at the absolute busiest and most stressful time of the release cycle.

BiteCode_dev 6y ago
The fact you are warned in advance that something bad is going to happen doesn't make it less bad. Just more manageable. The elephant in the room is that the core devs, after Python 2.7, have taken the very bad habit of breaking compat. For such a popular and core tech a Python, this is not responsible. It erodes trust. It also raises the cost of producing good software tremendously. There are so many moving parts in tech now a day you can't spend you time, as an open source dev, tracking everything that break all the time. Nor fixing it. We have limited resources, energy and spare time. And what if I miss it? It's human after all. A project of the scale of Python, that is so central, so essential, should not break on me more than once in a decade, at least willingly. The way the issue is dealt with currently is very short sighted. It's not about the short term cost of having some teams having to pay the price of a rewrite. It's about how bad it is for the Python community, ecosystem and project on the long run.

Helmut10001 6y ago

I've been using FastAPI and type annotations for over 2 years now and they have improved even small scripts. I love FastAPI, it is the best Framework I've come across in years. I wish there was a way to vote for PEP 649 instead of letting a small group decide about this alone. Maybe we can create a fork of Python 3.9 and call it different from hereon?

codethief 6y ago
> instead of letting a small group decide about this alone. "Small group". It's a group that consists of all people willing to spend their spare time (or money) on improving Python and its ecosystem. In fact, since you asked: You can become a member and get voting rights, too -> https://www.python.org/psf/membership/ https://www.python.org/psf/membership/

gigatexal 6y ago

It would be a shame if FastAPI is crippled because of PEPs. It truly is a wonderful API and a very nice API to work with. Here’s hoping a solution can be had and that the Python core maintainers “see the light” so to speak.

azinman2 6y ago

Flask does the same thing as FastAPI with @app… what will it do? Or am I not understanding correctly?

fleetfox 6y ago
Decorators and type hints are not the same thing.
- azinman2 6y ago
  Ok so then I’m really confused. I thought it said annotations were having to be strings versus objects. If this isn’t the what’s called annotations in other languages, then what is the actual issue that pydantic has? The python type hints are I guess objects, but they’re standalone non-instantiated.. unless pydantic does some crazy stuff. Their main home page doesn’t show anything out of the ordinary here? What’s the actual issue?

muktabh 6y ago

I absolutely love working in FastAPI and pydantic. So much so that the backend of a new app I just launched in private beta on is written in FastAPI. My startup has traditionally been a flask workshop but FastAPI is just so much more productive due to the documentation and pydantic type checks. I seriously hope Python developers understand that apart from the superb programming language it is, package ecosystem also is what makes people prefer Python as the language to invest their hard labor into. Small teams need productive libraries like FastAPI to develop systems. Please don't kill FastAPI and pydantic. I don't want to go back to writing flask again after working in FastAPI.

rtpg 6y ago

Doesn’t data classes have the same stuff going on in terms of parsing type hints? Do their problems go away if they were using decorators I wonder...

WhyNotHugo 6y ago

Reminds me for https://xkcd.com/1172/ https://xkcd.com/1172/ Type annotations were meant to be used as linting, and other forms of type-checking done NOT AT RUNTIME. Gradually, some support for using them at runtime popped up, and them annotations starting being [ab]used for all sorts of things. PEP 563 (the one breaking pydantic here) is actually super important: it allows referencing things defined later. Like, a function on line 1 can take a type on line 10 as parameter. It is super necessary to make type annotations usable for their original purpose. It breaks pydantic, but also makes annotations usable for their intended purpose. Previously you had to manually opt it on each file, and now it's the default, as planned. I feel like pydantic and tools built on top of it were built ignoring the direction on which Python was heading. Much like the user in xkcd#1172.