11 ms·
Keep Pydantic out of your Domain Layer
- NeutralForest 1y agoWhat's the motivation for doing this? When does Pydantic in the domain model starts being an issue?
- halfcat 1y agoWhen the structure of your team makes it a problem. Conway’s law. If you have one person maintaining a CRUD app, splitting out DTOs and APIs and all of these abstractions are completely not needed. Usually, you don’t even know yet what the right abstraction is, and making a premature wrong abstraction is WAY worse. Building stuff because you might need it later is a massive momentum killer. But at some point when the project has grown (if it grows, which it won’t if you spend all your time making wrong abstractions early on), the API team doesn’t want their stuff broken because someone changed a pydantic model. So you start to need separation, not because it’s great or because it’s “the right way” but because it will collapse if you don’t. It’s the least bad option.
- NeutralForest 1y agoI'm not sure I agree, you can still use Pydantic in the domain model and update the version of the API when you change the expected schemas of your CRUD application. Where I'm with you, is that you should take care of your boundaries and muddling the line between your Pydantic domain models and your CRUD models will be painful at some point. If your domain model is changing fast compared to the API you're exposing, that could be an issue. But that's not a "Pydantic in the domain layer" issue, that's a separation of concerns issue.
- chausen 1y agoOften you want your domain models to be structured differently than API models, to make them as convenient/understandable to work with as possible for your use case. If you already have different models, why would you want Pydantic in the domain? Even if they start out the same, this would allow them to more easily evolve to be different. I'm not a python expert, so I could be missing the point on Pydantic, but it seems like its value is at the edges of your application.
- NeutralForest 1y agoThat's all fair, I just think it has more to do with separation of concerns than Pydantic and that the OP doesn't make it clear at all.
- rtpg 1y agoIn the Django world I have gotten very frustrated at people rushing to go from DRFs serializers to Django Ninja + Pydantic. You have way less in terms of tools to actually provide nice straightforward APIs. I appreciate that Pydantic gives you type safety but at one point the actual ease of writing correct code goes beyond type safety Just real straightforward stuff around dealing with loading in user input becomes a whole song and dance because Pydantic is an extremely basic validation thing… the hacks in DRF like request contexts are useful! I’ve seen many projects do this and it feels like such a step back in offering simple-to-maintain APIs. Maybe I’m just biased cuz I “get” DRF (and did lose half a day recently to weird DRF behavior…)
- zo1 1y agoThis is the Javascript hipster effect. FastAPI and Pydantic are pushed heavily because of their fancy docs page and the evangelism which thrives on reinventing the wheel. So we are all now stuck with everything being Pydantic this Pydantic that, instead of existing frameworks which are frankly better.
- WesolyKubeczek 1y agoIt's also because Pydantic has VC money and needs to grow fast now, or else.
- murkt 1y agoWhat’s the story here, can anyone enlighten me? How can they make money being a Python library? I can stretch my imagination about Astral monetizing their tools, but this one is too difficult
- stephantul 1y agoPydantic (the company) owns logfire, a logging service. There’s a lot of money in logging/observability. The pydantic library itself is not monetizable, as you indicate.
- 1y ago
- IshKebab 1y agoThis seems ridiculously over-complicated. This guy would love Java. He doesn't even say why you should tediously duplicate everything instead of just using the Pydantic objects - just "You know you don’t want that"! No I don't. The only reason I've heard is performance... but... you're using Python. You don't give a shit about performance.
- photios 1y agoThe "gain" TFA is describing is very very questionable too. You're losing a lot in terms of complexity. You're going from a straightforward "Pydantic everywhere" solution to a weird concoction of: 1. Pydantic models 2. "Poor man's Pydantic models" (dataclasses) 3. Obscure third party dependencies (Dacite) Thanks, I'll pass.
- pletnes 1y agoPydantic seems to be fast (in the context, it’s written in rust) so it might make sense to keep using pydantic for performance reasons.
- yedpodtrzitko 1y agoJust because it's written in Rust it doesnt mean it's fast. I was working on a project where Pydantic was the bottleneck - there were multiple levels of nested Pydantic objects, and creating the instances was very slow due to the validation which is performed on input values. Even after disablign the validation, dataclasses were twice as fast, compiling the dataclasses with mypyc improved the performance ten times.
- ensignavenger 1y agoWere you using v2? Pydantic docs do clearly state that multple levels of nesting of Pydantic objects can make it much slower, so it isn't particularly surprising that such models were slow.
- franktankbank 1y ago> you're using Python. You don't give a shit about performance. That's dumb. You may not care about max performance but you've got some threshold where shit gets obviously way to slow to be workable. I've worked with a library heavy on pydantic where it was the bottleneck.
- lysecret 1y agoThe core thesis is that your types received by the api should not be the same as the types you process internally. I can see a situation where this makes sense and a situation where this senselessly duplicates everything. The blog post shows how to do it but never really dives into why/when.
- r9295 1y agoPersonally, I think that's a good idea. Design patterns naturally make sense (Visitor, Builder for e.g) once you encounter such a situation in your codebase. It almost makes complete sense then. Otherwise IMHO, it's just premature abstraction
- roland35 1y agoNo one is satisfied with premature abstraction :(
- tetha 1y agoIt does touch on what I was thinking as well at the end of the first section: Usually this makes sense if your application has to manage a lot of complexity, or rather, has to consume and produce the same domain objects in many different ways across many different APIs. For example, some systems interact with several different vendor, tracking and payment systems that are all kinda the same, but also kinda different. Here it makes sense to have an internal domain model and to normalize all of these other systems into your domain model at a very early level. Otherwise complexity rises very, very quickly due to the number of n things interacting with n other things. On the other hand, for a lot of our smaller and simpler systems that output JSON based of a database for other systems... it's a realistic question if maintaining the domain model and API translation for every endpoint in every change is actually less work than ripping out the API modelling framework, which occurs once every few years, if at all? Some teams would probably rewrite from scratch with new knowledge, especially if they have API-tests available.
- AlphaSite 1y agoI’d say where it’s more Important is when you need to manage database performance. This lets you design an api that’s pleasant for users, well normalised internally, while also performing well. Usually normalisation and performance lead to a poor api that’s hard for users to use and hard hard to evolve since you’re so tightly coupled to your external representation.
- vjerancrnjak 1y agoJust have 1 input type and 1 output type. You don’t need more data types in between. If pydantic packages valid input, use that for as long as you can. Loading stuff from db, you need validation again, either go from binary response to 1 validated type with pydantic, or ORM object that already validates. Then stop having any extra data types. Keeping pydantic only at the edge and then abandoning it by reshaping it into another data type is a weird exercise. It might make sense if you have N input types and 1 computation flow but I don’t see how in the world of duck typing you’d need an extra unified data type for that.
- sgarland 1y ago> Loading stuff from db, you need validation again, either go from binary response to 1 validated type with pydantic, or ORM object that already validates. You shouldn’t need to validate data coming from the database. IMO, this is a natural consequence of teams abandoning traditional RDBMS best practices like normalization and constraints in favor of heavy denormalization, and strings for everything. If you strictly follow 3NF (or higher, when necessary), it is literally impossible to have referential integrity violations. There may be some other edge cases that can be difficult to enforce, but a huge variety of data bugs simply don’t exist if you don’t treat the RDBMS as a dumb KV store.
- vjerancrnjak 1y agoDepends. If you do a query that computes something, the output columns have data types that you’d like to validate. Checking that you receive an int, string or enum is unavoidable. Even a JOIN might surprise you with null values.
- sgarland 1y ago> Checking that you receive an int, string or enum is unavoidable. How would you be unaware of the data type if you defined the schema? Also, an ENUM is returned as a string; it’s only stored internally as an integer. > Even a JOIN might surprise you with null values. If you have foreign key constraints, you should never be able to get into a situation where you’re surprised by a NULL from an OUTER JOIN. You can certainly still have NULLs, but they shouldn’t come as a surprise.
- politelemon 1y agoThe reasoning given here is more academic than anything else. I'm not seeing any actual problem here though. Perhaps this could show how this is bad. Until then, I don't think this excessive duplication and layering is necessary, and is more of a liability itself. > That’s when concerns like loose coupling and separation of responsibilities start to matter more.
- deleted 1y ago[deleted]
- gostsamo 1y agoI'm sure that the pydantic guys had a reason to rename .dict to .model_dump. This single change caused so much grieve when upgrading to pydantic2.1 The very idea of unnecessary breaking changes is a big reason not to over rely on pydantic, tbh. 1 we were using .dict to introduce pydantic in the mix of other entity schemes and handling this change later was a significant pain in the neck. Some python introspection mechanism that can facilitate deep object recasting might've been nice if possible.
- jmogly 1y agoHaha, ChatGPT recommends this: from pydantic import BaseModel class MyModel(BaseModel): name: str def dict(self, *args, **kwargs): return self.model_dump(*args, **kwargs)
- gostsamo 1y agoYep, and when you are done migrating, you need to remove this, and there is pydantic3 coming. Keeping in mind the number of libraries nad microservices involved, search and replace was the easier option. PS: thank you, I can think on my own and even failing that, chat gpt is not in closed beta any more.
- the__alchemist 1y agoRepresenting structured data as key/value pairs is a pattern I've only seen in Python, and don't understand why it became popular and canonical.
- halfcat 1y ago> Representing structured data as key/value pairs is a pattern I've only seen in Python Come on. We know you’ve seen JavaScript.
- brap 1y agoI’m far from being an experienced Pythonista, but one thing that really bugs me in Python (and other dynamic languages) is that when I accept an input of some type, like User, I have to wonder if it’s really a User. This is annoying throughout the codebase, not just the API layer. Especially when there are multiple contributors. The argument against using API models internally is something I agree with but it’s a separate question.
- jon-wood 1y agoI’m curious, what do you mean by having to wonder if it’s really a User? It’s optional in Python but you can use type annotations and then the type checker will shout at you for passing something that’s not a User instance to things that expect one.
- padjo 1y agoPython has reasonably good types these days. If you were to use pydantic to Marshall stuff from the API and then put type annotations on every method below that it would be pretty bulletproof.
- derriz 1y agoI've been using Python on and off for a few decades and agree. I don't know why you're being downvoted. I've authored tens of thousands of lines of Python code in that time - both for research tools and for "production". I use type hints everywhere in the Python I write but it's simply not enough. This issue is political and not so much technical as Typescript demonstrates how you can add a beautifully orthogonal and comprehensive type system to a dynamic language, thus improving the language's ergonomics and scaleability. The political aspect is the fact that early Python promoters decided that sanity checking arguments was not "pythonic" and this dogma/ideology has persisted to this day. The only philosophical basis for this position was that that Python offered no support for simple type checking. And apparently if you didn't/don't "appreciate" this philosophy, it reflected poorly on your software engineering abilities or skill with Python. To be fair, Python isn't the only language of that era, where promoters went to great lengths to invent alternate-reality bubbles to avoid facing the fact that their pet language had some deep flaws - and actually Perl and C++ circles were even worse and more inward facing. So the "pythonic" approach suggests having functions just accepting anything, whether it makes sense or not, and allowing your code to blow up somewhere deep in some library somewhere - that you probably didn't even know you're using. So instead of an error like "illegal create_user(name: str) call: name should be a str but was a float", it's apparently better (more "pythonic") to not provide such feed-back to users of your functions and instead allow them to have to deal with an exception in a 40 line stack trace with something like "illegal indexing of float by dict object" in some source file library your users haven't even heard of.
- dgan 1y agoi have to confess , i use Protobuffs for everything. They convert to pure python (a la dataclass), to json strings and to binary strings, so i literally shove it everywhere : network, logic, disk. BUT when doing heavy computation (c++, not python !) don't forget to convert to plain vectors, Protobuffs are horribly inefficient
- the__alchemist 1y agoProtobuf is fine if: A: You control both ends of the serialized line, or: B: The other end of the line expects protobufs. There are many [de]serialization scenarios where you are interfacing with a third party API. (HTTP/JSON web API, a given IC's comm protocol as defined in its datasheet etc)
- dontlaugh 1y agoYou can still use a protobuf schema to parse/generate JSON, in most cases.
- dgan 1y agoi think even if 3rd party API expects json, you could still map their models to proto ; i haven't encountered this case tho might still be challenging to convince proto to output what you want exactly
- the__alchemist 1y agoI don't understand then. Here is my mental model; as described, you can see why I'm confused: JSON: UTF-8 Serialization format, where brackets, commas, fields represented by strings etc. Protobuf: Binary serialization format that makes liberal use of varints, including to define field number, lengths etc. Kind of verbose, but not heinous. So, you could start and end your journey with the same structs and serialize with either. If you try to send a protobuf to an HTTP API that expects JSON, it won't work! If you try to send JSON to an ESP32 running ESP-Hosted, likewise.
- 1y ago
- leoff 1y ago>The less your core logic depends on specific tools or libraries, the easier it becomes to maintain, test, or even replace parts of your system without causing everything to break. It seems like the author doesn't like depending on `pydantic`, simply because it's a third party dependency. To solve this they introduce another, but more obscure, third party dependency called `dacite`, that converts `pydantic` to `dataclasses`. It's more likely that `dacite` is going to break your application, than `pydantic`, a library used by millions of users in huge projects, ever will. Not to mention the complexity overhead introduced by this non sense mapping.
- wiseowise 1y ago> simply because it's a third party dependency Not simply. This is one one of the most important reasons NOT to propagate something through your code. How many millions codebases use it is irrelevant.
- leoff 1y ago>How many millions codebases use it is irrelevant. It is relevant, because it speaks to the reliability of the dependency. `pydantic` has 24.7k Github stars and was last updated 52 minutes ago. Adding a random dependency `dacite`, which has 1.9k Github stars, no one has ever heard of, and was last updated 4 months ago, introduces way more complexity and sources of instabilities than propagating `pydantic`.
- murkt 1y agoMore updates means more changes and more instability. I have never seen dacite, but it’s pretty easy for a small library to just be complete. If it’s complete, why the need for constant changes?
- Lucasoato 1y agoActually Pydantic could be extremely useful if used in conjunction with SQLAlchemy, check out the SQLModel library, from the very same creators of Pydantic.
- jessekv 1y agoSebastián Ramírez created FastAPI and SQLModel, and was an early adopter of Pydantic. Samuel Colvin created Pydantic.
- cout 1y agoHaving used sqlmodel recently for a project, I was underimpressed. Documentation was sparse, I found myself going to the source code to figure out how to solve problems I ran into, and I ended up dropping into sqlalchemy a lot more than I wanted. I think the idea is sound, but the code is hard to follow, and there are a lot of missing common cases.
- iloveitaly 1y agoI agree. It still needs a lot of work and is a far cry from ActiveRecord. I've been working on a wrapper around it to make it more usable, hoping to get as much of the work merged upstream into SQLModel: https://github.com/iloveitaly/activemodel https://github.com/iloveitaly/activemodel Unfortunately, I just don't see a better option than SQLModel for ORM at this point.
- JackSlateur 1y agosqlmodel is a wrapper around sqlalchemy, made by the guy who made fastapi While it uses pydantic, sqlmodel has not been written by those guys
- stephantul 1y agoI think this article misses the main point by focusing on removing pydantic. The main point is that you should convert external types as soon as possible to decouple them from the rest of your code. Whether this involves pydantic or something else is not really important I guess
- nisten 1y agoFrom the article: "Why are there no laws requiring device manufacturers to open source all software and hardware for consumer devices no longer sold?" I think it's because people (us here included) love to yap and argue about problems instead of just implementing them and iterating on solutions in an organized manned. A good way these days to go about it would be to forego the facade of civility and use your public name to publicly tell your politician to just fuck it, do it it bad, and have plan to UNfuck after you fuck it up, until the fucking problem is fucking solved. Same goes for UBI and other semi-infuriating issues that seem to (and probably do) have obvious solutions that we just don't try.
- barbazoo 1y ago> But Pydantic is starting to creep into every layer, even your domain, and it starts to itch. I can’t relate yet. Itch how? It doesn’t really go into what the problem is they’re solving.
- ansc 1y agoIt expands further down in the article: >Pydantic is great, just not everywhere. [...] Not because it’s bad, but because your domain should be pure and independent. It itches because it should be pure and indepen- yeah I don't know. I haven't had this itch either to be frank.
- karolinepauls 1y agoI'll go further and elsewhere at once: APIs should not present nested objects but normalised data. It enables clients to easily to lay out their display structure independently of API resource schemas and eases out tricks like diffing between subsequent responses, pulling updates or requesting new data by passing IDs and timestamps of already known data, etc. API normalised data obviously shouldn't correspond to DB normalised data. Nested objects are superior only for use with jq.
- ejflick 1y ago> APIs should not present nested objects but normalised data If something is nested, let it be represented as a nested structure. I find flattening causes more mental overhead. If something is too flat, it becomes less obvious what data is exactly necessary to do what you want to do
- mindcrash 1y agoAnd that's why it is key in your architecture to differentiate between Data Transfer Objects (DTOs) or Models on one hand which has values which can and actually must be validated when they come from the outside, and Domain Entities / Value Objects on the other. Even though the DTO and Domain Entity might look similar. Thank me later.
- ripped_britches 1y agoThis persons’s head would explode if they saw what we’re doing over here in typescript with structural typing. It would make things way too simple.
- axpy906 1y agoThe trouble I have with pedantic is that everything is immutable. There are use cases where I need mutability and it’s not bad but a trade off.
- henning 1y agoOh boy, I love making adding a trivial nullable column take even more code and require even more tests and have even more places I forgot to update which results in a field being nullable somewhere. And don't forget, you get to duplicate this shit on the frontend too. And what is a modern app if we aren't doing event-driven microservice architecture? That won't scale!!!! So now I also have to worry about my Avro schema/Protobufs/whateverthefuck. But how does everyone else know about the schema? Avro schema registry! Otherwise we won't know what data is on the wire! And so on and so on into infinity until I have to tell a PM that adding a column will take me 5 pull requests and 8 deploys amounting to several days of work. Congratulations on making your own small contribution to a fucking ridiculous clown fiesta.
- jmward01 1y agoStrongly decoupling API implementation and, well, actual implementation, is pretty key when you start to evolve an application. People often focus on 'the design' like there is one perfect design for an application for its lifetime when in really it is about how easy the mass of code you have is able to change for the next feature/fix/change and not turn into a hairball of code. That perfect initial design where the internal and external objects are exactly the same generally works well for 1.0, but not 1.1 or 2.0 so strongly decoupling the API implementation is a good general practice if you think your code will continue to evolve.
- clickety_clack 1y agoI use pyrsistent in the domain, and pydantic for tricky validation at the boundary. Pyrsistent is a pretty neat solution if you want immutable data structures, with some nice methods for working with nested records.
- golly_ned 1y agoI still don’t quite get the motivation for “don’t use pydantic except at border” — it sounds like it’s “you don’t need it”, which might be true. But then adds dacite to translate between pydantic at the border and python objects internally. What exactly is wrong with pydantic internally too?
- chausen 1y agoCould be wrong, never used Pydantic. But looking it up it seems like it's used for validation/typing of external data. Sounds like it's mainly going to be doing schema validations. So, your data arrives at your domain layer and you have guarantees based on Pydantic's validations. At this point, your validations are going to semantic in nature based on your domain; what value is Pydantic bringing?
- ac130kz 1y agoAn easier/moderate approach: make a proper base DTO model, which can be extended by validators, such as Pydantic, and the db model is the Domain is just whatever an ORM offers/dataclasses.
- throwaway7783 1y agoReturn of Java DTOs!
- talos_ 1y agoYou should checkout the Python framework Litestar. It's an alternative to FastAPI that implements these ideas via their "Data Transfer Object" concept
- scolvin 1y agoPydantic creator here - I kind of agree with the article. I (obviously very biased) use Pydantic a fair bit, but there are places where it's the wrong tool and I use dataclasses, typeddicts or even tupleS a fair bit. Sad about the Pydantic hater jumping on this to suggest it means you shouldn't use Pydantic at all, but I guess success (even open source success where you pay us nothing) comes with haters. Thanks for the article.