11 ms·
Taming the beast that is the Django ORM – An introduction
- deleted 2y ago[deleted]
- cqqxo4zV46cp 2y agoDjango’s ORM is the first one that I ever spent a lot of time with. Throughout my career I’ve interacted with other ORMs from time to time. It wasn’t until I’d done that, that I realised, even though it’s not perfect, how fantastic the Django ORM is. I thought they’d all be that good, but no. I’ve read a lot of criticisms of ORMs, as I’m sure everyone else has. Some of them are certainly valid criticisms that are immovable and just inherent in what an ORM tries to do. Some of them just seem to be caused by not very many ORMs being good, and the writer not having used one of the better ones.
- 1propionyl 2y agoThis is generally my experience as well. I don't love using ORMs but at least Django's is relatively painless and I can generally find the escape hatch I need. The lack of support until very recently (and it's still lacking) for views is the main knock.
- winrid 2y agoI agree. Django's ORM is great because it handles the relationships well, where many ORMs barely do the object mapping part well. I'm rewriting a large Django project in Java (quarkus + jooq), because it's at the point where I need a type system now, but it still has a place in my heart.
- BarryMilo 2y agoAren't Django models close enough to types?
- wg0 2y agoNope. Python type hinting is far far from Java like types. And yes, I guess OP has now a large system that needs types enforced by the system to reduce the friction in evolving the stack.
- roze_sha 2y agoWhat about using pydantic?
- mehdix 2y agoFWIW, one python project I'm working on uses an obscure Framework, and ORM. I was contemplating to convert it to FastAPI+Pydantic, however the amount of effort needed was no different than rewriting the whole project.
- LtWorf 2y agopydantic + sql database? So have 2 components that ensure the types are correct? Why?
- winrid 2y agoI already use type annotations with Python for use within my IDE. It's just all tiring garbage. Java is already almost a scripting language and I can actually use a shared heap, etc. also I have some core code in java I use in the desktop app I want to reuse in the server. Right now I transpile that core code to TS and then JS to use client side, but I'd rather just have everything in Java.
- Izkata 2y agoIf this is what you're referring to: > Note: The Python runtime does not enforce function and variable type annotations. They can be used by third party tools such as type checkers, IDEs, linters, etc. Then yes, Django model definitions are more like Java types in that they error if you try to use an incorrect type. You can't just ignore them like with type hints. They also try to be 1:1 with database types, so for the most part any additional validation added on top of Django would be something you had to do anyway.
- Swizec 2y agoFor me Django and ActiveRecord stand out as 2 good examples of what an ORM should be like. Both feel like they make the simple stuff super easy, the complex stuff figure outable, and the super hard stuff trivially possible with raw SQL and a decent mapping from that back to regular code. Although over the years my code trends more and more towards `.rawSql` or whatever equivalent exists. Even for the simple stuff. It’s just so much easier than first thinking up my query then bending over backwards three times to make it fit into the ORM’s pet syntax. Plus raw sql is so much easier to copypasta between different tools when you’re debugging. And before you say “but sql injection!” – that’s what prepared statements/parametrized queries are for.
- wsc981 2y agoI've started working on a .NET project that uses EntityFramework and due to some of the magic, I prefer just using raw SQL as well. I've used other ORM solutions in the past as well and I am not a fan ... But the team chose EF due to it supposedly being easier to integrate with whatever database the customer might be using and that seems like valid reasoning.
- djeiejejejej 2y agoYes I agree. That’s a valid argument, but there are other ways as well like SQLKata and other query builders. Depending on the complexity of the product I think writing standard SQL92 will get you far as well. I don’t get how an ORM will handle Postgres’ CTEs and windowing functions. Those are pretty basic and extremely useful features to me, but those require dropping to raw SQL. Each vendor has those useful particularities that you immediately lose by going ORM. So in practice you are already required to make your queries as bland as possible where GROUP BY and JOIN is about as complex as it will get. I’d say just do SQL92 work around the limitations - which is something you’ll have to do anyway, but now more directly - and all major vendors will work OOB.
- Izkata 2y ago> I don’t get how an ORM will handle Postgres’ CTEs and windowing functions. Those are pretty basic and extremely useful features to me, but those require dropping to raw SQL. Each vendor has those useful particularities that you immediately lose by going ORM. Django has supported windowing functions since 2.0, almost 7 years ago: https://docs.djangoproject.com/en/2.0/ref/models/expressions/#django.db.models.expressions.Window https://docs.djangoproject.com/en/2.0/ref/models/expressions... And people are experimenting with CTEs - this looks pretty natural to use, at least with the basic example, and it supports recursive CTEs even if the example looks clunky (though I think that's their choice of example rather than the actual recursive part making it look bad): https://dimagi.github.io/django-cte/ https://dimagi.github.io/django-cte/ (repo: https://github.com/dimagi/django-cte https://github.com/dimagi/django-cte)
- choilive 2y agoDjango's ORM is acceptable - but I think Rails/ActiveRecord is superior albeit certainly more opinionated. Most likely just my personal bias speaking because Rails was the first web "framework" I cut my teeth on.
- wg0 2y agoI think Django's ORM is really great but only as long as that's the first thing you have seen. If you started from Rails and ActiveRecord, you're probably not be very appreciative of Django's ORM.
- bitexploder 2y agoSQLAlchemy is the best stand alone ORM I have found.
- baq 2y agosqlalchemy is easily one of the best ORMs ever made for any language. Hats off to zzzeek. Currently working in typescript and writing SQL queries out by hand since I just don’t trust anything to do the right thing wrt units of work.
- JodieBenitez 2y agoI don't know, the declarative API is awkward. I get SQLAlchemy is quality software but I found DX poor.
- greenie_beans 2y agoi've used both. they both have pros/cons. use whatever tool your team is most proficient with/stop wasting your time arguing about which framework is best
- malux85 2y agoI love the django ORM and then the REST framework on top of that. Define the object, hook it to a view, use our custom permission class, done. GET, POST, PATCH, DELETE all urls, filtering, pagination, search, ordering, complex multi-org multi-user object permissions handled automatically. Need custom logic? Just overwrite method. It’s a productivity superpower, sure there’s a lot to learn but that’s the price you pay for the tools that make you hyper productive
- nprateem 2y agoNinja is even better than the REST framework. Practically no boilerplate.
- Izkata 2y agoThe awkward part is realizing Django has been this good for well over a decade. The core design hasn't (ever?) changed. I started using Django on I think version 1.2 or 1.3 in 2011, back when it didn't have database migrations and you had to use a library like South for it. Even then, as an ORM/query language it was apparently better than what other languages have now.
- simonw 2y agoThe design of the ORM changed substantially once, early in Django's history, when the "magic removal branch" landed. There are some notes on the new syntax that introduced here: https://code.djangoproject.com/wiki/RemovingTheMagic#Codechangesyoullneedtomake https://code.djangoproject.com/wiki/RemovingTheMagic#Codecha... That branch merged on May 1st 2006: https://www.djangoproject.com/weblog/2006/may/01/magicremoval/ https://www.djangoproject.com/weblog/2006/may/01/magicremova... I've long found Django's commitment to not breaking too much at once inspiring. The release notes and upgrade guides are really solid.
- vandahm 2y agoI used Django a lot from about 2007 to 2010 and, then, went for several years without using it at all. When I came back to it, I was delighted to find that everything still worked like it was supposed to, just better. Congrats on getting it right on the first try. That's...not something that happens very often in software.
- JodieBenitez 2y ago> I thought they’d all be that good, but no. I had the "pleasure" to use Doctrine once. Never again !
- vandahm 2y agoI used Doctrine for a few years, and I remember thinking that it was about 50% awesome and 50% terrible. I wonder what I would think if I came back to it today.
- murkt 2y agoIn 15+ years I’ve only used two orms: Django ORM and SQLAlchemy. I’ve also skimmed docs for many others (some JS orms, etc), and those often look completely unusable. SQLAlchemy is leagues above and beyond Django ORM. I can’t say I have nightmares from dealing with it, but it certainly was not pleasure. A bit too much magic here and there, not flexible enough and provokes too much bad practices. Re-defining save() methods to do a lot of work, anyone? The best “orm” that I’ve ever used was not an ORM but a query builder — HoneySQL from Clojure. That one is fantastic. No ORM bullshit, no extra data loaded from DB, no accidental N+1 queries. Not reading docs on how to do a GROUP BY, everything’s just easy to write. Frankly, we often use SQLAlchemy as just a query builder too. Makes life so much easier.
- WesleyJohnson 2y agoPretty decent introduction. Will there be additional parts that cover how to create GROUP BY queries in the ORM? I find even seasoned Django developers struggle with these. Also, I believe your code for creating an empty "data" migration is missing the "makemigration" command itself.
- vandahm 2y ago> Will there be additional parts that cover how to create GROUP BY queries in the ORM? I find even seasoned Django developers struggle with these. The last time I needed to do that, I ended up crying "uncle" and writing manual SQL. I wasn't happy about doing it, but I was happy that the framework left me an escape hatch so that I could.
- Too 2y agoIt's actually easy once you get it, it's just that they choose bizarre terminology that doesn't translate directly from sql. .values() and .annotate() ?!
- AbundantSalmon 2y agoGood topic for a second blog post/skill share. Are there any other interesting topics you can think of? I think I know so much but in it too deep to remember what is unusual. These came up with the recent onboarding of some new devs. Thanks, I will update the data migration code!
- nprateem 2y agoThis is largely academic now. LLMs do a good job of writing highly complex queries with the django ORM. All you need is the django toolbar so you can check their efficiency, then keep telling it to make them more efficient.
- ruthmarx 2y agoLLMs are not at a point where we should be treating them as a solution to any software engineering issue, period.
- 9dev 2y agoYet people do, will continue doing so, and it works. Trying to stuff the toothpaste back into the tube isn’t going to work, insisting on people just using plain text mail, or stop calling the pound symbol hashtag. It’s just old men yelling at clouds.
- nprateem 2y agoI've used it to write horrendously complex queries across multiple tables. It can do things I don't know how to and that would have taken significant time to learn. All I know is it works and it's performant when you tell it to be. It's been particularly helpful for aggregations, subqueries etc. You may not think LLMs are there yet but my project is proof. Try it yourself.
- LtWorf 2y agoSpending time to learn? Nah better never learn and put together barely working stuff that nobody dares to change! This isn't something new.
- ruthmarx 2y ago> It can do things I don't know how to and that would have taken significant time to learn. All I know is it works Until it fucks up and you have no clue how to fix it, and either does the LLM.
- nprateem 2y agoFor me the killer feature of django is the auto-generated admin UI. I initially started my last project using Spring boot, but I was astonished to find there was no equivalent. I don't know how people build websites in any speed without such a tool. I guess they just waste time duplicating effort on an admin UI or pay for one of those tools that can generate one from an API (meaning they have to also build an API). It's such a massive time-saver I switched to django after a week or so.
- savolai 2y agoI used to think this too. LLms make generating such UIs in other frameworks too much easier. Also the admin UI gets clumsy very fast and you’ll need to roll your own for a good UX.
- Ingaz 2y agoI would say even more: - the main reason to use django today is the admin UI - and the only reason to use django ORM is the admin UI
- switch007 2y agoIt can be quite difficult to do anything mildly advanced with the admin UI, IIRC And it encourages CRUD thinking instead of thinking about business processes and user experience It's great for tiny/personal projects but in an organisation it can be a trap IMO
- yurishimo 2y agoIt depends on what kind of app you’re building. The Django UI could be okay for your business use case, but for many businesses, it is not. If I have to write the entire admin UI for some specific business reasons/workflow, then I might as well dogfood those components for the needs of our dev team.
- 9dev 2y agoI don’t really understand this, I was very much underwhelmed by the admin UI, and feel like I’m missing something? I mean what does it do other than list records and provide some basic forms to CUD those? It feels like much of the value in that is achieved even better by using a proper database client, say, JetBrains‘ database integration. I certainly wouldn’t let any business people touch a Django admin panel, or at least those I’ve seen built by our data guys. Way too technical. And don’t even get me started on extensibility. I wanted to have like, an additional page to include a Grafana iframe plus a bunch of buttons. Something that would take me about five lines in Laravel, for example. Good luck even searching the documentation for this…
- rowanseymour 2y agoI adore the Django ORM but as listed under cons... it makes it very hard to avoid accidental N+1 queries, and they don't seem interested in addressing this (https://code.djangoproject.com/ticket/30874 https://code.djangoproject.com/ticket/30874). Yes lazy loading is neat when you're messing around on the shell, but you should absolutely not be leaning on it in production at any kind of scale. So instead you have to use unit tests to hopefully catch any N+1 queries.
- wruza 2y agoWhat would be the best way?
- rowanseymour 2y agoIf it were up to me there'd be a way to completely disable it globally for all models. Let me explicitly enable it when I'm just shelling around or checking results in unit tests. It is not a feature for production environments.
- geraneum 2y agoWould auto-joins be better in production envs?
- yurishimo 2y agoLaravel does this (or at least has a config for it). You can disable lazy loading in non-production environments which will throw fatal errors whenever a model is lazy loaded. The nice part is, in prod, that code _allows_ lazy loading since application stability is more important. Hopefully at that point you have a good performance monitoring tool that will alert you to that problem. Laravel also has hooks to more granularly fire events about lazy loading if you want to roll your own notification solution. So I disagree with your assertion. Lazy loading is ONLY for production. While it is possible that such a feature could potentially bring down your DB server and therefor your app in general, if the feature is turned off locally, then hopefully you’re catching it well ahead of time. And if you’re running small app with no users, meh. For large apps, hopefully your team’s standards are enough to keep the monsters at bay. Lazy loading is a sharp knife that requires skill to use appropriately.
- abc-1 2y agoORMs do a great job at making easy things even easier and hard things a lot harder. If that sounds like a bad deal to you- it’s because it is!
- rowanseymour 2y agoThe Django ORM makes migrating database state (a hard problem) super easy. It also makes GROUP BY queries (a relatively easy problem) oddly difficult. Use an ORM where it helps, use SQL where it doesn't.
- AmericanChopper 2y agoI personally don’t think DB migrations are that difficult, and I’d say they can be done a lot safer without an ORM. I’ve also never found an ORM that was easier to learn and overall easier to use than SQL. I’ve never understood how ORMs became so popular, I don’t get how somebody can look at how opaque and complicated they are and conclude that learning an ORM is easier than just learning SQL.
- thestepafter 2y ago100%. ORMs are good for basic queries but the messes I have seen written joining 20 plus tables with conditions and left joins, etc, oof. Just use SQL, it’s much cleaner and easier to maintain.
- nprateem 2y agoBecause it's not about avoiding SQL (although that's a benefit), it's that you also get forms for free, view classes, easy APIs, validation, a free admin, free docs and a large pool of potential employees who all understand the foundations of your app.
- AmericanChopper 2y agoYou don’t get any of that for free. You get it at the cost of working with the abstractions, which can be very high. Your RDBMS is not an OOP/multi-paradigm system like Python is (or whatever other language). Its objects interact differently, and it has different access patterns. Any system you use to try and ignore these differences is just going to create a host of other problems for you, especially this idea that it will allow you to simply ignore database administration.
- ebcode 2y agoAnytime I hear ORM, I always think of this: https://blog.codinghorror.com/object-relational-mapping-is-the-vietnam-of-computer-science/ https://blog.codinghorror.com/object-relational-mapping-is-t...
- eru 2y agoYes. Object orientation is the problem here. Relations are (mostly) great to model business logic with. The article you linked mentions this: > This eliminates the problem quite neatly, because if there are no objects, there is no impedance mismatch. But: > Integration of relational concepts into the languages. Developers simply accept that this is a problem that should be solved by the language, not by a library or framework. > Over the last several years, however, interest in "scripting" languages with far stronger set and list support, like Ruby, has sparked the idea that perhaps another solution is appropriate: bring relational concepts (which, at heart, are set-based) into mainstream programming languages, making it easier to bridge the gap between "sets" and "objects". I agree that bringing relations into your language is a good thing, but many languages are strong enough to do that as libraries, there's no need for baking support into your language. Just as hash-tables can be implemented as a library in most languages. We have relations as a library in a Haskell dialect I was working with in a previous job. Worked like a charm, and no language support was needed. Python would also be flexible enough to use relations as a user-defined datatype. And so would many other languages.
- raziel2p 2y agoI tried reading and understanding the arguments made here, but just could not make any correlation with my day to day experience using ORMs. is it possible that the fact this article is written in 2006 simply makes it dated? it seems very catastrophizing but we've come a long way and are just more aware of how to work with or around the shortcomings and flaws of ORMs. I've often written programs where I'm trying to encapsulate pure in memory state into business objects and run into the same type of issues people complain about with ORMs. programming is just hard, we don't have to be such dogmatists about it.
- ebcode 2y ago
- santiagobasulto 2y agoI think Django's ORM is just AMAZING. And as with every other tool, it has to be used wisely. First, it let's you get started quickly and prototype. You can write unit tests, make sure everything is working as expected, then count queries and make sure you're being efficient with your SQL engine. Second, and even more importantly, it's crucial in the definition of the app and the Schema. Thinking in high level "classes and objects" helps with the abstraction and the design of the apps. Even if you then default to raw SQU queries, thinking and building your model with class abstractions is huge. Finally, there are some "tiny details" (but in my eyes, very important) that everybody oversees: * Migrations: the way Django has designed migrations is just marvelous. We've migrated tons of data and changed the structure of our DB multiple times without any issues. * Troubleshooting and "reporting": the ability to fire a quick shell, load a few models in Django's REPL, and do a few `.filters()` is for me key. On top of that, we add a Jupyter server connected to our read replica and we can do all sorts of reporting VERY QUICKLY. Not everybody needs a Data Lake :) PS: We've ran Django sites accessed by millions of people per month. And we never had an issue with the ORM. Yes, sometimes I have to tell more junior devs that the gymnastics they're doing with `select_related` and `prefetch_related` can be more easily and effectively resolved with a query. But that's it. I'd say less than 1% of all the queries in our codebase have to be migrated to raw SQL.
- santiagobasulto 2y agoOne common pitfall with Django ORM's is that it makes it very easy the use of inheritance in models. But as we know, the "impedance mismatch" between OOP and the Relational model is a problem. It has happened that a dev in our team was populating an endpoint that used inheritance and when looking at the number of queries we were over 100. But the solution in those cases is just OUTER JOINs and CASE. Especially since we use Postgres in the backend and it works so well. So yes, there are some pitfalls, but they're greatly overshadowed by the advantages of the ORM.
- andybak 2y agoYou have to use the ORM with a bit of awareness of the query it it generates. Django provides plenty of tools and great documentation for reducing, combining and managing queries. It's not like it's a deep dark corner. It's like understanding memory allocation if you're writing c. It's just part of the job
- ninetyninenine 2y agoorms are exercises in OCD. Databases are the bottleneck your classic website. We choose to query these databases in a extremely high level language called SQL. This language is so high level that you need to come up with tricks and query analyzers in order to hack the high level query into something performant. A better abstraction would be one that's a bit more similar to a standard programming language with an std that has query related operations/optimizers that can be composed so programmers can compose and choose query operations and avoid optimization issues that are hidden by high level languages like SQL. We are unfortunately, sort of stuck with SQL (there are other options, but SQL remains popular because years of development has made it pretty good in spite of the fact that it's a poor initial design choice). This is already a historical mistake that we have to live with. Same with javascript (which has been paved over with typescript), same with CSS, etc. The web is full of this stuff. It's fine. My main problem is the ORM. The ORM is just another layer of indirection. You already have a high level language you're dealing with, now you want to put Another High level language on top of it? ORMs are basically high level languages that compile into SQL. What is the point? The ORM isn't actually making things easier because SQL is pretty much as high level of a language you can get outside of having an LLM translating direct english. The point is OCD. Programmers find it jarring to work with SQL strings inside their beautiful language so they want to chop up a SQL string into web app language primitives that they can compose together. Query builders operate on the same concept but are better because they aren't as high level. This is basically the main reason why Many programmers experience the strange counter intuitive phenomena about why ORMs actually makes things harder. You have to Hack SQL to optimize it. Now you have to hack another ORM on top of it in order to get it to compile it into the hacked query.
- 9dev 2y agoYou are correct in your entire assessment, yet, you seem to underestimate the number of boring CRUD applications serviced by mediocre programmers. Limiting the number of technologies required to have at least a little knowledge in is a benefit, even if it hampers performance, because performance doesn’t matter for the vast majority of cases. Software engineers tend to overvalue that bit; the users of line-of-business apps don’t have a say anyway, but what matters is quick adaptability to changes in business logic. So having an ORM in place that is tightly integrated in Python and Django lets even the junior developer fresh from the bootcamp make changes to an existing application. It’s not a pretty story, but in my opinion the reason for staggering layers of complexity is the ability to move quickly even without experts on the team.
- v3ss0n 2y agoWhy Django ORM is considered a beast? It is easiest ORM to date and very convenient API to work with. If you think Django ORM is a beast, try SQL alchemy
- Kalanos 2y agoI used sqlalchemy once. I found it more complicated with way less functionality
- rollcat 2y agoAs someone who's had numerous grand battles with both, I find that SQLAlchemy's overall design philosophy gives you more precise control over both the generated SQL, and the lifecycle of each query/transaction/connection; you get to choose exactly how much abstraction you need for the task, but it won't hide things from you where you MUST make a conscious decision. I find it's also a better choice of the two, if you're writing something that has absolutely NOTHING to do with web whatsoever (e.g. a DB-backed desktop app), as you don't have to drag in a whole kitchen sink just to drink a glass of water; but you may also want to investigate something lighter than SQLAlchemy for that use-case. Meanwhile Django's ORM just does the job perfectly fine in the 99% common case, and for the 1%, again 99% of the time it has enough escape hatches to solve the hard problems without making things excessively complicated. But $DEITY save you if you're in that 0.01%...
- andrewstuart 2y agoI thought Django’s ORM was awesome (seriously). And yet using it was such a shit experience I switched permanently to writing SQL. I see zero advantage to using an ORM over SQL and in fact see many downsides. Don’t use an ORM just learn SQL. It’s faster, more direct, more powerful, easier to understand, allows you to make full use of the power of the database and your knowledge isn’t suddenly valueless when you go to a project with a different ORM.
- DarkNova6 2y agoDepends on what you are doing. A good ORM lets you create entities from tables, or creates tables from entities. Good ORMs make sure your data model is always aligned with the DB, and there is no way in hell a runtime error can occur. What's more, it automates the mindless data class/table matching for 99% of your use-cases. Once you got a more sophisticated query, please use SQL. But for simple stuff, use ORM.
- 9dev 2y agoI’ve used several ORMs and experienced the same things everyone else does—CRUD is great, but good luck with your first actual reporting dashboard. Then I tried the new generation of typescript SQL query builders that automatically infer types from your database and provide end-to-end type-safe query results, even for highly complex queries. And since then I became convinced the answer isn’t trying to shoehorn classes on top of relational tables, but having a sufficiently integrated and capable query builder that provides properly typed results.
- listenallyall 2y agoYou'd probably really like jooQ, building queries with typed classes and returning strongly-typed results. Of course, it runs on the JVM where languages have actual types, not a facade over an untyped language.
- Kalanos 2y agoPeople focus too much on the query aspect of an ORM. Even though you can still write raw query strings in an ORM if you want to. Routes, form validation, REST API, templating (if you don't need react), auth, etc. You'll probably wind up recreating a lot of ORM and surrounding functionality at lower quality