3 ms·

I adore the Django ORM but as listed under cons... it makes it very hard to avoid accidental N+1 queries, and they don't seem interested in addressing this (htt

I adore the Django ORM but as listed under cons... it makes it very hard to avoid accidental N+1 queries, and they don't seem interested in addressing this (https://code.djangoproject.com/ticket/30874 https://code.djangoproject.com/ticket/30874). Yes lazy loading is neat when you're messing around on the shell, but you should absolutely not be leaning on it in production at any kind of scale. So instead you have to use unit tests to hopefully catch any N+1 queries.

wruza 2y ago
What would be the best way?
- rowanseymour 2y ago
  If it were up to me there'd be a way to completely disable it globally for all models. Let me explicitly enable it when I'm just shelling around or checking results in unit tests. It is not a feature for production environments.
  - geraneum 2y ago
    Would auto-joins be better in production envs?
  - yurishimo 2y ago
    Laravel does this (or at least has a config for it). You can disable lazy loading in non-production environments which will throw fatal errors whenever a model is lazy loaded. The nice part is, in prod, that code _allows_ lazy loading since application stability is more important. Hopefully at that point you have a good performance monitoring tool that will alert you to that problem. Laravel also has hooks to more granularly fire events about lazy loading if you want to roll your own notification solution. So I disagree with your assertion. Lazy loading is ONLY for production. While it is possible that such a feature could potentially bring down your DB server and therefor your app in general, if the feature is turned off locally, then hopefully you’re catching it well ahead of time. And if you’re running small app with no users, meh. For large apps, hopefully your team’s standards are enough to keep the monsters at bay. Lazy loading is a sharp knife that requires skill to use appropriately.
    - rowanseymour 2y ago
      Hmm I would call lazy loading the opposite of a sharp knife that requires skill.. you're not being explicit about database work. You're relying on the framework to bail you out where you've forgotten a fetch. And by default Django will do that silently so missing prefetches easily go undiscovered.
- andreareina 2y ago
  Static lint wouldn't be a bad way
- stavros 2y ago
  There's a Django app (that I forget the name of) that disables lazy loading in templates, thus requiring you to load everything explicitly in the view.
  - rowanseymour 2y ago
    I think there's a few 3rd party solutions like https://github.com/charettes/django-seal https://github.com/charettes/django-seal but I don't love the idea of using something that I assume is monkey patching Django code.
    - charettes 2y ago
      There is an ongoing effort to include back the equivalent in core [here](https://github.com/django/django/pull/17554 https://github.com/django/django/pull/17554)
LaundroMat 2y ago
Just to say there are libraries to help you find n+1 queries too (when your code is running). I use https://pypi.org/project/django-nplusone/ https://pypi.org/project/django-nplusone/ for instance. Sentry also warns of these by the way.
ruthmarx 2y ago
> they don't seem interested in addressing this (https://code.djangoproject.com/ticket/30874 https://code.djangoproject.com/ticket/30874). Actually it seems they are: https://code.djangoproject.com/ticket/22492#comment:11 https://code.djangoproject.com/ticket/22492#comment:11
DarkNova6 2y ago
It is a huge red flag to me when people recommend Django ORM without admitting the pitfalls of the Active Record pattern. This is a problem which simlpy doesn't need to exist and it's wasting computing and dev resources time and time again.
oliwarner 2y ago
Not really. .prefetch_related (for whole models) and annotate/Subquery (for fields or aggregates) have existed for many years, alongside a pile of aggregate functions which have existed forever and have improved. Whether or not you use the tools given to you is a sign of developer quality and experience. You can easily avoid n+1 99% of the time but you have to appreciate the problem exists. I think the Django project demanding some competence is okay.
- rowanseymour 2y ago
  Not sure you understand my complaint because forcing the user to track down N+1 queries to know where they're missing .prefetch_related is the problem. I don't want to fix something 99% of the time. I want my ORM to enforce correctness 100% of the time.

wruza 2y ago

What would be the best way?

rowanseymour 2y ago
If it were up to me there'd be a way to completely disable it globally for all models. Let me explicitly enable it when I'm just shelling around or checking results in unit tests. It is not a feature for production environments.
- geraneum 2y ago
  Would auto-joins be better in production envs?
- yurishimo 2y ago
  Laravel does this (or at least has a config for it). You can disable lazy loading in non-production environments which will throw fatal errors whenever a model is lazy loaded. The nice part is, in prod, that code _allows_ lazy loading since application stability is more important. Hopefully at that point you have a good performance monitoring tool that will alert you to that problem. Laravel also has hooks to more granularly fire events about lazy loading if you want to roll your own notification solution. So I disagree with your assertion. Lazy loading is ONLY for production. While it is possible that such a feature could potentially bring down your DB server and therefor your app in general, if the feature is turned off locally, then hopefully you’re catching it well ahead of time. And if you’re running small app with no users, meh. For large apps, hopefully your team’s standards are enough to keep the monsters at bay. Lazy loading is a sharp knife that requires skill to use appropriately.
  - rowanseymour 2y ago
    Hmm I would call lazy loading the opposite of a sharp knife that requires skill.. you're not being explicit about database work. You're relying on the framework to bail you out where you've forgotten a fetch. And by default Django will do that silently so missing prefetches easily go undiscovered.
andreareina 2y ago
Static lint wouldn't be a bad way
stavros 2y ago
There's a Django app (that I forget the name of) that disables lazy loading in templates, thus requiring you to load everything explicitly in the view.
- rowanseymour 2y ago
  I think there's a few 3rd party solutions like https://github.com/charettes/django-seal https://github.com/charettes/django-seal but I don't love the idea of using something that I assume is monkey patching Django code.
  - charettes 2y ago
    There is an ongoing effort to include back the equivalent in core [here](https://github.com/django/django/pull/17554 https://github.com/django/django/pull/17554)

LaundroMat 2y ago

Just to say there are libraries to help you find n+1 queries too (when your code is running). I use https://pypi.org/project/django-nplusone/ https://pypi.org/project/django-nplusone/ for instance. Sentry also warns of these by the way.

ruthmarx 2y ago

> they don't seem interested in addressing this (https://code.djangoproject.com/ticket/30874 https://code.djangoproject.com/ticket/30874). Actually it seems they are: https://code.djangoproject.com/ticket/22492#comment:11 https://code.djangoproject.com/ticket/22492#comment:11

DarkNova6 2y ago

It is a huge red flag to me when people recommend Django ORM without admitting the pitfalls of the Active Record pattern. This is a problem which simlpy doesn't need to exist and it's wasting computing and dev resources time and time again.

oliwarner 2y ago

Not really. .prefetch_related (for whole models) and annotate/Subquery (for fields or aggregates) have existed for many years, alongside a pile of aggregate functions which have existed forever and have improved. Whether or not you use the tools given to you is a sign of developer quality and experience. You can easily avoid n+1 99% of the time but you have to appreciate the problem exists. I think the Django project demanding some competence is okay.

rowanseymour 2y ago
Not sure you understand my complaint because forcing the user to track down N+1 queries to know where they're missing .prefetch_related is the problem. I don't want to fix something 99% of the time. I want my ORM to enforce correctness 100% of the time.