5 ms·

No More N+1. Period

19 points by evgeniydemin 4y ago

evgeniydemin 4y ago
Why would you spend your time dealing with annoying N+1 problems when you can enable auto-preloading and forget about it for good? Are you surprised it's possible? Check it out!
evntdrvn 4y ago
The problem that I’ve seen a bunch of times with the similar lazy-autoload approach in .NET Entity Framework is that someone accidentally/naively serializes one of the EF objects rather than projecting it to a POCO first, and then Prod blows up when the serializer dutifully follows all the relationships in a complex nested object and ends up pulling a massive amount of data from the DB and then tries to log/send 1GB objects at N requests per second. Oops
lawxls 4y ago
related: auto prefetch fk values in django - https://github.com/tolomea/django-auto-prefetch https://github.com/tolomea/django-auto-prefetch
rizky05 4y ago
[dead]
amelius 4y ago
Glad I immediately saw the gif-memes and could close the article within seconds.
- volkk 4y ago
  sir, you've dropped your top hat
- a4isms 4y ago
  An ad hominem appeal to propriety: "The moment I heard my opponent in the debate speaking in the vulgar dialect, I knew at once I could dismiss their arguments as those made by The Wrong Sort Of People. If they were serious about politics, they would have gone to a proper Public School."
  - amelius 4y ago
    This could be a real issue if my attention was worth nothing and I didn't have thousands of other articles screaming for it.
    - deleted 4y ago
      [deleted]
- kayodelycaon 4y ago
  I'm only seeing one at the top of the article.
- evgeniydemin 4y ago
  I'm sorry; I didn't know memes are considered bad things. I'm experimenting with "facing" pictures and haven't found my style so far. What kind of image would you suggest I have on posts like this?
LesZedCB 4y ago
I think datomic is the only DB I'm aware of that can actually make the promises in the title.
kayodelycaon 4y ago
This looks pretty great and I'd definitely consider adding it to new projects, but it's not a magic bullet. You still need to think about what queries this will trigger and it's a lot less obvious than includes(). If you have a large application with an ORM like Active Record, you'll often end up in situations where queries inside methods trigger the N+1 problem. Sometimes this loading is highly conditional. You end up trading the N+1 problem for a "loading an entire unnecessary table" problem. I suspect this is the reason this feature hasn't been added to Active Record. It would be used a lot when it's only narrowly useful and has worse side effects. (It's also yet one more layer of complexity on top of an already complex ORM.)
recursivedoubts 4y ago
my opinion, worth what you paid for it: you shouldn't worry about n+1 issues until you have a verified perf issue in your app if you have paging and N is relatively small, the additional complexity may not be worth it (although ActiveRecord makes it very easy to tweak locally as needed, which is the right thing) what is important in my opinion is a server side framework is logging of perf related details out of the box: - show how long each request takes and flag slow ones - show what queries occur in each request The more humane formatting of these logs, the better...
infamia 4y ago
Note: If you're using SQLite as your production database, N+1 queries are generally not a problem [1]. This is because queries are just a function call and don't involve any messaging round trip to a server (i.e., they're very low latency since there is no server). This makes most N+1 queries so fast, there isn't a difference worth noticing. The only caveat is if you're pulling large amounts of data (e.g., binary blobs) off the disk or something really big/complicated in your loop; otherwise it typically shouldn't be of any concern. [1] https://sqlite.org/np1queryprob.html https://sqlite.org/np1queryprob.html
evgeniydemin 4y ago
Thank you very much for your comments! I appreciate that! So far, I see that N+1 is a "measurable" problem that doesn't have to be removed with an additional layer of hidden complexity.

19 points by evgeniydemin 4y ago

Why would you spend your time dealing with annoying N+1 problems when you can enable auto-preloading and forget about it for good? Are you surprised it's possible? Check it out!

evntdrvn 4y ago

The problem that I’ve seen a bunch of times with the similar lazy-autoload approach in .NET Entity Framework is that someone accidentally/naively serializes one of the EF objects rather than projecting it to a POCO first, and then Prod blows up when the serializer dutifully follows all the relationships in a complex nested object and ends up pulling a massive amount of data from the DB and then tries to log/send 1GB objects at N requests per second. Oops

lawxls 4y ago

related: auto prefetch fk values in django - https://github.com/tolomea/django-auto-prefetch https://github.com/tolomea/django-auto-prefetch

rizky05 4y ago

[dead]

amelius 4y ago

Glad I immediately saw the gif-memes and could close the article within seconds.

volkk 4y ago
sir, you've dropped your top hat
a4isms 4y ago
An ad hominem appeal to propriety: "The moment I heard my opponent in the debate speaking in the vulgar dialect, I knew at once I could dismiss their arguments as those made by The Wrong Sort Of People. If they were serious about politics, they would have gone to a proper Public School."
- amelius 4y ago
  This could be a real issue if my attention was worth nothing and I didn't have thousands of other articles screaming for it.
  - deleted 4y ago
    [deleted]
kayodelycaon 4y ago
I'm only seeing one at the top of the article.
evgeniydemin 4y ago
I'm sorry; I didn't know memes are considered bad things. I'm experimenting with "facing" pictures and haven't found my style so far. What kind of image would you suggest I have on posts like this?

LesZedCB 4y ago

I think datomic is the only DB I'm aware of that can actually make the promises in the title.

kayodelycaon 4y ago

This looks pretty great and I'd definitely consider adding it to new projects, but it's not a magic bullet. You still need to think about what queries this will trigger and it's a lot less obvious than includes(). If you have a large application with an ORM like Active Record, you'll often end up in situations where queries inside methods trigger the N+1 problem. Sometimes this loading is highly conditional. You end up trading the N+1 problem for a "loading an entire unnecessary table" problem. I suspect this is the reason this feature hasn't been added to Active Record. It would be used a lot when it's only narrowly useful and has worse side effects. (It's also yet one more layer of complexity on top of an already complex ORM.)

recursivedoubts 4y ago

my opinion, worth what you paid for it: you shouldn't worry about n+1 issues until you have a verified perf issue in your app if you have paging and N is relatively small, the additional complexity may not be worth it (although ActiveRecord makes it very easy to tweak locally as needed, which is the right thing) what is important in my opinion is a server side framework is logging of perf related details out of the box: - show how long each request takes and flag slow ones - show what queries occur in each request The more humane formatting of these logs, the better...

infamia 4y ago

Note: If you're using SQLite as your production database, N+1 queries are generally not a problem [1]. This is because queries are just a function call and don't involve any messaging round trip to a server (i.e., they're very low latency since there is no server). This makes most N+1 queries so fast, there isn't a difference worth noticing. The only caveat is if you're pulling large amounts of data (e.g., binary blobs) off the disk or something really big/complicated in your loop; otherwise it typically shouldn't be of any concern. [1] https://sqlite.org/np1queryprob.html https://sqlite.org/np1queryprob.html

evgeniydemin 4y ago

Thank you very much for your comments! I appreciate that! So far, I see that N+1 is a "measurable" problem that doesn't have to be removed with an additional layer of hidden complexity.