6 ms·
Finding and optimizing N+1 queries on a relational database
- echelon 2y agoOne solution is to not use an ORM. These problems are framework problems. I find the cognitive overhead of worrying whether a framework might do the wrong thing or diagnosing problems after they arise often override any cost savings frameworks provide in the first place.
- 9dev 2y agoI totally share your frustrations over ORMs and the issues they cause, but I seriously doubt your cost calculations. The sheer number of working CRUD applications that don’t rely on fancy queries is mind-boggling, and most developers engaging in high-level system design seem mostly oblivious to this. The ORM may not make for pretty or efficient queries, but it enables armies of mediocre programmers to crunch out generic line of business applications very fast. The performance of these apps may not be what it could be, but while we are still busy discussing the finer details of query planner optimisation, folks are making a fortune with N+1 queries in production that virtually nobody cares about.
- ta2234234242 2y ago> but it enables armies of mediocre programmers to crunch out generic line of business applications very fast. Or good programmers that see writing raw SQL as a premature optimization. Someone on here once said that often companies end up reinventing their own ORM once the number of queries they have to maintain get up into the thousands. But they've usually made a really shitty ORM by that point in time.
- Daishiman 2y agoNot using an ORM is a solution in the same way that the solution to changing a car's oil is to not own a car. I've spent a good 20 years building web apps. The speed and productivity of having an ORM like Django's a well-integrated into business logic is unparalleled with the possible exception of Rails and Laravel, maybe. Systems that have 50+ tables become _trivial_ to maintain by one person. You can refactor an underlying database table without having to rewrite queries. There are entire classes of refactoring that go away with the ORM. Serving database objects with Django Rest Framework or building up CRUD interfaces with the Django admin would simply be impossible otherwise. Getting the hang of the ORM took me all of 6 months to build up to the nastiest of edge cases, which are functionally as difficult to debug as any complex SQL query with window functions or esoteric functions. The discourse is irrationally framed towards the first months of learning, for a technology that I've been using for 14 years with _no_ sign of finding a replacement.
- henryfjordan 2y agoBut now I need a developer with >6months of experience in that particular ORM to maintain my app, instead of just someone who knows SQL. Nothing is free.
- Daishiman 2y agoYou would have needed someone to get up to speed in all the things you had to reinvent that frameworks already did for you, except that frameworks do it better due to the thousands of eyeballs that have looked at the same problems over and over again.
- deleted 2y ago[deleted]
- ljm 2y agoYou’ll get N+1 or quadratic queries whether or not you use an ORM. At the end of the day you are allocating time on the database and it doesn’t care about the way you build the query. Runtime compute tends to be a lot cheaper than DB compute too - might as well run those cycles before the DB does.
- henryfjordan 2y agoIf you actually have to reckon with the SQL queries in your app, N+1 becomes less likely. Consider `rows.map { row -> repository.fetchRelatedRow(row.id) }`. That's much more obviously going to cause issues than `rows.map { row.relatedRow }`. You don't need to cross-reference the model code for row to see if relatedRow is eager loaded or not, the answer is right there for you.
- rand_r 2y agoIt is true that DB compute is relatively expensive, but doing a join on the DB itself is obscenely more efficient vs doing it manually in a nested loop on an app server. Also, the DB is generally a lot smarter than doing quadratic operations. The usual worst case is a Merge Join which involves sorting both tables first, to do an efficient join in nlogn time. It’s just not feasible to compete with the DB for efficiency without reimplementing a DB, which is pretty yikes. See https://www.cybertec-postgresql.com/en/join-strategies-and-performance-in-postgresql/ https://www.cybertec-postgresql.com/en/join-strategies-and-p...
- switchbak 2y agoI don't get that. If you're not using an ORM, you're simply looking at writing the queries to satisfy your requirements - nothing is guiding you towards an N+1 path. There's nothing inherent about how we access data in a relational world that means we have to adopt such an anti-pattern. It does mean that you have to think ahead a little bit though.
- henning 2y agoDatabase queries are likely to be the most constrained resource in a typical enterprise application. They are essential but also extremely slow. Because of this, that limited resource needs to be carefully managed and it needs to be straightforward to reason about the I/O behavior of your application. Attempting to hide them away and obscure what is actually happening in your application is extremely counter-productive.
- cool_dude85 2y agoNot knowing anything about web development or Django or whatever, why isn't this just in a real database? Where you can do a join instead of 2000 individual queries?
- senko 2y agoYou can do joins in Django just fine, and you can even NOT write them and have Django automagically do the correct thing if you know how to use it.
- ljm 2y agoThe ORMs provided by full stack frameworks like Rails, Django and DotNet are basically engines in and of themselves. They’re doing a lot of work at runtime to reduce pressure on the DB and ensure connections are returned to the pool as soon as possible. They’re not ‘automagic’; you can reason with them quite easily.
- cool_dude85 2y agoThen what's up with the pattern they show of querying every board and then iterating through to query again for every report associated with the board?
- the_gorilla 2y agoI was expecting them to say after introducing this, that they started with an intentionally bad example so they could improve on it. But instead they went on to talk about a 2.5x speedup on something that would already be 100x faster in any other database. I don't get why they're doing this at all.
- henryfjordan 2y agoObject Relational Mappers like the one Django use are a whole topic of debate unto themselves. They strive to map SQL concepts into a more code-friendly domain. Rows are represented as objects, references become attributes of those objects which can be eager-loaded by a join when you fetch the parent object or lazy-loaded with their own query when they are actually accessed. The N+1 problem comes from this lazy-loading scenario. This programming model abstracts away the SQL which can be really cool and easy until you try to do something off the beaten path. Junior devs love it for the ease, and diving into the internals once things go wrong turns them into Senior devs!
- senko 2y agoHeh, wrote something similar (django-queryinspect) a while ago, then abandoned it since iirc django debug toolbar has it builtin, and I moved to some other stuff.. maybe I should dust it off. N+1 is fairly well understood and documented gotcha, and Django has excellent tools that make it easy to avoid it (as the article mentions). To use Django ORM effectively in nontrivial apps, you do need to understand how relational databases work, and it's helpful to understand SQL. But it does make me a whole lot more productive (and happy!) than writing SQL by hand.
- IceHegel 2y agoThis is the best write-up of tools to use to avoid shooting yourself in the foot with Django N+1s that I have seen. However, having used Django professionally for a while, I can say that it is really starting to show its age even if you deal with the N+1 problems. You still have to deal with: 1. Blocking database calls that cannot easily be used in an asyncio context 2. No types 3. 50% of package code being written without any regard for performance If you're starting a project today and considering Django, I highly recommend you go with Nestjs and Prisma instead.
- dangsux 2y ago[dead]
- ggregoire 2y ago> If you're starting a project today and considering Django, I highly recommend you go with Nestjs and Prisma instead. I recommend you to take a look at PostgREST. I replaced all our Django, Flask, Falcon, FastAPI and custom Python HTTP APIs with it. It's phenomenal. It's faster than any of those solutions and adding an endpoint with roles & permissions is just creating a view and executing a bunch of GRANT/CREATE POLICY (takes about 2 min of work). I'm genuinely surprised this didn't replace CRUD APIs in most Postgres shops.
- megaman821 2y agoNestjs has no real ecosystem compared to Django, Rails, Laravel, ASP.NET or Spring. Prisma has some nice typing but it is not even half as capable as the Django ORM.
- sgarland 2y agoPretty rich using Prisma as a counter-example to non-performance, considering it launched without doing any JOINs – just pulled every tuple back into the engine, then filtered. Also, Django, being Python, supports type hints just fine.
- Sytten 2y agoWhat is this argument? It's not because something was bad 7 years ago that it is bad now, tech evolves.
- werdnapk 2y agoIn Rails you can make this 3 queries. One to retrieve the board, one to get all the report IDs associated with the board and a third to get all the reports with the associated report IDs. An example in Rails would look like: board = Board.includes(:reports).find(board_id)
- colinbartlett 2y agoI install the Bullet gem in every Rails project because it not only detects these N+1 queries but tells you how to fix them, helping to ensure they never even make it to production. https://github.com/flyerhzm/bullet https://github.com/flyerhzm/bullet
- pjm331 2y agoAs of rails 6.1 you can use active records built in strict_loading to solve this problem as well
- boxed 2y agoIn iommi we built something imo much smarter: https://docs.iommi.rocks/en/latest/dev_tools.html#sql-trace https://docs.iommi.rocks/en/latest/dev_tools.html#sql-trace The idea is that: 1. We store all the stack traces and SQL calls 2. At the end of the request we group them to find N+1 3. If there are any N+1 (given some threshold), we print a big warning to stdout. Full traceback for one of the SQL calls and a few examples of SQL. This is imo MUCH better. It means the console is silent when there is no issue. It also means the console is loud when there IS an issue. And the output you get is very helpful. This approach is better than the one in the article, and superior to Django Debug Toolbar where you have to actively go look for problems instead of the machine telling you.
- KronisLV 2y ago> Django Querycount This seems pretty much like a must have when you think about it: so where’s the equivalents for .NET, Java, Node, Go, Rails and others? I can’t believe that every web framework that has an ORM or just a connection to the database doesn’t provide summaries like this. At best, you can look into some tracing options that might reveal where the the execution time went. I do hope that there are options out there (or might have to try and fail writing something myself) because I can immediately think of a few legacy projects where N+1 is prevalent, maybe even in the new code, depending on who wrote it; some people genuinely believe that using the service pattern and a lot of DB calls for a single API request is okay.
- danpalmer 2y agoEvery web framework I've used has had an equivalent, they're widely available. Specifically I've seen this same pattern for .NET MVC and for Rails. I've never looked for one of these and not found it.
- KronisLV 2y agoThat’s odd because even in .NET the common suggestion I’ve heard is to just crank the logging level and try to extract data from that, e.g. https://learn.microsoft.com/en-us/ef/core/logging-events-diagnostics/ https://learn.microsoft.com/en-us/ef/core/logging-events-dia... Or the equivalent in other stacks as well, for example with Hibernate or myBatis logging in Java, almost never a pretty summary table of how many DB actions of what kind were called per API request. I might need to dig around more!
- nxicvyvy 2y agoJust orm things... https://wikipedia.org/wiki/Object%E2%80%93relational_impedance_mismatch https://wikipedia.org/wiki/Object%E2%80%93relational_impedan...
- mattbillenstein 2y agoYou either spend your time learning SQL or debugging an ORM - I prefer the former...
- ta2234234242 2y agoEver since the paper on the Vietnam of Computer Science came out, I basically grew to wonder why SQL was the only language. Good? debatable. Composable? no. spreadsheet friendly, but not json friendly. There are what, 1000 programming languages, but only one SQL? Why?
- jiggawatts 2y agoThere are other query languages out there, many of which fix the glaring issues with SQL. For example, Microsoft's Kusto Query Language (KQL) has the "FROM" clause at the beginning to enable tab-complete, you can chain multiple "WHERE" clauses without needing to use "AND", etc...
- randomdata 2y ago> I basically grew to wonder why SQL was the only language You were well ahead of the pack simply by realizing that other options are possible. It is only within the last year or two that the HN crowd has finally come to accept that SQL and relational calculus are not the same thing. If you want a good laugh, check out the older posts on HN where the comments are adamant that there cannot be anything other than SQL for relational querying because it is some kind of fundamental mathematical truth. > There are what, 1000 programming languages, but only one SQL? Why? There are the family of languages that get lumped together under the ORM umbrella. They are quite popular – maybe even more popular than SQL. There are plenty of other languages that are less popular, but they are often impractical to use in any real-world scenario which maintains their unpopular status. > but not json friendly. Not standard SQL, but if we're counting extensions duality views are JSON friendly. The Oracle tax may be a tough pill to swallow, though.
- anon28428s938a 2y agoI see the same thing happening in the hardware language space. There exists a large amount of gatekeeping by people who believe that Verilog/VHDL can ever only be the one true language. So research into alternative languages die. And interfaces are obfuscated behind the language.
- stephen 2y agoWe built Joist, an ActiveRecord-style ORM in TypeScript, on top of Facebook's dataloader library, and haven't had a single N+1 in ~4 years: https://joist-orm.io/docs/goals/avoiding-n-plus-1s https://joist-orm.io/docs/goals/avoiding-n-plus-1s Any "query in a loop", whether a manually-written `for` loop, or an "emergent for loop" from 100 `after_update` lifecycle hooks firing at once, are always auto-batched, every time. Joist is also likely why we actually like GraphQL, vs. it generally having a reputation for being a pita to implement (N+1s everywhere w/o tedious, manual batching).