3 ms·

One thing that is bugging me is that, while the article mentions DataLoader, it kind of just breezes over it without explaining the reasons why DataLoader isn't

One thing that is bugging me is that, while the article mentions DataLoader, it kind of just breezes over it without explaining the reasons why DataLoader isn't a good long-term solution. > Besides the popular Dataloader approach, another very common way of tackling this problem is by taking a GraphQL query, and coming up with the exact SQL needed to resolve it But why? Why not just use DataLoader? Is there something intrinsically wrong with it that makes it a bad solution? I want to know! My understanding was the DataLoader was extremely performant and worked for 90% of all situations where a join is needed to get values for a GraphQL query. > Data fetching remains something to improve Certainly this might be the case, but the article didn't address the most popular form of data fetching and explain why a graphql-to-sql compiler would be a better solution.

chrisweekly 6y ago
Great question! I want to know the answer too (hence the comment not just the +1).
kevsim 6y ago
We’ve been using data loader a lot and it’s working ok. We were very careful to dump all sql queries our app was making per request to ensure we were not making more queries than necessary. One thing I will say as we do make a lot of “IN” queries as a result. You fetch a bunch of rows with a data loader and then you want to fetch a bunch of rows from some other table based on those first results so you end up with an IN query where you might have had crafted a JOIN if you were writing it by hand.
- xyzzy_plugh 6y ago
  But does the query plan perform a join? The text of your query matters less than the plan it generates.
  - kevsim 6y ago
    No, it can't because it's actually making two separate queries. For example, let's say we're fetching classified ads with the users that posted them. First your GraphQL resolvers would fetch the ads (and you won't know that the users are being asked for). Then for each ad, you'd tell your data loader to fetch the user associated with that add. That will ultimately result in a single SQL query with an IN like `SELECT * FROM users WHERE id IN (?)` where you'd pass in all the user IDs you gathered from the results of the ads query. However, another commenter pointed out above that some implementations allow peeking into the GraphQL query. That way if you know you're going to need the user anyway, you can already perform the join proactively when you fetch the ads. In a few cases we've done this because we always want the joined results so we just do the join regardless of the query we get.
    - xyzzy_plugh 6y ago
      Erm, are you telling me there is no way to pipeline the ads query to be inline within the IN clause? Because that'd generate a join, generally. I still don't understand why this abstraction is useful, if not harmful.
      - kevsim 6y ago
        I think there is a way to do it, that was the peeking I referred to. But it's not how it works out of the box. I think the abstraction is somewhat useful if your GraphQL queries vary tremendously (one device asks for the user along with the ads, another device doesn't). Expand this out to a bunch of different data types and you'd find yourself writing a lot of variations of the queries. GraphQL with Data loader allows you to write these things exactly once albeit with potentially suboptimal queries. All of that being said, we're finding that our queries vary very little and are somewhat questioning the value of data loaders and of GraphQL altogether in our case.
- snuxoll 6y ago
  > One thing I will say as we do make a lot of “IN” queries as a result. You fetch a bunch of rows with a data loader and then you want to fetch a bunch of rows from some other table based on those first results so you end up with an IN query where you might have had crafted a JOIN if you were writing it by hand. I don't know if it's common across other GraphQL implementations, but the graphql-java library lets you peek [0] into the selections so you can opt to limit data you select, execute a join instead of using an IN query, etc. I've used this with good results when making a prototype graphql interface to our billing software, anything that had a 1:1 relationship with a parent record would get JOINed by the resolver, while 1:N or N:M relationships would execute separate queries. [0]:https://www.graphql-java.com/documentation/v14/fieldselection/ https://www.graphql-java.com/documentation/v14/fieldselectio...
andrewingram 6y ago
I think you might’ve misinterpreted the article. Whilst it does gloss over the Dataloader approach, it doesn’t dismiss it. The conclusion ends up essentially advocating for it. The Dataloader approach gives you a potentially (arguably likely) suboptimal baseline performance for simple projects, or ones where tight coupling with the database is acceptable. But it’s one that tends to scale better and more predictably with complexity (both codebase size and architecture). A hybrid model where you use Dataloader but include some database query optimisations driven by your selection sets is likely a winning approach, but will probably need to be fairly bespoke to your architecture.
bernawil 6y ago
it kind of just breezes over it without explaining the reasons why DataLoader isn't a good long-term solution nit: I don't think the author argues that dataloader isn't a good long term solution. If I had to guess, because it's something that you just turn on in your project and it just works -or doesn't-. I've used it in a project together with graphql-sequelize, which gives you "automatic" resolvers for your queries. Turn dataloader on, perform a query that uses any of those automatic resolvers and all underlying SQL queries to the same table are magically merged together. Now, replace those automatic resolvers with custom ones and you lose this magic. Now you have to write your own ORM query that groups anything that would end up being multiple queries into one, basically the case the author is pointing out.

chrisweekly 6y ago

Great question! I want to know the answer too (hence the comment not just the +1).

kevsim 6y ago

We’ve been using data loader a lot and it’s working ok. We were very careful to dump all sql queries our app was making per request to ensure we were not making more queries than necessary. One thing I will say as we do make a lot of “IN” queries as a result. You fetch a bunch of rows with a data loader and then you want to fetch a bunch of rows from some other table based on those first results so you end up with an IN query where you might have had crafted a JOIN if you were writing it by hand.

xyzzy_plugh 6y ago
But does the query plan perform a join? The text of your query matters less than the plan it generates.
- kevsim 6y ago
  No, it can't because it's actually making two separate queries. For example, let's say we're fetching classified ads with the users that posted them. First your GraphQL resolvers would fetch the ads (and you won't know that the users are being asked for). Then for each ad, you'd tell your data loader to fetch the user associated with that add. That will ultimately result in a single SQL query with an IN like `SELECT * FROM users WHERE id IN (?)` where you'd pass in all the user IDs you gathered from the results of the ads query. However, another commenter pointed out above that some implementations allow peeking into the GraphQL query. That way if you know you're going to need the user anyway, you can already perform the join proactively when you fetch the ads. In a few cases we've done this because we always want the joined results so we just do the join regardless of the query we get.
  - xyzzy_plugh 6y ago
    Erm, are you telling me there is no way to pipeline the ads query to be inline within the IN clause? Because that'd generate a join, generally. I still don't understand why this abstraction is useful, if not harmful.
    - kevsim 6y ago
      I think there is a way to do it, that was the peeking I referred to. But it's not how it works out of the box. I think the abstraction is somewhat useful if your GraphQL queries vary tremendously (one device asks for the user along with the ads, another device doesn't). Expand this out to a bunch of different data types and you'd find yourself writing a lot of variations of the queries. GraphQL with Data loader allows you to write these things exactly once albeit with potentially suboptimal queries. All of that being said, we're finding that our queries vary very little and are somewhat questioning the value of data loaders and of GraphQL altogether in our case.
snuxoll 6y ago
> One thing I will say as we do make a lot of “IN” queries as a result. You fetch a bunch of rows with a data loader and then you want to fetch a bunch of rows from some other table based on those first results so you end up with an IN query where you might have had crafted a JOIN if you were writing it by hand. I don't know if it's common across other GraphQL implementations, but the graphql-java library lets you peek [0] into the selections so you can opt to limit data you select, execute a join instead of using an IN query, etc. I've used this with good results when making a prototype graphql interface to our billing software, anything that had a 1:1 relationship with a parent record would get JOINed by the resolver, while 1:N or N:M relationships would execute separate queries. [0]:https://www.graphql-java.com/documentation/v14/fieldselection/ https://www.graphql-java.com/documentation/v14/fieldselectio...

andrewingram 6y ago

I think you might’ve misinterpreted the article. Whilst it does gloss over the Dataloader approach, it doesn’t dismiss it. The conclusion ends up essentially advocating for it. The Dataloader approach gives you a potentially (arguably likely) suboptimal baseline performance for simple projects, or ones where tight coupling with the database is acceptable. But it’s one that tends to scale better and more predictably with complexity (both codebase size and architecture). A hybrid model where you use Dataloader but include some database query optimisations driven by your selection sets is likely a winning approach, but will probably need to be fairly bespoke to your architecture.

bernawil 6y ago

it kind of just breezes over it without explaining the reasons why DataLoader isn't a good long-term solution nit: I don't think the author argues that dataloader isn't a good long term solution. If I had to guess, because it's something that you just turn on in your project and it just works -or doesn't-. I've used it in a project together with graphql-sequelize, which gives you "automatic" resolvers for your queries. Turn dataloader on, perform a query that uses any of those automatic resolvers and all underlying SQL queries to the same table are magically merged together. Now, replace those automatic resolvers with custom ones and you lose this magic. Now you have to write your own ORM query that groups anything that would end up being multiple queries into one, basically the case the author is pointing out.