6 ms·

hand rolling a custom query engine - the exact opposite of what every business wanted when the engineers sold it graphql

obi1kenobi 4y ago
Why hand-roll one when you can use one that's already available and thoroughly tested :) https://github.com/obi1kenobi/trustfall https://github.com/obi1kenobi/trustfall (Hi Dustin!)
- bfz 4y ago
  I've yet to encounter a GraphQL off-the-shelf server (from Python and JS spaces) where hitting a slow query didn't immediately turn into half a day's work The whole concept is what happens when you let a smart person work on a small problem for far, far too long
  - obi1kenobi 4y ago
    I'd recommend checking out the project link in the comment to which you replied. It is designed _specifically_ to avoid the problem you mention: instead of a fully materialized, fully-nested result, it returns flattened row-oriented results (like a SQL database). This allows for lazy evaluation i.e. rows are produced only as they are consumed. So if you accidentally write a query that would produce a billion rows but only load 20, the execution of the query only happens for 20 rows + any batching or prefetch optimizations in the adapter used to bind the dataset to the query engine.
  - PaulHoule 4y ago
    It is a fundamental problem of a "graph". (1) There are usually some nodes of very high degree and traversing those nodes will explode your query, (2) if you are following N links and the average degree is d, you are going to come across dᴺ nodes and that is a lot of nodes as N gets big! Tim-Berners Lee told me that if you can't send the whole graph you should send a subset of the graph that contains the most important facts. It's a right answer but also a frustrating one to a programmer who sees correct implementation of algorithms to mean that you get the ticket done and they don't come at you with a ticket about it again. That is, that query I'm writing is part of an algorithm that depends on getting a certain answer and getting an uncertain answer for one query is like some spoiled milk that ruins the whole batch.
    - bfz 4y ago
      > It is a fundamental problem of a "graph". So why are we using it for so many naturally non-graph problems? 90%+ of developers' exposure to graphs is through tightly abstract interfaces, I could name maybe 3 graph-related algorithms off the top of my head, but could implement none of them without reading. We could represent the text of this comment in a graph using one node for each unique character, but the result would be stupid, the operations would be slow, the representation needlessly complex, and implementations guaranteeably hard to work with > Tim-Berners Lee told me that if you can't send the whole graph you should send a subset of the graph that contains the most important facts. Indeed, I also caught the ReST buzz around the 2000-2003 timeframe, and turns out 20 years later nobody does that either, because in its purest form it's a pain in the ass for comparable reasons to the topic at hand
      - PaulHoule 4y ago
        It's funny to see a blog post on HN almost every day where somebody rediscovers the power of columnar query answering engines which are almost the opposite of graph databases. I've lost count of how many columnar SQL databases have been donated to the apache project and there are so many systems like Actian and Alteryx where data analysts hook together relational operators with boxes and lines. I had a prototype of a stream processing engine that passed RDF graphs along the lines between the boxes that enable an "object-relational" model, you could eliminate the need for hard-to-maintain joins but I found that firms that had bought multiple columnar processing database companies believed in performance at all cost and couldn't care less for any system that couldn't be implemented with SIMD instructions.
        eurasiantiger 4y ago
        How are they opposite? There are plenty of graph databases out there using columnar storage, even ones directly compatible with GraphQL Federation. Best of both worlds, so to speak.
      - eurasiantiger 4y ago
        Because graphs are a good abstraction for relations and with the right tech choices, are much more manageable and malleable than traditional relational databases.

obi1kenobi 4y ago

Why hand-roll one when you can use one that's already available and thoroughly tested :) https://github.com/obi1kenobi/trustfall https://github.com/obi1kenobi/trustfall (Hi Dustin!)

bfz 4y ago
I've yet to encounter a GraphQL off-the-shelf server (from Python and JS spaces) where hitting a slow query didn't immediately turn into half a day's work The whole concept is what happens when you let a smart person work on a small problem for far, far too long
- obi1kenobi 4y ago
  I'd recommend checking out the project link in the comment to which you replied. It is designed _specifically_ to avoid the problem you mention: instead of a fully materialized, fully-nested result, it returns flattened row-oriented results (like a SQL database). This allows for lazy evaluation i.e. rows are produced only as they are consumed. So if you accidentally write a query that would produce a billion rows but only load 20, the execution of the query only happens for 20 rows + any batching or prefetch optimizations in the adapter used to bind the dataset to the query engine.
- PaulHoule 4y ago
  It is a fundamental problem of a "graph". (1) There are usually some nodes of very high degree and traversing those nodes will explode your query, (2) if you are following N links and the average degree is d, you are going to come across dᴺ nodes and that is a lot of nodes as N gets big! Tim-Berners Lee told me that if you can't send the whole graph you should send a subset of the graph that contains the most important facts. It's a right answer but also a frustrating one to a programmer who sees correct implementation of algorithms to mean that you get the ticket done and they don't come at you with a ticket about it again. That is, that query I'm writing is part of an algorithm that depends on getting a certain answer and getting an uncertain answer for one query is like some spoiled milk that ruins the whole batch.
  - bfz 4y ago
    > It is a fundamental problem of a "graph". So why are we using it for so many naturally non-graph problems? 90%+ of developers' exposure to graphs is through tightly abstract interfaces, I could name maybe 3 graph-related algorithms off the top of my head, but could implement none of them without reading. We could represent the text of this comment in a graph using one node for each unique character, but the result would be stupid, the operations would be slow, the representation needlessly complex, and implementations guaranteeably hard to work with > Tim-Berners Lee told me that if you can't send the whole graph you should send a subset of the graph that contains the most important facts. Indeed, I also caught the ReST buzz around the 2000-2003 timeframe, and turns out 20 years later nobody does that either, because in its purest form it's a pain in the ass for comparable reasons to the topic at hand
    - PaulHoule 4y ago
      It's funny to see a blog post on HN almost every day where somebody rediscovers the power of columnar query answering engines which are almost the opposite of graph databases. I've lost count of how many columnar SQL databases have been donated to the apache project and there are so many systems like Actian and Alteryx where data analysts hook together relational operators with boxes and lines. I had a prototype of a stream processing engine that passed RDF graphs along the lines between the boxes that enable an "object-relational" model, you could eliminate the need for hard-to-maintain joins but I found that firms that had bought multiple columnar processing database companies believed in performance at all cost and couldn't care less for any system that couldn't be implemented with SIMD instructions.
      - eurasiantiger 4y ago
        How are they opposite? There are plenty of graph databases out there using columnar storage, even ones directly compatible with GraphQL Federation. Best of both worlds, so to speak.
    - eurasiantiger 4y ago
      Because graphs are a good abstraction for relations and with the right tech choices, are much more manageable and malleable than traditional relational databases.