3 ms·

We think about this at Hasura deeply! I know it's counter intuitive, but we think of this as an opportunity to solve those problems at the Hasura layer. Shardi

We think about this at Hasura deeply! I know it's counter intuitive, but we think of this as an opportunity to solve those problems at the Hasura layer. Sharding and caching are hard but also not impossible to make declarative if you know enough about how the data is modelled, how data ownership works (authorization) and how the cache is managed (caching hot queries, caching complex queries but infrequent, invalidated on time or data). I'm not trying to say the problem isn't hard, I'm trying to ask what it would take to solve the problem. :) We're also actively working on solving them. Imagine if those challenges got solved for a sufficient number of use cases! Incidentally, databases are getting better at preventing users outgrowing them too while preserving the SQL API as much as possible, or providing a transform. Citus / YugaByte / Planetscale / Cockroach / Spanner to name a few.

sixdimensional 6y ago
I worked with a team on this problem once before. It is something definitely solvable, it does take some engineering time though. Also, that team I worked with missed something pretty important - you'll never solve this problem completely when data becomes too big to dynamically join (or even cache) across heterogeneous systems - but it is something I think you could maybe solve differently using event queues and materialized views for specific complex cases. Have you guys thought about looking at some other tools that might be symbiotic? Hasura's interface would be an incredible companion to something like Dremio (also open source), since it is a fully-SQL engine across a number of heterogeneous systems, with probably the most advanced caching I have ever seen in a system like it (Apache Arrow in-memory and columnar storage). If you can make your query interface adapted to tools like Dremio, you shouldn't have any problem exploiting caching in those kinds of platforms - might be even more efficient than building into Hasura for some use cases. Edit: I think there is a newer type of interface architecture forming here, and I think GraphQL represents one of the pieces of the puzzle, would be really interested to see more detail on Hasura's take on that.

sixdimensional 6y ago

I worked with a team on this problem once before. It is something definitely solvable, it does take some engineering time though. Also, that team I worked with missed something pretty important - you'll never solve this problem completely when data becomes too big to dynamically join (or even cache) across heterogeneous systems - but it is something I think you could maybe solve differently using event queues and materialized views for specific complex cases. Have you guys thought about looking at some other tools that might be symbiotic? Hasura's interface would be an incredible companion to something like Dremio (also open source), since it is a fully-SQL engine across a number of heterogeneous systems, with probably the most advanced caching I have ever seen in a system like it (Apache Arrow in-memory and columnar storage). If you can make your query interface adapted to tools like Dremio, you shouldn't have any problem exploiting caching in those kinds of platforms - might be even more efficient than building into Hasura for some use cases. Edit: I think there is a newer type of interface architecture forming here, and I think GraphQL represents one of the pieces of the puzzle, would be really interested to see more detail on Hasura's take on that.