3 ms·
Calcite is actually pretty damn fast, the overhead is surprisingly minimal. I am using it as the backbone for my hobby project that auto-generated federated Gr
by gavinray 5y ago
Calcite is actually pretty damn fast, the overhead is surprisingly minimal.
I am using it as the backbone for my hobby project that auto-generated federated GraphQL API's:
https://github.com/GavinRay97/GraphQLCalcite https://github.com/GavinRay97/GraphQLCalcite
The experience has been incredibly positive and the community has been incredibly helpful & supportive. It's one of the coolest technical projects I've ever seen and has sparked my interest in query engines and relational databases.
I posted some JMH benchmarks of an app that parses a GraphQL query, converts it to Calcite relational expressions, and then executes it against an in-memory DB and it ran on the orders of milliseconds:
https://lists.apache.org/thread/hofjx628864t0kt4kk8vo4tjfrxbqf7o https://lists.apache.org/thread/hofjx628864t0kt4kk8vo4tjfrxb...
Something very similar to Calcite but much lesser-known is the "Teiid" project:
https://github.com/teiid/teiid https://github.com/teiid/teiid
Highly recommend checking the code out. It's got a brilliant query optimizer/planner tailored to cross-datasource queries, a cache system, and a translator architecture that can convert a generic SQL dialect to dozens of SQL/NoSQL flavors.
Also integrates other data sources like REST API's, S3, flat files etc as queryable data sources.
- michaelmior 5y agoI was moreso referring to the pieces of queries that Calcite may have to execute itself, often because they are not supported by the underlying data source. That piece of Calcite hasn't been very highly optimized. There's definitely ongoing work in that area though. For example, I have a former student who's been working on an adapter to Apache Arrow. Beyond having a fast execution engine, this should also eventually enable Calcite to work pretty seamlessly with other systems using Arrow without serialization overhead.
- gavinray 5y agoIs this student in contact with Jacques Nadeau? He's one of the only people I know that's a committer/PMC of both Arrow and Calcite, super nice and brilliant guy. Maybe he'd have good feedback or ideas around this? https://twitter.com/intjesus https://twitter.com/intjesus
- michaelmior 5y agoIndirectly. I know Jacques. I'm also on the Calcite PMC :) Thanks for the pointer though!
- gavinray 5y agoIf you've responded to any of my emails, no thank YOU!