6 ms·

For anyone interested, Apache Calcite[0] is an open source data management framework which seems to do many of the same things that Hydra claims to do, but taki

For anyone interested, Apache Calcite[0] is an open source data management framework which seems to do many of the same things that Hydra claims to do, but taking a different approach. Operating as a Java library, Calcite contains "adapters" to many different data sources from existing JDBC connectors to Elasticsearch to Cassandra. All of these different data sources can be joined together as desired. Calcite also has it's own optimizer which is able to push down relevant parts of the query to the different data sources. However, you get full SQL on data sources which don't support it, with Calcite executing the remaining bits itself. Generally all that is required to connect to multiple data sources from CSV to Elasticsearch is just writing a JSON configuration file. Then can get SQL access via JDBC with the able to join all those sources together. Unfortunately, I would not be too surprised if the query execution Calcite was found to be less performance-optimized than Hydra. There is ongoing work for improvement there. That said, there are users of Calcite at Google, Uber, Spotify, and others who have made great use of various parts of the framework. [0] https://calcite.apache.org/ https://calcite.apache.org/

imachine1980_ 5y ago
What's the difference whit regular ORM?(genuine ask)
- michaelmior 5y ago
  ORMs map relational databases to objects. Calcite does not do that. Calcite takes different data sources (potentially with different data models) and presents them all in the relational model. You interact with your data entirely in SQL while ORMs typically have some DSL which uses whatever object model the ORM defines. ORMs are also typically designed to connect to a single data source at a time. I'm personally not familiar with any ORMs that allow combining data from multiple sources within the same application. The primary advantage of Calcite for connecting multiple data sources are 1) easily joining data sources that use completely different APIs (assuming there is a Calcite adapter available) and 2) supporting more complex queries than the original data source supports without having to write code (other than SQL) to do the processing.
  - gavinray 5y ago
    Don't forget having two industrial-grade academic query planners/optimizer implementations that took collective decades of engineering effort! And the project being founded by someone who has written multiple RDBMS. Calcite is a wonder.
wuputah 5y ago
Calcite is definitely some cool tech. I can see why it would be attractive for bigger teams, but it seems like a big lift for smaller teams. Our goal is to make it easy for devs already familiar with Postgres to be able to use add databases without learning new tools or adding software... besides adding Hydra, of course!
- michaelmior 5y ago
  That's a fair point. Although if you're already using a JVM language, it's incredibly easy to integrate. Just another JDBC data source with some JARs to add to your classpath :)
sdesol 5y ago
I've never heard of Calcite before and dug a bit deeper into this project and found it to be quite active. Only taking into consideration changes to Java files, this is the repo activity for the last 12 months: month | authors | commits | files | churn ---------+---------+---------+-------+------- 2022-02 | 21 | 62 | 84 | 4387 2022-01 | 26 | 109 | 243 | 32429 2021-12 | 33 | 99 | 198 | 10461 2021-11 | 19 | 49 | 77 | 6960 2021-10 | 26 | 64 | 371 | 13626 2021-09 | 18 | 41 | 68 | 2258 2021-08 | 11 | 17 | 25 | 1924 2021-07 | 17 | 30 | 51 | 2704 2021-06 | 14 | 31 | 28 | 1708 2021-05 | 9 | 17 | 35 | 1606 2021-04 | 11 | 46 | 99 | 4224 2021-03 | 16 | 36 | 143 | 8471
- riverdroid 5y ago
  What do you use to produce this analysis?
  - sdesol 5y ago
    My product (https://gitsense.com https://gitsense.com) moves most of Git's history into a Postgres database and from there, you can execute the following SQL statement: select commit_ym AS month, count(distinct(author_email)) as authors, count(distinct(commit_id)) as commits, count(distinct(path_id)) as files, sum(total) as churn from z1_commits_422 as commits, z1_changes_422 as changes, z1_code_churn_422 as churn where commits.id=changes.commit_id and changes.code_churn_id=churn.id and lang='java' group by commit_ym order by commit_ym desc limit 12 By having most of Git's history in SQL, I can slice, dice and cross-reference code history, which is how my product works.
- michaelmior 5y ago
  The community is also great. Honestly, the main thing preventing things from moving faster is lack of time to review PRs. There's a lot of interesting things in the pipe!
gavinray 5y ago
Calcite is actually pretty damn fast, the overhead is surprisingly minimal. I am using it as the backbone for my hobby project that auto-generated federated GraphQL API's: https://github.com/GavinRay97/GraphQLCalcite https://github.com/GavinRay97/GraphQLCalcite The experience has been incredibly positive and the community has been incredibly helpful & supportive. It's one of the coolest technical projects I've ever seen and has sparked my interest in query engines and relational databases. I posted some JMH benchmarks of an app that parses a GraphQL query, converts it to Calcite relational expressions, and then executes it against an in-memory DB and it ran on the orders of milliseconds: https://lists.apache.org/thread/hofjx628864t0kt4kk8vo4tjfrxbqf7o https://lists.apache.org/thread/hofjx628864t0kt4kk8vo4tjfrxb... Something very similar to Calcite but much lesser-known is the "Teiid" project: https://github.com/teiid/teiid https://github.com/teiid/teiid Highly recommend checking the code out. It's got a brilliant query optimizer/planner tailored to cross-datasource queries, a cache system, and a translator architecture that can convert a generic SQL dialect to dozens of SQL/NoSQL flavors. Also integrates other data sources like REST API's, S3, flat files etc as queryable data sources.
- michaelmior 5y ago
  I was moreso referring to the pieces of queries that Calcite may have to execute itself, often because they are not supported by the underlying data source. That piece of Calcite hasn't been very highly optimized. There's definitely ongoing work in that area though. For example, I have a former student who's been working on an adapter to Apache Arrow. Beyond having a fast execution engine, this should also eventually enable Calcite to work pretty seamlessly with other systems using Arrow without serialization overhead.
  - gavinray 5y ago
    Is this student in contact with Jacques Nadeau? He's one of the only people I know that's a committer/PMC of both Arrow and Calcite, super nice and brilliant guy. Maybe he'd have good feedback or ideas around this? https://twitter.com/intjesus https://twitter.com/intjesus
    - 5y ago

imachine1980_ 5y ago

What's the difference whit regular ORM?(genuine ask)

michaelmior 5y ago
ORMs map relational databases to objects. Calcite does not do that. Calcite takes different data sources (potentially with different data models) and presents them all in the relational model. You interact with your data entirely in SQL while ORMs typically have some DSL which uses whatever object model the ORM defines. ORMs are also typically designed to connect to a single data source at a time. I'm personally not familiar with any ORMs that allow combining data from multiple sources within the same application. The primary advantage of Calcite for connecting multiple data sources are 1) easily joining data sources that use completely different APIs (assuming there is a Calcite adapter available) and 2) supporting more complex queries than the original data source supports without having to write code (other than SQL) to do the processing.
- gavinray 5y ago
  Don't forget having two industrial-grade academic query planners/optimizer implementations that took collective decades of engineering effort! And the project being founded by someone who has written multiple RDBMS. Calcite is a wonder.

wuputah 5y ago

Calcite is definitely some cool tech. I can see why it would be attractive for bigger teams, but it seems like a big lift for smaller teams. Our goal is to make it easy for devs already familiar with Postgres to be able to use add databases without learning new tools or adding software... besides adding Hydra, of course!

michaelmior 5y ago
That's a fair point. Although if you're already using a JVM language, it's incredibly easy to integrate. Just another JDBC data source with some JARs to add to your classpath :)

sdesol 5y ago

I've never heard of Calcite before and dug a bit deeper into this project and found it to be quite active. Only taking into consideration changes to Java files, this is the repo activity for the last 12 months: month | authors | commits | files | churn ---------+---------+---------+-------+------- 2022-02 | 21 | 62 | 84 | 4387 2022-01 | 26 | 109 | 243 | 32429 2021-12 | 33 | 99 | 198 | 10461 2021-11 | 19 | 49 | 77 | 6960 2021-10 | 26 | 64 | 371 | 13626 2021-09 | 18 | 41 | 68 | 2258 2021-08 | 11 | 17 | 25 | 1924 2021-07 | 17 | 30 | 51 | 2704 2021-06 | 14 | 31 | 28 | 1708 2021-05 | 9 | 17 | 35 | 1606 2021-04 | 11 | 46 | 99 | 4224 2021-03 | 16 | 36 | 143 | 8471

riverdroid 5y ago
What do you use to produce this analysis?
- sdesol 5y ago
  My product (https://gitsense.com https://gitsense.com) moves most of Git's history into a Postgres database and from there, you can execute the following SQL statement: select commit_ym AS month, count(distinct(author_email)) as authors, count(distinct(commit_id)) as commits, count(distinct(path_id)) as files, sum(total) as churn from z1_commits_422 as commits, z1_changes_422 as changes, z1_code_churn_422 as churn where commits.id=changes.commit_id and changes.code_churn_id=churn.id and lang='java' group by commit_ym order by commit_ym desc limit 12 By having most of Git's history in SQL, I can slice, dice and cross-reference code history, which is how my product works.
michaelmior 5y ago
The community is also great. Honestly, the main thing preventing things from moving faster is lack of time to review PRs. There's a lot of interesting things in the pipe!

gavinray 5y ago

Calcite is actually pretty damn fast, the overhead is surprisingly minimal. I am using it as the backbone for my hobby project that auto-generated federated GraphQL API's: https://github.com/GavinRay97/GraphQLCalcite https://github.com/GavinRay97/GraphQLCalcite The experience has been incredibly positive and the community has been incredibly helpful & supportive. It's one of the coolest technical projects I've ever seen and has sparked my interest in query engines and relational databases. I posted some JMH benchmarks of an app that parses a GraphQL query, converts it to Calcite relational expressions, and then executes it against an in-memory DB and it ran on the orders of milliseconds: https://lists.apache.org/thread/hofjx628864t0kt4kk8vo4tjfrxbqf7o https://lists.apache.org/thread/hofjx628864t0kt4kk8vo4tjfrxb... Something very similar to Calcite but much lesser-known is the "Teiid" project: https://github.com/teiid/teiid https://github.com/teiid/teiid Highly recommend checking the code out. It's got a brilliant query optimizer/planner tailored to cross-datasource queries, a cache system, and a translator architecture that can convert a generic SQL dialect to dozens of SQL/NoSQL flavors. Also integrates other data sources like REST API's, S3, flat files etc as queryable data sources.

michaelmior 5y ago
I was moreso referring to the pieces of queries that Calcite may have to execute itself, often because they are not supported by the underlying data source. That piece of Calcite hasn't been very highly optimized. There's definitely ongoing work in that area though. For example, I have a former student who's been working on an adapter to Apache Arrow. Beyond having a fast execution engine, this should also eventually enable Calcite to work pretty seamlessly with other systems using Arrow without serialization overhead.
- gavinray 5y ago
  Is this student in contact with Jacques Nadeau? He's one of the only people I know that's a committer/PMC of both Arrow and Calcite, super nice and brilliant guy. Maybe he'd have good feedback or ideas around this? https://twitter.com/intjesus https://twitter.com/intjesus
  - 5y ago