5 ms·

Relational Operators in Apache Calcite (2021)

60 points by yeesian 4y ago

maxloh 4y ago
I once heard that Apache is the graveyard for open source software. Is that really the case?
- mickeyp 4y ago
  It is and it isn't. There's some good stuff there, but there's ever the risk that you'll get bushwhacked by something that you need to pay an expensive consultancy -- who just so happens to be the team behind the product -- to fix. Though I feel bad for the person who decides to peruse Apache's list of projects and decide what technology to use based solely on the high-level overviews. Good luck picking the wheat from the chaff!
  - michaelmior 4y ago
    I'll just note that the risk mentioned certainly does not apply to Apache Calcite. (Disclaimer: I am a member of the Apache Calcite PMC, but not terribly active at the moment.)
- jamesblonde 4y ago
  A lot of open-source Apache projects were backed by commercial companies, and clouds (primarly AWS) forked those projects competing directly with the companies backing the Apache projects. The end result was that the commercial companies mostly released new versions of the software with more restrictive licenses that could not be sold for free by the cloud companies. That doesn't make Apache software a graveyard. It just makes that go-to-market open-source model leveraging Apache's brand a dead-end.
- di4na 4y ago
  Yes and no. A lot of projects in Apache control started as one (big) company project, got open sourced then move to Apache, in order to allow collaboration with a "community" without as much conflict of interest. But then the original sponsor also usually lose interest and/or no community get creates. Hence, graveyard. Or it got opensource as last ditch try to keep it running after the sponsor stopped wanting it. And then it is usually a graveyard move. It is not that Apache kills them. It is that they were already dead and Apache is the place that take them in to accompany their demise.
- gdubya 4y ago
  No
- kitd 4y ago
  It depends. There are a large number of dead projects. There are also a large number of libraries and tools that are active and widely depended on, but you may not have heard of them. Then there are quite a few large projects donated by corps with active communities. See this page https://projects.apache.org/statistics.html https://projects.apache.org/statistics.html and scroll down to "Largest/Busiest repositories". There's a list of widely used enterprise software in there which are definitely not dead.
- Podollin 4y ago
  to me it feels the same, plenty of great projects that essentially died once they got into apache. It has that java feeling.
- michaelmior 4y ago
  It's certainly not the case for Apache Calcite. Calcite is pretty widely used by a number of companies and there are a number of highly active developers.
- BenoitP 4y ago
  Calcite is relatively unknown, but it is the base so many other projects are built on. Among them is Apache Flink, which is quite alive and top of its category (low latency distributed stream processing) https://calcite.apache.org/docs/powered_by.html https://calcite.apache.org/docs/powered_by.html
noisy_boy 4y ago
I recently had to parse source code of views to produce a mapping between the view columns and the underlying table columns - it basically worked with some caveats and was pretty painful to do (most databases expose this as information schema/metadata but that was not an option for us for various reasons). For anyone who is knowledgable about Calcite features, is this something that Calcite can produce without having to write a ton of code (which is what my current solution is)? I would find it very satisfying to replace hundreds of lines of bespoke code with a few library calls.
- michaelmior 4y ago
  This should be pretty easily doable in Calcite if you have access to the query which defines the view. Calcite has a pretty rich metadata system. Take a look at RelMetadataQuery.getColumnOrigin.
hobofan 4y ago
I'd recommend looking into Substrait[0] for anyone interested in this mid-/low-level database/query engine space. It tries to establish a vendor-agnostic set of relational operators (and all the things surrounding it) to make it easier for query engine implementations to share individual components. [0]: https://substrait.io https://substrait.io
- yeesian 4y ago
  That's cool, thanks for sharing! Do you know how close they are to their example use cases [1]? So far I've only been able to find a tool for calcite SQL parsing [2] but not the portion connecting to Arrow C++ compute kernel yet. [1]: https://substrait.io/#example-use-cases https://substrait.io/#example-use-cases [2]: https://substrait.io/tools/producer_tools/ https://substrait.io/tools/producer_tools/
  - hobofan 4y ago
    I'd check out the Slack, which is where I've seen a few projects integrating it coordinating. I think DuckDB is one of the projects that has the best support for executing Substrait query plans. I think for most other projects there are some forks with substrait support (e.g. datafusion) but nothing merged to upstream yet. I'm not sure if there are any systems where it is integrated and yields tangible benefits yet (though there is decent progress on common tooling, so that shouldn't be too far in the future).

60 points by yeesian 4y ago

maxloh 4y ago

I once heard that Apache is the graveyard for open source software. Is that really the case?

mickeyp 4y ago
It is and it isn't. There's some good stuff there, but there's ever the risk that you'll get bushwhacked by something that you need to pay an expensive consultancy -- who just so happens to be the team behind the product -- to fix. Though I feel bad for the person who decides to peruse Apache's list of projects and decide what technology to use based solely on the high-level overviews. Good luck picking the wheat from the chaff!
- michaelmior 4y ago
  I'll just note that the risk mentioned certainly does not apply to Apache Calcite. (Disclaimer: I am a member of the Apache Calcite PMC, but not terribly active at the moment.)
jamesblonde 4y ago
A lot of open-source Apache projects were backed by commercial companies, and clouds (primarly AWS) forked those projects competing directly with the companies backing the Apache projects. The end result was that the commercial companies mostly released new versions of the software with more restrictive licenses that could not be sold for free by the cloud companies. That doesn't make Apache software a graveyard. It just makes that go-to-market open-source model leveraging Apache's brand a dead-end.
di4na 4y ago
Yes and no. A lot of projects in Apache control started as one (big) company project, got open sourced then move to Apache, in order to allow collaboration with a "community" without as much conflict of interest. But then the original sponsor also usually lose interest and/or no community get creates. Hence, graveyard. Or it got opensource as last ditch try to keep it running after the sponsor stopped wanting it. And then it is usually a graveyard move. It is not that Apache kills them. It is that they were already dead and Apache is the place that take them in to accompany their demise.
gdubya 4y ago
No
kitd 4y ago
It depends. There are a large number of dead projects. There are also a large number of libraries and tools that are active and widely depended on, but you may not have heard of them. Then there are quite a few large projects donated by corps with active communities. See this page https://projects.apache.org/statistics.html https://projects.apache.org/statistics.html and scroll down to "Largest/Busiest repositories". There's a list of widely used enterprise software in there which are definitely not dead.
Podollin 4y ago
to me it feels the same, plenty of great projects that essentially died once they got into apache. It has that java feeling.
michaelmior 4y ago
It's certainly not the case for Apache Calcite. Calcite is pretty widely used by a number of companies and there are a number of highly active developers.
BenoitP 4y ago
Calcite is relatively unknown, but it is the base so many other projects are built on. Among them is Apache Flink, which is quite alive and top of its category (low latency distributed stream processing) https://calcite.apache.org/docs/powered_by.html https://calcite.apache.org/docs/powered_by.html

noisy_boy 4y ago

I recently had to parse source code of views to produce a mapping between the view columns and the underlying table columns - it basically worked with some caveats and was pretty painful to do (most databases expose this as information schema/metadata but that was not an option for us for various reasons). For anyone who is knowledgable about Calcite features, is this something that Calcite can produce without having to write a ton of code (which is what my current solution is)? I would find it very satisfying to replace hundreds of lines of bespoke code with a few library calls.

michaelmior 4y ago
This should be pretty easily doable in Calcite if you have access to the query which defines the view. Calcite has a pretty rich metadata system. Take a look at RelMetadataQuery.getColumnOrigin.

hobofan 4y ago

I'd recommend looking into Substrait[0] for anyone interested in this mid-/low-level database/query engine space. It tries to establish a vendor-agnostic set of relational operators (and all the things surrounding it) to make it easier for query engine implementations to share individual components. [0]: https://substrait.io https://substrait.io

yeesian 4y ago
That's cool, thanks for sharing! Do you know how close they are to their example use cases [1]? So far I've only been able to find a tool for calcite SQL parsing [2] but not the portion connecting to Arrow C++ compute kernel yet. [1]: https://substrait.io/#example-use-cases https://substrait.io/#example-use-cases [2]: https://substrait.io/tools/producer_tools/ https://substrait.io/tools/producer_tools/
- hobofan 4y ago
  I'd check out the Slack, which is where I've seen a few projects integrating it coordinating. I think DuckDB is one of the projects that has the best support for executing Substrait query plans. I think for most other projects there are some forks with substrait support (e.g. datafusion) but nothing merged to upstream yet. I'm not sure if there are any systems where it is integrated and yields tangible benefits yet (though there is decent progress on common tooling, so that shouldn't be too far in the future).