5 ms·

If you want to take part in a community effort to fix this problem, consider contributing the to Apache Calcite project. It has relational algebra that spans ac

If you want to take part in a community effort to fix this problem, consider contributing the to Apache Calcite project. It has relational algebra that spans across many different databases and dialects, including NoSQL databases like MongoDB and Elasticsearch. I haven't been very involved with the project directly, but I've worked on several projects that heavily depended on it, and have written extensions for those projects.

xyzzy_plugh 9y ago
I took a peek into calcite some time ago and was disappointed. The project seems to be overly bureaucratic, and the code quality quite low. (Side note, I've noticed a steady decline in quality of Apache projects for at least the past decade or so.) I wanted to perform some introspection on queries in-flight, and potentially re-write them, but this seemed like an awful chore with calcite. One, calcite didn't implement my dialect -- it looked non-trivial to add new dialects. Two, going from query to parse tree (with metadata) back to query doesn't seem to be something that's intended to be supported. Calcite is really a replacement frontend with its own SQL dialect that you can tweak the compiled output (SQL, CQL, etc.). It has its own worts and is not generally applicable to other projects. It works well within the Apache ecosystem but I don't envision much adoption outside of it (especially non-Java, like Postgres). If you were writing a Tableau replacement, you might consider using Calcite to generate your queries. I don't see many other use cases.
- derefr 9y ago
  > Side note, I've noticed a steady decline in quality of Apache projects for at least the past decade or so. That's a weird trend to notice, given that none of the same developers are working on "Apache projects" generally. The ASF isn't like e.g. Mozilla; it isn't a monolithic org with its own devs. It's a meta-bureaucracy that takes ownership of projects for legal (copyright + patent assignment) reasons, and then offers project-management charters to member projects, giving them process and structure for contribution without a canonical owner. An ASF project is sort of like a "working group" (e.g. Khronos, WHATWG), except that it's usually one or two large corporations and a bunch of FOSS devs, rather than being exclusively a bunch of corporations. --- On the other hand, if there is a trend, it might be because of the increasing reliance on the "open core" strategy for corporate FOSS contributors to make money. My own experience is with Cloudant, who contribute to Apache CouchDB, but also have their own private fork of CouchDB running on their SaaS. Originally, Cloudant did what they liked in their own fork, and then tried to push the results upstream. The ASF didn't like this very much, since these contributions weren't planned in the open, so Cloudant increasingly tried instead to mirror its internal issues as ASF Bugzilla issues and get public-member sign-off before going forward on private solutions. Which is kind of a shadow play, since many of the founding members of the CouchDB ASF project either have always worked for Cloudant, or have since been hired by Cloudant, so it's Cloudant employees (in their roles as ASF project members) signing off on these Cloudant issues. But it still slows things down. A good comparison that people might be familiar with, with the same dynamics, is WebKit. The WebKit project has its own separate development organization, but most of the developers happen to also work for Apple. Previously, WebKit was Apple and Google, but even two corporate contributors were too big for the small pond. Which, to me, shows that they were each there expecting to dominate the decision-making process, rather than find consensus with the FOSS contributors; and having an equally-powerful player that they had to form consensus with was too much for them.
  - anarazel 9y ago
    How's that dispositive of a declining average code quality? ASF has influence over which projects get incubated, and even some over how projects are managed.
    - deleted 9y ago
      [deleted]
- jaltekruse 9y ago
  I don't think it is fair to say the code quality is low. There are a lot of complex concepts in the optimizer and the community is always working to add tooling and documentation to make it easier for newer developers. There are some organizational quirks to how the code is written. It has been around for a long time, so it still has some design elements from working within older Java versions. Like many projects it is reliant on integration tests, likely more than would be desirable. As far as the wider Apache ecosystem, I know there are projects that get by with pretty low code quality. Unfortunately there aren't many central voices at Apache enforcing specific policies around how codebases are managed, they are more focused on community development. I think they may be better off trying to take a closer look at codebases during incubation. Then again, there is no excess of resources waiting around to review code, and a lot of budding communities that want to take part.
- altstar 9y ago
  You are more than welcome to join any Apache project and improve the code quality.

xyzzy_plugh 9y ago

I took a peek into calcite some time ago and was disappointed. The project seems to be overly bureaucratic, and the code quality quite low. (Side note, I've noticed a steady decline in quality of Apache projects for at least the past decade or so.) I wanted to perform some introspection on queries in-flight, and potentially re-write them, but this seemed like an awful chore with calcite. One, calcite didn't implement my dialect -- it looked non-trivial to add new dialects. Two, going from query to parse tree (with metadata) back to query doesn't seem to be something that's intended to be supported. Calcite is really a replacement frontend with its own SQL dialect that you can tweak the compiled output (SQL, CQL, etc.). It has its own worts and is not generally applicable to other projects. It works well within the Apache ecosystem but I don't envision much adoption outside of it (especially non-Java, like Postgres). If you were writing a Tableau replacement, you might consider using Calcite to generate your queries. I don't see many other use cases.

derefr 9y ago
> Side note, I've noticed a steady decline in quality of Apache projects for at least the past decade or so. That's a weird trend to notice, given that none of the same developers are working on "Apache projects" generally. The ASF isn't like e.g. Mozilla; it isn't a monolithic org with its own devs. It's a meta-bureaucracy that takes ownership of projects for legal (copyright + patent assignment) reasons, and then offers project-management charters to member projects, giving them process and structure for contribution without a canonical owner. An ASF project is sort of like a "working group" (e.g. Khronos, WHATWG), except that it's usually one or two large corporations and a bunch of FOSS devs, rather than being exclusively a bunch of corporations. --- On the other hand, if there is a trend, it might be because of the increasing reliance on the "open core" strategy for corporate FOSS contributors to make money. My own experience is with Cloudant, who contribute to Apache CouchDB, but also have their own private fork of CouchDB running on their SaaS. Originally, Cloudant did what they liked in their own fork, and then tried to push the results upstream. The ASF didn't like this very much, since these contributions weren't planned in the open, so Cloudant increasingly tried instead to mirror its internal issues as ASF Bugzilla issues and get public-member sign-off before going forward on private solutions. Which is kind of a shadow play, since many of the founding members of the CouchDB ASF project either have always worked for Cloudant, or have since been hired by Cloudant, so it's Cloudant employees (in their roles as ASF project members) signing off on these Cloudant issues. But it still slows things down. A good comparison that people might be familiar with, with the same dynamics, is WebKit. The WebKit project has its own separate development organization, but most of the developers happen to also work for Apple. Previously, WebKit was Apple and Google, but even two corporate contributors were too big for the small pond. Which, to me, shows that they were each there expecting to dominate the decision-making process, rather than find consensus with the FOSS contributors; and having an equally-powerful player that they had to form consensus with was too much for them.
- anarazel 9y ago
  How's that dispositive of a declining average code quality? ASF has influence over which projects get incubated, and even some over how projects are managed.
  - deleted 9y ago
    [deleted]
jaltekruse 9y ago
I don't think it is fair to say the code quality is low. There are a lot of complex concepts in the optimizer and the community is always working to add tooling and documentation to make it easier for newer developers. There are some organizational quirks to how the code is written. It has been around for a long time, so it still has some design elements from working within older Java versions. Like many projects it is reliant on integration tests, likely more than would be desirable. As far as the wider Apache ecosystem, I know there are projects that get by with pretty low code quality. Unfortunately there aren't many central voices at Apache enforcing specific policies around how codebases are managed, they are more focused on community development. I think they may be better off trying to take a closer look at codebases during incubation. Then again, there is no excess of resources waiting around to review code, and a lot of budding communities that want to take part.
altstar 9y ago
You are more than welcome to join any Apache project and improve the code quality.