4 ms·
This is an awesome utility and great that it was open sourced. With even more SQL dialects, this can get even more interesting as one could try to understand u
by sixdimensional 9y ago
This is an awesome utility and great that it was open sourced. With even more SQL dialects, this can get even more interesting as one could try to understand usage across heterogeneous databases.
I'm going to go all the way with this thought, as it is something I've been wanting to do for a long time.
I've been wanting to build an open source query proxy server. The idea is that, it would sit in front of any database that accepts any type of query expression (first use case, SQL, of course). You would register database drivers with the platform and then clients wouldn't need them anymore.
The proxy would accept queries destined for targets via a service call. Traffic to the service can be secured and encrypted. It would be able to be deployed in a cluster, so there are multiple endpoints to hit to load balance traffic.
It would parse all queries, recording the metadata about the queries and destination servers / tables / objects using something like Queryparser, so you'd have the benefit of all this metadata, but now you'd have it across data sources too.
This kind of data can also help power a data catalog, which could be a web app on top of the metadata / query repository, to help users find and/or annotate the data flowing around their organization (e.g. data democracy).
It could also have caching for result sets, additional security measures (RBAC), or even be linked up with a processing layer for additional processing (like Apache Spark).
There are commercial solutions for this, but I want to see an open source one. Especially because, I'd like to see all dialects and query type expressions supported, and I think only an open source project could get this kind of support across industry players.
One way I have thought about this type of query proxy is, it's a bit like a GraphQL back-end web service endpoint but with SQL instead of GraphQL as the interface. Using the native SQL/query expressions of the destination platform would simplify the work, since one would not need to create a generic query expression abstraction. We just need a way to extract the relevant information from the dialect/query that we are receiving.
We can let data platforms have different query semantics/dialects but still get metadata. For example, there are ways to map concepts for MongoDb to relational ones (collections are like tables, fields are like columns, etc.) so even a MongoDb query expression could go through the query proxy, and have metadata recorded in a consistent format, as long as you could express it to the query proxy's web service interface.
Maybe I'm just daydreaming.. but I would love to work on that and let it be open source.
- dllthomas 9y agoI really wanted to see it used for type checking (of domain-relevant types, not just the sql representational types), but I left before that got anywhere.
- contingencies 9y agoIIRC 10+ years ago MySQL already had one of these and it was Lua based, the primary intent was for transparent sharding.
- rzzzt 9y agoYou might be thinking of this project: https://github.com/mysql/mysql-proxy https://github.com/mysql/mysql-proxy
- contingencies 9y agoYep.
- elserj 9y agoSince someone has already mentioned Apache Calcite once... Take a look at https://calcite.apache.org/avatica/ https://calcite.apache.org/avatica/ -- it does some of what you're mentioning.