5 ms·
This is true, but to clarify and expand a bit, you add support for FlightSQL to some datasource by adding Protobuf RPC handlers to it. This doesn't necessarily
by gavinray 5y ago
This is true, but to clarify and expand a bit, you add support for FlightSQL to some datasource by adding Protobuf RPC handlers to it. This doesn't necessarily have to run in-process, the FlightSQL RPC service might run side-by-side.
The datasource also doesn't necessarily have to be a database. It just needs to implement the RPC methods. So you could implement your own data-access layer over any datasource (flat file, an external API, a regular DBMS, etc) by providing an implementation for "get all schemas", "get all tables", "execute a query", etc.
Some useful quotes:
> The idea is that the driver is a “one-size-fits-all” driver — a user or tool vendor only needs to supply a generic driver that can connect to an infinite number of databases. This is even future-proof — if a new database comes out, it can work with existing tools as long as an Arrow Flight SQL endpoint is provided. In fact, by adding an Arrow Flight SQL endpoint they would automatically enable JDBC connectivity too.
> Not only will Arrow Flight SQL reduce the technical burden on applications and users, but it leverages Arrow, which means it will provide better performance. ... having a single reference JDBC driver allows any data source that adds an Arrow Flight SQL endpoint to get JDBC “for free” as an onramp. So the selling point is add an Arrow Flight SQL endpoint to your data source and automatically get JDBC connectivity.
I have been experimenting in my free time with building a platform that autogenerates GraphQL CRUD API's on top of arbitrary datasources.
It lets you do federated/distributed queries and cross-databsource joins.
So you can connect it to a Postgres DB, a MySQL DB, a MongoDB, and a CSV file and then run a query that joins and filters data across them in a single operation.
I am using Apache Calcite for this, but am interested in potentially offloading the query plan execution using FlightSQL/Substrait[0] for better performance, since I am targeting OLTP workloads and it's latency-sensitive.
https://github.com/GavinRay97/GraphQLCalcite https://github.com/GavinRay97/GraphQLCalcite
[0] https://substrait.io/ https://substrait.io/