3 ms·

tldr; it's a new wire protocol that columnar databases can opt into that is better optimized for columnar data than ODBC/JDBC. Key quotes: While it can be

tldr; it's a new wire protocol that columnar databases can opt into that is better optimized for columnar data than ODBC/JDBC. Key quotes: While it can be directly used for database access, it is not a direct replacement for JDBC/ODBC. Instead, Flight SQL serves as a concrete wire protocol/driver implementation that can support a JDBC/ODBC driver and reduces implementation burden on databases. Flight SQL means database servers can implement a standard interface that is designed around Apache Arrow and columnar data from the start. Just like how Arrow provides a standard in-memory format, Flight SQL saves developers from having to design and implement an entirely new wire protocol. As mentioned, Flight already implements features like encryption on the wire and authentication of requests, which databases do not need to re-implement.

gavinray 5y ago
This is true, but to clarify and expand a bit, you add support for FlightSQL to some datasource by adding Protobuf RPC handlers to it. This doesn't necessarily have to run in-process, the FlightSQL RPC service might run side-by-side. The datasource also doesn't necessarily have to be a database. It just needs to implement the RPC methods. So you could implement your own data-access layer over any datasource (flat file, an external API, a regular DBMS, etc) by providing an implementation for "get all schemas", "get all tables", "execute a query", etc. Some useful quotes: > The idea is that the driver is a “one-size-fits-all” driver — a user or tool vendor only needs to supply a generic driver that can connect to an infinite number of databases. This is even future-proof — if a new database comes out, it can work with existing tools as long as an Arrow Flight SQL endpoint is provided. In fact, by adding an Arrow Flight SQL endpoint they would automatically enable JDBC connectivity too. > Not only will Arrow Flight SQL reduce the technical burden on applications and users, but it leverages Arrow, which means it will provide better performance. ... having a single reference JDBC driver allows any data source that adds an Arrow Flight SQL endpoint to get JDBC “for free” as an onramp. So the selling point is add an Arrow Flight SQL endpoint to your data source and automatically get JDBC connectivity. I have been experimenting in my free time with building a platform that autogenerates GraphQL CRUD API's on top of arbitrary datasources. It lets you do federated/distributed queries and cross-databsource joins. So you can connect it to a Postgres DB, a MySQL DB, a MongoDB, and a CSV file and then run a query that joins and filters data across them in a single operation. I am using Apache Calcite for this, but am interested in potentially offloading the query plan execution using FlightSQL/Substrait[0] for better performance, since I am targeting OLTP workloads and it's latency-sensitive. https://github.com/GavinRay97/GraphQLCalcite https://github.com/GavinRay97/GraphQLCalcite [0] https://substrait.io/ https://substrait.io/

gavinray 5y ago

This is true, but to clarify and expand a bit, you add support for FlightSQL to some datasource by adding Protobuf RPC handlers to it. This doesn't necessarily have to run in-process, the FlightSQL RPC service might run side-by-side. The datasource also doesn't necessarily have to be a database. It just needs to implement the RPC methods. So you could implement your own data-access layer over any datasource (flat file, an external API, a regular DBMS, etc) by providing an implementation for "get all schemas", "get all tables", "execute a query", etc. Some useful quotes: > The idea is that the driver is a “one-size-fits-all” driver — a user or tool vendor only needs to supply a generic driver that can connect to an infinite number of databases. This is even future-proof — if a new database comes out, it can work with existing tools as long as an Arrow Flight SQL endpoint is provided. In fact, by adding an Arrow Flight SQL endpoint they would automatically enable JDBC connectivity too. > Not only will Arrow Flight SQL reduce the technical burden on applications and users, but it leverages Arrow, which means it will provide better performance. ... having a single reference JDBC driver allows any data source that adds an Arrow Flight SQL endpoint to get JDBC “for free” as an onramp. So the selling point is add an Arrow Flight SQL endpoint to your data source and automatically get JDBC connectivity. I have been experimenting in my free time with building a platform that autogenerates GraphQL CRUD API's on top of arbitrary datasources. It lets you do federated/distributed queries and cross-databsource joins. So you can connect it to a Postgres DB, a MySQL DB, a MongoDB, and a CSV file and then run a query that joins and filters data across them in a single operation. I am using Apache Calcite for this, but am interested in potentially offloading the query plan execution using FlightSQL/Substrait[0] for better performance, since I am targeting OLTP workloads and it's latency-sensitive. https://github.com/GavinRay97/GraphQLCalcite https://github.com/GavinRay97/GraphQLCalcite [0] https://substrait.io/ https://substrait.io/