4 ms·
As I understand it, Arrow is about the data format, to best optimize query capabilities in any language. Think, “CapnProto” or “Protobufs” but for querying dat
by yazaddaruvala 5y ago
As I understand it, Arrow is about the data format, to best optimize query capabilities in any language.
Think, “CapnProto” or “Protobufs” but for querying data rather than only data transfer.
In theory Arrow will enable users to “trivially” create high performance SQLite type querying in any programming language. Arrow doesn’t help with any other features of SQLite tho.
Maybe one day Arrow will also target on-disc analytics/persistence but for now it delegates to Parquet.
- lmeyerov 5y agoProtobufs is a great comparison point and exactly what we upgraded from. It's like protobuf but specialized for rich data tables and, for those living outside google's proprietary layers, modern sensibilities. This brings wins like self-describing (no .proto, it optionally includes a schema), streaming, columnar by default, etc. By standardizing the notion of a table (and chunk within it, a record batch), it plugs into compute/io frameworks that work on tables: spark, pandas, etc. Most compute/db framework that gives you lists/tables will likely have out of the box support for it, and even when not explicit, via tools like turbodc for SQL DBs. Arrow has compute libs, but we don't use them (yet). More as interop for compute engines that are fast (ex: rapids for gpu) or feature rich (ex: pandas for chunks). Likewise, for I/O, interop for good formats there: on-the-fly arrow -> persistent parquet|orc. It's enabling us to do cool stuff like fast interop between our cpu<>gpu code, and in the latest initiative, crossing even process & language runtime boundaries w/ zero-copy.