4 ms·

When I first saw Arrow I thought the biggest benefit was a good file format. (Feather I think is the format. ) However it seems to be designed for passing table

When I first saw Arrow I thought the biggest benefit was a good file format. (Feather I think is the format. ) However it seems to be designed for passing tables between languages in the same process. If I never use multiple languages in the same process can I safely ignore it or are there other benefits?

tomrod 6y ago
To_feather and from_feather are a wonderful addition if you use R and Python, or have a mixed language data science shop.
- x87678r 6y ago
  Yes but is it any better than parquet or hdf? edit - this looks a good explanation, parquet is more widely used and smaller, but slower. https://stackoverflow.com/questions/48083405/what-are-the-differences-between-feather-and-parquet https://stackoverflow.com/questions/48083405/what-are-the-di...
  - kylebarron 6y ago
    The compression improvements in Parquet can be really great, especially for large, wide tables where you might only need a few columns for analysis.
  - tomrod 6y ago
    Hdf I've not used in any production workflow. Parquet I've really only used with impala. If I'm working in that system I'm in Spark already, typically.
poorman 6y ago
It also abstracts away optimizations for doing operations. For example tolowercase on a large number of elements.
lmeyerov 6y ago
In Python, we use Arrow a lot for going between diff frameworks: file -> ( pandas <> cudf <> cugraph ). When we work with DB vendors, we now push them to provide Arrow instead of whatever proprietary in-house thing they've been pushing for the same, e.g., slow JSON / ODBC, untyped CSV, or some ad-hoc wire protocol. Also, we use for going between our services within a server + across. On the nodejs services + frontend JS side, there was no real equiv tool for interop -- traditional soln is slowly round-tripping through SQL/ORM or manually doing protobuf through some sort of pubsub -- so Arrow is part of how we have been taming that mess too. It'll be a longer journey for Arrow or something like it to get adopted in JS land, but I can see folks doing TypeScript and serverless wanting it for a cleaner solution to typed data, faster serialization & streaming, etc. (There is no true JS equiv of pandas nor a typed variant.) We were a bit early here b/c we wanted streaming through WebGL + OpenCL/CUDA, and while we are fans of typed data, found protobuf tooling to be too unintegrated and manual in practice.
- infinite8s 6y ago
  How successful have you been in pushing database vendors to use arrow on the wire? We've started using turbodbc (to connect to SQL Server), but as I understand it the data on the wire is still in odbc's row-oriented wire format, and turbodbc is responsible for packing into arrow column oriented buffers. As you mention in your second paragraph, arrow is perfect for getting typed query results from a database server directly to a web frontend with minimal overhead. With the Transferable interface, it's even possible to zero-copy transfer the arrow data from the network buffer to a webworker.

tomrod 6y ago

To_feather and from_feather are a wonderful addition if you use R and Python, or have a mixed language data science shop.

x87678r 6y ago
Yes but is it any better than parquet or hdf? edit - this looks a good explanation, parquet is more widely used and smaller, but slower. https://stackoverflow.com/questions/48083405/what-are-the-differences-between-feather-and-parquet https://stackoverflow.com/questions/48083405/what-are-the-di...
- kylebarron 6y ago
  The compression improvements in Parquet can be really great, especially for large, wide tables where you might only need a few columns for analysis.
- tomrod 6y ago
  Hdf I've not used in any production workflow. Parquet I've really only used with impala. If I'm working in that system I'm in Spark already, typically.

poorman 6y ago

It also abstracts away optimizations for doing operations. For example tolowercase on a large number of elements.

lmeyerov 6y ago

In Python, we use Arrow a lot for going between diff frameworks: file -> ( pandas <> cudf <> cugraph ). When we work with DB vendors, we now push them to provide Arrow instead of whatever proprietary in-house thing they've been pushing for the same, e.g., slow JSON / ODBC, untyped CSV, or some ad-hoc wire protocol. Also, we use for going between our services within a server + across. On the nodejs services + frontend JS side, there was no real equiv tool for interop -- traditional soln is slowly round-tripping through SQL/ORM or manually doing protobuf through some sort of pubsub -- so Arrow is part of how we have been taming that mess too. It'll be a longer journey for Arrow or something like it to get adopted in JS land, but I can see folks doing TypeScript and serverless wanting it for a cleaner solution to typed data, faster serialization & streaming, etc. (There is no true JS equiv of pandas nor a typed variant.) We were a bit early here b/c we wanted streaming through WebGL + OpenCL/CUDA, and while we are fans of typed data, found protobuf tooling to be too unintegrated and manual in practice.

infinite8s 6y ago
How successful have you been in pushing database vendors to use arrow on the wire? We've started using turbodbc (to connect to SQL Server), but as I understand it the data on the wire is still in odbc's row-oriented wire format, and turbodbc is responsible for packing into arrow column oriented buffers. As you mention in your second paragraph, arrow is perfect for getting typed query results from a database server directly to a web frontend with minimal overhead. With the Transferable interface, it's even possible to zero-copy transfer the arrow data from the network buffer to a webworker.