6 ms·

Arrow is the most important thing happening in the data ecosystem right now. It's going to allow you to run your choice of execution engine, on top of your choi

Arrow is the most important thing happening in the data ecosystem right now. It's going to allow you to run your choice of execution engine, on top of your choice of data store, as though they are designed to work together. It will mostly be invisible to users, the key thing that needs to happen is that all the producers and consumers of batch data need to adopt Arrow as the common interchange format. BigQuery recently implemented the storage API, which allows you to read BQ tables, in parallel, in Arrow format: https://cloud.google.com/bigquery/docs/reference/storage https://cloud.google.com/bigquery/docs/reference/storage Snowflake has adopted Arrow as the in-memory format for their JDBC driver, though to my knowledge there is still no way to access data in parallel from Snowflake, other than to export to S3. As Arrow spreads across the ecosystem, users are going to start discovering that they can store data in one system and query it in another, at full speed, and it's going to be amazing.

tristanz 6y ago
Agreed! Thank you Arrow community for such a great project. It's a long road but it opens up tremendous potential for efficient data systems that talk to one another. The future looks bright with so many vendors backing Arrow independently and Wes McKinney founding Ursa Labs and now Ursa Computing. https://ursalabs.org/blog/ursa-computing/ https://ursalabs.org/blog/ursa-computing/
wesm 6y ago
Microsoft is also on top of this with their Magpie project http://cidrdb.org/cidr2021/papers/cidr2021_paper08.pdf http://cidrdb.org/cidr2021/papers/cidr2021_paper08.pdf "A common, efficient serialized and wire format across data engines is a transformational development. Many previous systems and approaches (e.g., [26, 36, 38, 51]) have observed the prohibitive cost of data conversion and transfer, precluding optimizers from exploiting inter-DBMS performance advantages. By contrast, inmemory data transfer cost between a pair of Arrow-supporting systems is effectively zero. Many major, modern DBMSs (e.g., Spark, Kudu, AWS Data Wrangler, SciDB, TileDB) and data-processing frameworks (e.g., Pandas, NumPy, Dask) have or are in the process of incorporating support for Arrow and ArrowFlight. Exploiting this is key for Magpie, which is thereby free to combine data from different sources and cache intermediate data and results, without needing to consider data conversion overhead."
- data_ders 6y ago
  way cool! Is magpie end-user facing anywhere yet? We were using the azureml-dataprep library for a while which seems similar but not all of magpie
- polskibus 6y ago
  I wish MS put in some resources behind Arrow in .NET. I tried raising some remarks about it on dotnet repos (esp. within ML.NET), but to no avail. Hopefully it would change now that Arrow is more popular, and also written about by MS itself.
nomel 6y ago
Until arrow has proper support for multidimensional arrays [1], its not really appropriate for many use cases. Not having first class support for multidimensional arrays, in a modern framework, really surprised me and my sensor data. [1] https://lists.apache.org/x/thread.html/9b142c1709aa37dc35f1ce8db4e1ced94fcc4cdd96cc72b5772b373b@%3Cdev.arrow.apache.org%3E https://lists.apache.org/x/thread.html/9b142c1709aa37dc35f1c...
- lmeyerov 6y ago
  FWIW, you can support tensors etc. natively today by using Arrow's structs, such as `column i_am_packed: {x: int32, y: int32, z: int32, time: datetime64[ns], temp: int32}`. This will be a dense, packed representation for whatever fixed type the column has, including dynamically generated. It ends up being quite nice to have that passed around, esp. vs say protobufs. You wouldn't be sending mixed-length values, and ~all metrics systems I've worked with would be fine there. The nullables end up a win for those. However, you'd need to do a manual (likely zero-copy) casts between the struct-type to whatever tensor-type you're using. Massively popular ml systems like huggingface do this fine afaict for their Arrow-based tensor work. Likewise, as we do a lot of GPU stuff, what's additionally common is compacting the in-memory stuff as big memory blocks ('long recordbatches') instead of CPU-land's typically more fragmented ones, and that ends up making casts even easier. Annoying to have to add an explicit cast for some interop cases, but preserves end-to-end type safety & hasn't been a deal breaker for us. Having to write the cast being annoying/difficult, esp. for whoever does it first. More frustrating for us has been sparse data and compression controls, but most formats are even worse here..
  - deleted 6y ago
    [deleted]
- wesm 6y ago
  Almost no database systems support multidimensional arrays. So they are not appropriate for many use cases? * BigQuery: no * Redshift: no * Spark SQL: no * Snowflake: no * Clickhouse: no * Dremio: no * Impala: no * Presto: no ... list continues We've invited developers to add the extension types for tensor data, but no one has contributed them yet. I'm not seeing a lot of tabular data with embedded tensors out in the wild.
waynesonfire 6y ago
Uhh.. maybe. It's a serde that's trying to be cross-language / platform. I guess it also offers some APIs to process the data so you can minimize serde operations. But, I dunno. It's been hard to understand the benefit of the libabry and the posts here don't help.
- TuringTest 6y ago
  If it works as a universal intermediate exchange language, it could help standardize connections among disparate systems. When you have N systems, it takes N^2 translators to build direct connections to transfer data between them; but it only takes N translators if all them can talk the same exchange language.
  - waynesonfire 6y ago
    can you define what at translator is? I don't understand the complexity you're constructing. I have N systems and they talk protobuf. What's the problem?
    - TuringTest 6y ago
      By a translator, I mean a library that allows accessing data from different subsystems (either languages or OS processes). In this case, the advantages are that 1) Arrow is language agnostic, so it's likely that it can be used as a native library in your program and 2) it doesn't copy data to make it accessible to another process, so it saves a lot of marshalling / unmarshalling steps (assuming both sides use data in tabular format, which is typical of data analysis contexts).
    - deleted 6y ago
      [deleted]
- mumblemumble 6y ago
  It's not just a serde. One of its key use cases is eliminating serde.
rubicon33 6y ago
Wait so you're telling me I could store data as a PDF file, and access it easily / quickly as SQL?
- sethhochberg 6y ago
  If you found/wrote a adapter to translate your structured PDF into Arrow's format, yes - the idea is that you can wire up anything that can produce Arrow data to anything that can consume Arrow data.
  - staticassertion 6y ago
    I'm kinda confused. Is that not the case for literally everything? "You can send me data of format X, all I ask is that you be able to produce format X" ? I'm assuming that I'm missing something fwiw, not trying to diminish the value.
    - deleted 6y ago
      [deleted]
    - humbleMouse 6y ago
      The difference is that arrow’s mapping behind the scenes enables automatic translation to any implemented “plugin” that is on the user’s implementation of arrow. You can extend arrows format to make it automatically map to whatever you want, basically. And it’s all stored in memory - so much faster access to complex data relationships than anything that exists to my knowledge.
      - staticautomatic 6y ago
        Could I write a plug-in that mapped to Cypher? I’ve got a graph use case in mind where I want to use RedisGraph but don’t feel comfy with Redis as a primary DB and would totally consider a columnar store as a primary if I didn’t have to serialize.
shafiemukhre 6y ago
True. Arrow is awesome and Dremio is using it as well as their built in memory. I tried it and it is increadibily fast. The future of data ecosystem is gonna be amazing
breck 6y ago
Arrow is definitely one of the top 10 new things I'm most excited about in the data science space, but not sure I'd call it the most important thing. ;) It is pretty awesome, however, particularly for folks like me that are often hopping between Python/R/Javascript. I've definitely got in on the roadmap for all my data science libraries. Btw, Arquero from that UW lab looks really neat as well, and is supporting Arrow out of the gate (https://github.com/uwdata/arquero https://github.com/uwdata/arquero).
- georgewfraser 6y ago
  Much of the value of Arrow is in the things that will get built after Arrow is widely supported by data warehouses. Much of the data ecosystem we have today was designed to avoid the cost of moving data between systems. The whole Hadoop ecosystem is written in Java and shoehorned into map-reduce for this reason. Imagine if, for example, you could use Mathematica or R to analyze data in your Snowflake cluster, with no bottleneck reading data from the warehouse even for giant datasets. This is the future that’s going to be enabled by Arrow.
  - RainBoooow 6y ago
    Is it not what Presto (now Trino) is solving as well (among other things) ? Even though it is focused only on analytics and not on ML use cases.
deleted 6y ago
[deleted]
yeshengm 6y ago
Are these execution engines internally using Arrow columnar format or are they just exposing Arrow as a client wire format? AFAIK Spark and Presto does not use Arrow as execution columnar format, but just data sources/sinks.
- groceryheist 6y ago
  You can configure Spark to use arrow for passing data between Java and Python via spark.sql.execution.arrow.pyspark.enabled but yes, Spark uses Java datatypes internally.
otabdeveloper4 6y ago
It's nice that the cyclical technology pendulum is finally swinging back from XML/JSON to files-with-C-structs again, but any serious analytics data store (e.g. Clickhouse) uses its own aggressively optimized storage format, so the process of loading from Arrow files to database won't go away.
davidkell 6y ago
Any Snowflake developers reading this - the current snowflake-connector-python is pinned to 0.17, almost 1 year out of date now. Would be great to get that bumped to a more recent version :-)

tristanz 6y ago

Agreed! Thank you Arrow community for such a great project. It's a long road but it opens up tremendous potential for efficient data systems that talk to one another. The future looks bright with so many vendors backing Arrow independently and Wes McKinney founding Ursa Labs and now Ursa Computing. https://ursalabs.org/blog/ursa-computing/ https://ursalabs.org/blog/ursa-computing/

wesm 6y ago

Microsoft is also on top of this with their Magpie project http://cidrdb.org/cidr2021/papers/cidr2021_paper08.pdf http://cidrdb.org/cidr2021/papers/cidr2021_paper08.pdf "A common, efficient serialized and wire format across data engines is a transformational development. Many previous systems and approaches (e.g., [26, 36, 38, 51]) have observed the prohibitive cost of data conversion and transfer, precluding optimizers from exploiting inter-DBMS performance advantages. By contrast, inmemory data transfer cost between a pair of Arrow-supporting systems is effectively zero. Many major, modern DBMSs (e.g., Spark, Kudu, AWS Data Wrangler, SciDB, TileDB) and data-processing frameworks (e.g., Pandas, NumPy, Dask) have or are in the process of incorporating support for Arrow and ArrowFlight. Exploiting this is key for Magpie, which is thereby free to combine data from different sources and cache intermediate data and results, without needing to consider data conversion overhead."

data_ders 6y ago
way cool! Is magpie end-user facing anywhere yet? We were using the azureml-dataprep library for a while which seems similar but not all of magpie
polskibus 6y ago
I wish MS put in some resources behind Arrow in .NET. I tried raising some remarks about it on dotnet repos (esp. within ML.NET), but to no avail. Hopefully it would change now that Arrow is more popular, and also written about by MS itself.

nomel 6y ago

Until arrow has proper support for multidimensional arrays [1], its not really appropriate for many use cases. Not having first class support for multidimensional arrays, in a modern framework, really surprised me and my sensor data. [1] https://lists.apache.org/x/thread.html/9b142c1709aa37dc35f1ce8db4e1ced94fcc4cdd96cc72b5772b373b@%3Cdev.arrow.apache.org%3E https://lists.apache.org/x/thread.html/9b142c1709aa37dc35f1c...

lmeyerov 6y ago
FWIW, you can support tensors etc. natively today by using Arrow's structs, such as `column i_am_packed: {x: int32, y: int32, z: int32, time: datetime64[ns], temp: int32}`. This will be a dense, packed representation for whatever fixed type the column has, including dynamically generated. It ends up being quite nice to have that passed around, esp. vs say protobufs. You wouldn't be sending mixed-length values, and ~all metrics systems I've worked with would be fine there. The nullables end up a win for those. However, you'd need to do a manual (likely zero-copy) casts between the struct-type to whatever tensor-type you're using. Massively popular ml systems like huggingface do this fine afaict for their Arrow-based tensor work. Likewise, as we do a lot of GPU stuff, what's additionally common is compacting the in-memory stuff as big memory blocks ('long recordbatches') instead of CPU-land's typically more fragmented ones, and that ends up making casts even easier. Annoying to have to add an explicit cast for some interop cases, but preserves end-to-end type safety & hasn't been a deal breaker for us. Having to write the cast being annoying/difficult, esp. for whoever does it first. More frustrating for us has been sparse data and compression controls, but most formats are even worse here..
- deleted 6y ago
  [deleted]
wesm 6y ago
Almost no database systems support multidimensional arrays. So they are not appropriate for many use cases? * BigQuery: no * Redshift: no * Spark SQL: no * Snowflake: no * Clickhouse: no * Dremio: no * Impala: no * Presto: no ... list continues We've invited developers to add the extension types for tensor data, but no one has contributed them yet. I'm not seeing a lot of tabular data with embedded tensors out in the wild.

waynesonfire 6y ago

Uhh.. maybe. It's a serde that's trying to be cross-language / platform. I guess it also offers some APIs to process the data so you can minimize serde operations. But, I dunno. It's been hard to understand the benefit of the libabry and the posts here don't help.

TuringTest 6y ago
If it works as a universal intermediate exchange language, it could help standardize connections among disparate systems. When you have N systems, it takes N^2 translators to build direct connections to transfer data between them; but it only takes N translators if all them can talk the same exchange language.
- waynesonfire 6y ago
  can you define what at translator is? I don't understand the complexity you're constructing. I have N systems and they talk protobuf. What's the problem?
  - TuringTest 6y ago
    By a translator, I mean a library that allows accessing data from different subsystems (either languages or OS processes). In this case, the advantages are that 1) Arrow is language agnostic, so it's likely that it can be used as a native library in your program and 2) it doesn't copy data to make it accessible to another process, so it saves a lot of marshalling / unmarshalling steps (assuming both sides use data in tabular format, which is typical of data analysis contexts).
  - deleted 6y ago
    [deleted]
mumblemumble 6y ago
It's not just a serde. One of its key use cases is eliminating serde.

rubicon33 6y ago

Wait so you're telling me I could store data as a PDF file, and access it easily / quickly as SQL?

sethhochberg 6y ago
If you found/wrote a adapter to translate your structured PDF into Arrow's format, yes - the idea is that you can wire up anything that can produce Arrow data to anything that can consume Arrow data.
- staticassertion 6y ago
  I'm kinda confused. Is that not the case for literally everything? "You can send me data of format X, all I ask is that you be able to produce format X" ? I'm assuming that I'm missing something fwiw, not trying to diminish the value.
  - deleted 6y ago
    [deleted]
  - humbleMouse 6y ago
    The difference is that arrow’s mapping behind the scenes enables automatic translation to any implemented “plugin” that is on the user’s implementation of arrow. You can extend arrows format to make it automatically map to whatever you want, basically. And it’s all stored in memory - so much faster access to complex data relationships than anything that exists to my knowledge.
    - staticautomatic 6y ago
      Could I write a plug-in that mapped to Cypher? I’ve got a graph use case in mind where I want to use RedisGraph but don’t feel comfy with Redis as a primary DB and would totally consider a columnar store as a primary if I didn’t have to serialize.

shafiemukhre 6y ago

True. Arrow is awesome and Dremio is using it as well as their built in memory. I tried it and it is increadibily fast. The future of data ecosystem is gonna be amazing

breck 6y ago

Arrow is definitely one of the top 10 new things I'm most excited about in the data science space, but not sure I'd call it the most important thing. ;) It is pretty awesome, however, particularly for folks like me that are often hopping between Python/R/Javascript. I've definitely got in on the roadmap for all my data science libraries. Btw, Arquero from that UW lab looks really neat as well, and is supporting Arrow out of the gate (https://github.com/uwdata/arquero https://github.com/uwdata/arquero).

georgewfraser 6y ago
Much of the value of Arrow is in the things that will get built after Arrow is widely supported by data warehouses. Much of the data ecosystem we have today was designed to avoid the cost of moving data between systems. The whole Hadoop ecosystem is written in Java and shoehorned into map-reduce for this reason. Imagine if, for example, you could use Mathematica or R to analyze data in your Snowflake cluster, with no bottleneck reading data from the warehouse even for giant datasets. This is the future that’s going to be enabled by Arrow.
- RainBoooow 6y ago
  Is it not what Presto (now Trino) is solving as well (among other things) ? Even though it is focused only on analytics and not on ML use cases.

deleted 6y ago

[deleted]

yeshengm 6y ago

Are these execution engines internally using Arrow columnar format or are they just exposing Arrow as a client wire format? AFAIK Spark and Presto does not use Arrow as execution columnar format, but just data sources/sinks.

groceryheist 6y ago
You can configure Spark to use arrow for passing data between Java and Python via spark.sql.execution.arrow.pyspark.enabled but yes, Spark uses Java datatypes internally.

otabdeveloper4 6y ago

It's nice that the cyclical technology pendulum is finally swinging back from XML/JSON to files-with-C-structs again, but any serious analytics data store (e.g. Clickhouse) uses its own aggressively optimized storage format, so the process of loading from Arrow files to database won't go away.

davidkell 6y ago

Any Snowflake developers reading this - the current snowflake-connector-python is pinned to 0.17, almost 1 year out of date now. Would be great to get that bumped to a more recent version :-)