4 ms·

So, maybe I'm stupid and never faced enough big data, but what's the advantage of this versus a custom script that queries A and inserts into B? I do that kind

So, maybe I'm stupid and never faced enough big data, but what's the advantage of this versus a custom script that queries A and inserts into B? I do that kind of stuff all the time with Go and it's pretty fast with 20~40 million records, averaging 100 KB each. Are those tools oriented to billions, instead of millions? What are the benefits?

th0ma5 6y ago
I see Arrow as a "data frame server" and data frames have a higher level of abstraction for use in analysis. But you're right that raw processing can be done with most languages natively much faster. But Arrow can let a Python statistician and an R statistician compare apples to apples easily.
cocobro 6y ago
This article might help, it explains Arrow's performance benefits: https://www.dremio.com/apache-arrow-explained/ https://www.dremio.com/apache-arrow-explained/
yokaze 6y ago
One major use-case is exchanging data between processes (in different languages) using IPC, avoiding the overhead of serialisation. Best example is probably pyspark.
- lenkite 6y ago
  Can't one do this using protobuf, flatbuffers, etc ? What advantage does arrow offer here ?
  - wenc 6y ago
    That's precisely it -- you don't have to use protobuf etc. because you don't have to serialize/deserialize at all. Languages that use Arrow can interact with the source data directly in-memory, in-process. No need to move any data around. The fastest serde operation is one that you don't have to do at all.
    - lenkite 6y ago
      This is already possible with flatbuffers.
      - wenc 6y ago
        From the Arrow FAQ How does Arrow relate to Flatbuffers? Flatbuffers is a low-level building block for binary data serialization. It is not adapted to the representation of large, structured, homogenous data, and does not sit at the right abstraction layer for data analysis tasks. Arrow is a data layer aimed directly at the needs of data analysis, providing a comprehensive collection of data types required to analytics, built-in support for “null” values (representing missing data), and an expanding toolbox of I/O and computing facilities. The Arrow file format does use Flatbuffers under the hood to serialize schemas and other metadata needed to implement the Arrow binary IPC protocol, but the Arrow data format uses its own representation for optimal access and computation.
  - FridgeSeal 6y ago
    Zero copy serialisation and performance benefits over doing “row-at-a-time” like you would with protobuf.
    - lenkite 6y ago
      Zero copy serialisation is offered by flatbuffers - and one additionally has the advantage of IDL as a contract between different technology platforms.
wesm 6y ago
Arrow: * Standardizes binary interop and "serialization" of large structured data, removing all conversions / serialization at ingest and export boundaries. This alone can mean > 2-100x performance improvement in an application that processes a lot of data * The Arrow in-memory format is an ideal data structure to code analytical algorithms against. Check out my 18min talk from a few years ago about the vision for the project https://www.youtube.com/watch?v=wdmf1msbtVs https://www.youtube.com/watch?v=wdmf1msbtVs
- boomskats 6y ago
  This is a great talk and should be a top-level comment explaining this release.
dragonsh 6y ago
Check out the post from author of pandas on pandas, pandas2 and arrow [1]. This might answer your question on what is the significance of arrow, given today pandas are kind of basic ingredient in scientific computing, AI and ML. [1] https://wesmckinney.com/blog/apache-arrow-pandas-internals/ https://wesmckinney.com/blog/apache-arrow-pandas-internals/
oxfordmale 6y ago
At my company, we query from A and store the data in PyArrow Parquet files on S3. Using Athena this data can then be directly queried. This is a cheap way to store data, that doesn't need the performance of database server.

th0ma5 6y ago

I see Arrow as a "data frame server" and data frames have a higher level of abstraction for use in analysis. But you're right that raw processing can be done with most languages natively much faster. But Arrow can let a Python statistician and an R statistician compare apples to apples easily.

cocobro 6y ago

This article might help, it explains Arrow's performance benefits: https://www.dremio.com/apache-arrow-explained/ https://www.dremio.com/apache-arrow-explained/

yokaze 6y ago

One major use-case is exchanging data between processes (in different languages) using IPC, avoiding the overhead of serialisation. Best example is probably pyspark.

lenkite 6y ago
Can't one do this using protobuf, flatbuffers, etc ? What advantage does arrow offer here ?
- wenc 6y ago
  That's precisely it -- you don't have to use protobuf etc. because you don't have to serialize/deserialize at all. Languages that use Arrow can interact with the source data directly in-memory, in-process. No need to move any data around. The fastest serde operation is one that you don't have to do at all.
  - lenkite 6y ago
    This is already possible with flatbuffers.
    - wenc 6y ago
      From the Arrow FAQ How does Arrow relate to Flatbuffers? Flatbuffers is a low-level building block for binary data serialization. It is not adapted to the representation of large, structured, homogenous data, and does not sit at the right abstraction layer for data analysis tasks. Arrow is a data layer aimed directly at the needs of data analysis, providing a comprehensive collection of data types required to analytics, built-in support for “null” values (representing missing data), and an expanding toolbox of I/O and computing facilities. The Arrow file format does use Flatbuffers under the hood to serialize schemas and other metadata needed to implement the Arrow binary IPC protocol, but the Arrow data format uses its own representation for optimal access and computation.
- FridgeSeal 6y ago
  Zero copy serialisation and performance benefits over doing “row-at-a-time” like you would with protobuf.
  - lenkite 6y ago
    Zero copy serialisation is offered by flatbuffers - and one additionally has the advantage of IDL as a contract between different technology platforms.

wesm 6y ago

Arrow: * Standardizes binary interop and "serialization" of large structured data, removing all conversions / serialization at ingest and export boundaries. This alone can mean > 2-100x performance improvement in an application that processes a lot of data * The Arrow in-memory format is an ideal data structure to code analytical algorithms against. Check out my 18min talk from a few years ago about the vision for the project https://www.youtube.com/watch?v=wdmf1msbtVs https://www.youtube.com/watch?v=wdmf1msbtVs

boomskats 6y ago
This is a great talk and should be a top-level comment explaining this release.

dragonsh 6y ago

Check out the post from author of pandas on pandas, pandas2 and arrow [1]. This might answer your question on what is the significance of arrow, given today pandas are kind of basic ingredient in scientific computing, AI and ML. [1] https://wesmckinney.com/blog/apache-arrow-pandas-internals/ https://wesmckinney.com/blog/apache-arrow-pandas-internals/

oxfordmale 6y ago

At my company, we query from A and store the data in PyArrow Parquet files on S3. Using Athena this data can then be directly queried. This is a cheap way to store data, that doesn't need the performance of database server.