14 ms·

Can someone ELI5 what problems are best solved by apache arrow?

Diederich 6y ago
Rather curious myself. https://en.wikipedia.org/wiki/Apache_Arrow https://en.wikipedia.org/wiki/Apache_Arrow was interesting, but I think many of us would benefit from a broader, problem focused description of Arrow from someone in the know.
adgjlsfhk1 6y ago
The big thing is that it is one of the first standardized, cross language binary data formats. CSV is an OK text format, but parsing it is really slow because of string escaping. The files it produces are also pretty big since it's text. Arrow is really fast to parse (up to 1000x faster than CSV), supports data compression, enough data-types to be useful, and deals with metadata well. The closest competitor is probably protobuf, but protobuf is a total pain to parse.
- makapuf 6y ago
  Seems nice. How does it compare to hdf5?
  - BadInformatics 6y ago
    HDF5 is pretty terrible as a wire format, so it's not a 1-1 comparison to Arrow. Generally people are not going to be saving Arrow data to disk either (though you can with the IPC format), but serializing to a more compact representation like Parquet.
    - gbrits 6y ago
      As I understand, arrow is particularly interesting since it’s wire format can be immediately queried/operated on without deserialization. Would saving an Arrow-structure as parquet not defeat that purpose, since your would need the costly deserialization step again on read? Honest question
      - BadInformatics 6y ago
        The FAQ [1] and this SO answer [2] explain it better than I can, but basically yes. However, the (de)serialization overhead is probably better than most alternative formats you could save to. [1] https://arrow.apache.org/faq/ https://arrow.apache.org/faq/ [2] https://stackoverflow.com/questions/56472727/difference-between-apache-parquet-and-arrow/56481636#56481636 https://stackoverflow.com/questions/56472727/difference-betw...
- MrPowers 6y ago
  CSV is a file format and Arrow is an in memory data format. The CSV vs Parquet comparison makes more sense. Conflating Arrow / Parquet is a pet peeve of Wes: https://news.ycombinator.com/item?id=23970586 https://news.ycombinator.com/item?id=23970586
  - nolta 6y ago
    To be fair, arrow absorbed parquet-cpp, so a little confusion is to be expected.
- aldanor 6y ago
  The closest competitor would be HDF5, not Protobuf.
- imtringued 6y ago
  The worst part about CSV is that it appears to be so simple that many people don't even try to understand how complicated it is and just roll their own broken dialect instead of just using a library. There is also no standardized encoding. Delimiters have to be negotiated out of band and people love getting them wrong.
  - gregw2 6y ago
    Mostly agree, but RFC-4180 does specify delimiters. I've used pipe-delimited files for years to ensure explicitly-defined conformity, but recently due to the plethora of libraries have been thinking that CSV may be simpler.
neovintage 6y ago
The premise around arrow is that when you want share data with another system, or even on the same machine between processes, most of the compute time spent is in serializing and deserializing data. Arrow removes that step by defining a common columnar format that can be used in many different programming languages. Theres more to arrow than just the file format that makes working with data even easier like better over the wire transfers (arrow flight). How this would manifest for your customers using your applications? They'd like see speeds increase. Arrow makes a lot of sense when working with lots of data in analytical or data science use cases.
- MrPowers 6y ago
  Exactly. Some specific examples. Read a Parquet file into a Pandas DataFrame. Then read the Pandas DataFrame into a Spark DataFrame. Spark & Pandas are using the same Arrow memory format, so no serde is needed. See the "Standardization Saves" diagram here: https://arrow.apache.org/overview/ https://arrow.apache.org/overview/
  - pindab0ter 6y ago
    I hope you don’t mind me asking dumb questions, but how does this differ from the role that say Protocol Buffers fills? To my ears they both facilitate data exchange. Are they comparable in that sense?
    - rcoveson 6y ago
      Better to compare it to Cap'n Proto instead. Arrow data is already laid out in a usable way. For example, an Arrow column of int64s is an 8-byte aligned memory region of size 8*N bytes (plus a bit vector for nullity), ready for random access or vectorized operations. Protobuf, on the other hand, would encode those values as variable-width integers. This saves a lot of space, which might be better for transfer over a network, but means that writers have to take a usable in-memory array and serialize it, and readers have to do the reverse on their end. Think of Arrow as standardized shared memory using struct-of-arrays layout, Cap'n Proto as standardized shared memory using array-of-structs layout, and Protobuf as a lightweight purpose-built compression algorithm for structs.
CrazyPyroLinux 6y ago
This is not the greatest answer, but one anecdote: I am looking at it as an alternative to MS SSIS for moving batches of data around between databases.
RobinL 6y ago
I wrote a bit about this from the perspective of a data scientist here: https://www.robinlinacre.com/demystifying_arrow/ https://www.robinlinacre.com/demystifying_arrow/ I cover some of the use cases, but more importantly try and explain how it all fits together, justifying why - as another commenters has said - it's the most important thing happening in the data ecosystem right now. I wrote it because i'd heard a lot about Arrow, and even used it quite a lot, but realised I hadn't really understood what it was!
prepend 6y ago
I recently found it useful for the dumbest reason. A dataset was about 3GB as a CSV and 20MB as a parquet file created and consumed by arrow. The file also worked flawlessly across different environments and languages. So it’s a good transport tool. It also happens to be fast to load and query, but I only used it because of the compact way it stores data without any hoops to jump through. Of course one might say that it’s stupid to try to pass around a 2GB or 20MB file, but in my case I needed to do that.
- kristianp 6y ago
  Did you have to reformat the data to get that size saving?
  - prepend 6y ago
    Not at all. It was just swapping out readcsv with readparquet in R and Python. It was painless. Granted my dataset was mostly categorical so that’s why it compressed down so much, but it was a real lifesaver. It was also nice to be able to read while bundles of parquet a into a single dataframe easily. So is nice for “sharding” really big parquets over multiple files. Or fitting under file size limits on git repos.
    - alisonkisk 6y ago
      gzip probably would have been fine for that case even without parquet?
      - prepend 6y ago
        Gzip did help, but not as much as parquet. I don’t remember the exact sizes but I think parquet was was 3GB->20MB and gzip was like 3GB->180MB or something. Also gzip was an extra step of unzip then read.
- CameronNemo 6y ago
  Parquet is not Arrow. Parquet has optimizations for storage size at the expense of compute readiness. Arrow maximizes computation efficiency, at the expense of storage size. Read more here: https://stackoverflow.com/questions/56472727/difference-between-apache-parquet-and-arrow https://stackoverflow.com/questions/56472727/difference-betw...
IshKebab 6y ago
When you want to process large amounts of in-memory tabular data from different languages. You can save it to disk too using Apache Parquet but I evaluated Parquet and it is very immature. Extremely incomplete documentation and lots of Arrow features are just not supported in Parquet unfortunately.
- Lucasoato 6y ago
  Do you mean the Parquet format? I don't think Parquet is immature, it is used in so many enterprise environments, it's is one of the few columnar file format for batch analysis and processing. It preforms so well... But I'm curious to know your opinion on this, so feel free to add some context to your position!
  - IshKebab 6y ago
    Yeah I do. For example Apache Arrow supports in memory compression. But Parquet does not support that. I had to look through the code to find that out, and I found many instances of basically `throw "Not supported"`. And yeah as I said the documentation is just non-existent. If you are already using Arrow, or you absolutely must use a columnar file format then it's probably a good option.
    - BadInformatics 6y ago
      Is that a problem in the Parquet format or in PyArrow? My understanding is that Parquet is primarily meant for on-disk storage (hence the default on-disk compression), so you'd read into Arrow for in-memory compression or IPC.
      - IshKebab 6y ago
        I don't know whether it is a limitation of the format or of the implementation. I used the C++ library (or maybe it was C, I can't remember), which I assume PyArrow uses too. > hence the default on-disk compression No, Parquet doesn't support some compression formats that Arrow does.
hbcondo714 6y ago
The press release on the first version of Apache Arrow discussed here 5 years ago is actually a good introduction IMO: https://news.ycombinator.com/item?id=11118274 https://news.ycombinator.com/item?id=11118274
foobarian 6y ago
To really grok why this is useful, put yourself in the shoes of a data warehouse user or administrator. This is a farm of machines with disks that hold data too big to have in a typical RDBMS. Because the data is so big and stored spread out across machines, a whole parallel universe of data processing tools and practices exists that run various computations over shards of the data locally on each server, and merge the results etc. (Map-reduce originally from Google was the first famous example). Then there is a cottage industry of wrappers around this kind of system to let you use SQL to query the data, make it faster, let you build cron jobs and pipelines with dependencies, etc. Now, so far all these tools did not really have a common interchange format for data, so there was a lot of wheel reinvention and incompatibility. Got file system layer X on your 10PB cluster? Can't use it with SQL engine Y. And I guess this is where Arrow comes in, where if everyone uses it then interop will get a lot better and each individual tool that much more useful. Just my naive take.
deleted 6y ago
[deleted]
juntan 6y ago
In Perspective (https://github.com/finos/perspective https://github.com/finos/perspective), we use Apache Arrow as a fast, cross-language/cross-network data encoding that is extremely useful for in-browser data visualization and analytics. Some benefits: - super fast read/write compared to CSV & JSON (Perspective and Arrow share an extremely similar column encoding scheme, so we can memcpy Arrow columns into Perspective wholesale instead of reading a dataset iteratively). - the ability to send Arrow binaries as an ArrayBuffer between a Python server and a WASM client, which guarantees compatibility and removes the overhead of JSON serialization/deserialization. - because Arrow columns are strictly typed, there's no need to infer data types - this helps with speed and correctness. - Compared to JSON/CSV, Arrow binaries have a super compact encoding that reduces network transport time. For us, building on top of Apache Arrow (and using it wherever we can) reduces the friction of passing around data between clients, servers, and runtimes in different languages, and allows larger datasets to be efficiently visualized and analyzed in the browser context.
tomrod 6y ago
Python Pandas can output to lots of things. R can read lots of things. Arrow lets you push (mostly*) seamlessly. (I hit one edge case with an older version with JSON as a datatype). Not having to maintain data type definitions when sending around the data, nor caring whether my colleagues were using R or Python, worked great.

Rather curious myself. https://en.wikipedia.org/wiki/Apache_Arrow https://en.wikipedia.org/wiki/Apache_Arrow was interesting, but I think many of us would benefit from a broader, problem focused description of Arrow from someone in the know.

adgjlsfhk1 6y ago

The big thing is that it is one of the first standardized, cross language binary data formats. CSV is an OK text format, but parsing it is really slow because of string escaping. The files it produces are also pretty big since it's text. Arrow is really fast to parse (up to 1000x faster than CSV), supports data compression, enough data-types to be useful, and deals with metadata well. The closest competitor is probably protobuf, but protobuf is a total pain to parse.

makapuf 6y ago
Seems nice. How does it compare to hdf5?
- BadInformatics 6y ago
  HDF5 is pretty terrible as a wire format, so it's not a 1-1 comparison to Arrow. Generally people are not going to be saving Arrow data to disk either (though you can with the IPC format), but serializing to a more compact representation like Parquet.
  - gbrits 6y ago
    As I understand, arrow is particularly interesting since it’s wire format can be immediately queried/operated on without deserialization. Would saving an Arrow-structure as parquet not defeat that purpose, since your would need the costly deserialization step again on read? Honest question
    - BadInformatics 6y ago
      The FAQ [1] and this SO answer [2] explain it better than I can, but basically yes. However, the (de)serialization overhead is probably better than most alternative formats you could save to. [1] https://arrow.apache.org/faq/ https://arrow.apache.org/faq/ [2] https://stackoverflow.com/questions/56472727/difference-between-apache-parquet-and-arrow/56481636#56481636 https://stackoverflow.com/questions/56472727/difference-betw...
MrPowers 6y ago
CSV is a file format and Arrow is an in memory data format. The CSV vs Parquet comparison makes more sense. Conflating Arrow / Parquet is a pet peeve of Wes: https://news.ycombinator.com/item?id=23970586 https://news.ycombinator.com/item?id=23970586
- nolta 6y ago
  To be fair, arrow absorbed parquet-cpp, so a little confusion is to be expected.
aldanor 6y ago
The closest competitor would be HDF5, not Protobuf.
imtringued 6y ago
The worst part about CSV is that it appears to be so simple that many people don't even try to understand how complicated it is and just roll their own broken dialect instead of just using a library. There is also no standardized encoding. Delimiters have to be negotiated out of band and people love getting them wrong.
- gregw2 6y ago
  Mostly agree, but RFC-4180 does specify delimiters. I've used pipe-delimited files for years to ensure explicitly-defined conformity, but recently due to the plethora of libraries have been thinking that CSV may be simpler.

neovintage 6y ago

The premise around arrow is that when you want share data with another system, or even on the same machine between processes, most of the compute time spent is in serializing and deserializing data. Arrow removes that step by defining a common columnar format that can be used in many different programming languages. Theres more to arrow than just the file format that makes working with data even easier like better over the wire transfers (arrow flight). How this would manifest for your customers using your applications? They'd like see speeds increase. Arrow makes a lot of sense when working with lots of data in analytical or data science use cases.

MrPowers 6y ago
Exactly. Some specific examples. Read a Parquet file into a Pandas DataFrame. Then read the Pandas DataFrame into a Spark DataFrame. Spark & Pandas are using the same Arrow memory format, so no serde is needed. See the "Standardization Saves" diagram here: https://arrow.apache.org/overview/ https://arrow.apache.org/overview/
- pindab0ter 6y ago
  I hope you don’t mind me asking dumb questions, but how does this differ from the role that say Protocol Buffers fills? To my ears they both facilitate data exchange. Are they comparable in that sense?
  - rcoveson 6y ago
    Better to compare it to Cap'n Proto instead. Arrow data is already laid out in a usable way. For example, an Arrow column of int64s is an 8-byte aligned memory region of size 8*N bytes (plus a bit vector for nullity), ready for random access or vectorized operations. Protobuf, on the other hand, would encode those values as variable-width integers. This saves a lot of space, which might be better for transfer over a network, but means that writers have to take a usable in-memory array and serialize it, and readers have to do the reverse on their end. Think of Arrow as standardized shared memory using struct-of-arrays layout, Cap'n Proto as standardized shared memory using array-of-structs layout, and Protobuf as a lightweight purpose-built compression algorithm for structs.

CrazyPyroLinux 6y ago

This is not the greatest answer, but one anecdote: I am looking at it as an alternative to MS SSIS for moving batches of data around between databases.

RobinL 6y ago

I wrote a bit about this from the perspective of a data scientist here: https://www.robinlinacre.com/demystifying_arrow/ https://www.robinlinacre.com/demystifying_arrow/ I cover some of the use cases, but more importantly try and explain how it all fits together, justifying why - as another commenters has said - it's the most important thing happening in the data ecosystem right now. I wrote it because i'd heard a lot about Arrow, and even used it quite a lot, but realised I hadn't really understood what it was!

prepend 6y ago

I recently found it useful for the dumbest reason. A dataset was about 3GB as a CSV and 20MB as a parquet file created and consumed by arrow. The file also worked flawlessly across different environments and languages. So it’s a good transport tool. It also happens to be fast to load and query, but I only used it because of the compact way it stores data without any hoops to jump through. Of course one might say that it’s stupid to try to pass around a 2GB or 20MB file, but in my case I needed to do that.

kristianp 6y ago
Did you have to reformat the data to get that size saving?
- prepend 6y ago
  Not at all. It was just swapping out readcsv with readparquet in R and Python. It was painless. Granted my dataset was mostly categorical so that’s why it compressed down so much, but it was a real lifesaver. It was also nice to be able to read while bundles of parquet a into a single dataframe easily. So is nice for “sharding” really big parquets over multiple files. Or fitting under file size limits on git repos.
  - alisonkisk 6y ago
    gzip probably would have been fine for that case even without parquet?
    - prepend 6y ago
      Gzip did help, but not as much as parquet. I don’t remember the exact sizes but I think parquet was was 3GB->20MB and gzip was like 3GB->180MB or something. Also gzip was an extra step of unzip then read.
CameronNemo 6y ago
Parquet is not Arrow. Parquet has optimizations for storage size at the expense of compute readiness. Arrow maximizes computation efficiency, at the expense of storage size. Read more here: https://stackoverflow.com/questions/56472727/difference-between-apache-parquet-and-arrow https://stackoverflow.com/questions/56472727/difference-betw...

IshKebab 6y ago

When you want to process large amounts of in-memory tabular data from different languages. You can save it to disk too using Apache Parquet but I evaluated Parquet and it is very immature. Extremely incomplete documentation and lots of Arrow features are just not supported in Parquet unfortunately.

Lucasoato 6y ago
Do you mean the Parquet format? I don't think Parquet is immature, it is used in so many enterprise environments, it's is one of the few columnar file format for batch analysis and processing. It preforms so well... But I'm curious to know your opinion on this, so feel free to add some context to your position!
- IshKebab 6y ago
  Yeah I do. For example Apache Arrow supports in memory compression. But Parquet does not support that. I had to look through the code to find that out, and I found many instances of basically `throw "Not supported"`. And yeah as I said the documentation is just non-existent. If you are already using Arrow, or you absolutely must use a columnar file format then it's probably a good option.
  - BadInformatics 6y ago
    Is that a problem in the Parquet format or in PyArrow? My understanding is that Parquet is primarily meant for on-disk storage (hence the default on-disk compression), so you'd read into Arrow for in-memory compression or IPC.
    - IshKebab 6y ago
      I don't know whether it is a limitation of the format or of the implementation. I used the C++ library (or maybe it was C, I can't remember), which I assume PyArrow uses too. > hence the default on-disk compression No, Parquet doesn't support some compression formats that Arrow does.

hbcondo714 6y ago

The press release on the first version of Apache Arrow discussed here 5 years ago is actually a good introduction IMO: https://news.ycombinator.com/item?id=11118274 https://news.ycombinator.com/item?id=11118274

foobarian 6y ago

To really grok why this is useful, put yourself in the shoes of a data warehouse user or administrator. This is a farm of machines with disks that hold data too big to have in a typical RDBMS. Because the data is so big and stored spread out across machines, a whole parallel universe of data processing tools and practices exists that run various computations over shards of the data locally on each server, and merge the results etc. (Map-reduce originally from Google was the first famous example). Then there is a cottage industry of wrappers around this kind of system to let you use SQL to query the data, make it faster, let you build cron jobs and pipelines with dependencies, etc. Now, so far all these tools did not really have a common interchange format for data, so there was a lot of wheel reinvention and incompatibility. Got file system layer X on your 10PB cluster? Can't use it with SQL engine Y. And I guess this is where Arrow comes in, where if everyone uses it then interop will get a lot better and each individual tool that much more useful. Just my naive take.

deleted 6y ago

[deleted]

juntan 6y ago

In Perspective (https://github.com/finos/perspective https://github.com/finos/perspective), we use Apache Arrow as a fast, cross-language/cross-network data encoding that is extremely useful for in-browser data visualization and analytics. Some benefits: - super fast read/write compared to CSV & JSON (Perspective and Arrow share an extremely similar column encoding scheme, so we can memcpy Arrow columns into Perspective wholesale instead of reading a dataset iteratively). - the ability to send Arrow binaries as an ArrayBuffer between a Python server and a WASM client, which guarantees compatibility and removes the overhead of JSON serialization/deserialization. - because Arrow columns are strictly typed, there's no need to infer data types - this helps with speed and correctness. - Compared to JSON/CSV, Arrow binaries have a super compact encoding that reduces network transport time. For us, building on top of Apache Arrow (and using it wherever we can) reduces the friction of passing around data between clients, servers, and runtimes in different languages, and allows larger datasets to be efficiently visualized and analyzed in the browser context.

tomrod 6y ago

Python Pandas can output to lots of things. R can read lots of things. Arrow lets you push (mostly*) seamlessly. (I hit one edge case with an older version with JSON as a datatype). Not having to maintain data type definitions when sending around the data, nor caring whether my colleagues were using R or Python, worked great.