5 ms·

Yay! Apache Arrow is great project. Whenever I use is, people think I’m some kind of genius. It should get a lot more press.

zinclozenge 6y ago
Can you talk about how you use it? At a cursory glance, it seems like it's being used to great effect in databases, or systems designed to query data. Is that what you're using for?
- Q6T46nT668w6i3m 6y ago
  An alternative to CSV or SQLite for interchange.
  - woadwarrior01 6y ago
    From my experience, Parquet files are a better alternative to CSV or SQLite files for data at rest. I've observed that parquet files tend to be smaller than Feather files (which is essentially the Arrow format in a file) for the same data. Arrow is great for in-memory data, but even with compression, the on-disk size of Feather files tend to be quite a bit bigger than Parquet files.
    - ttymck 6y ago
      "for interchange" in data science might well mean data on-the-move rather than data at-rest: i.e. a pipeline in two stages, step A in python and step B in R. The data is exchanged (relatively) quickly via disk, and speed beats compression.
      - woadwarrior01 6y ago
        Even with fast SSDs, memcpy() is roughly an order of magnitude faster than writing to disk. Using a fast compression algorithm like lz4, zstd, etc can help reduce that gap. And this is invariant of whether the serialization format is Arrow or Parquet. It’s not either, or. If you’re writing to disk, with compression you can have both speed and reduction in the disk footprint.
michaelcampbell 6y ago
The docs are dire. Lots and lots about internals and how it works, and after a few minutes of searching, I've yet to see what it actually does or what I'd want to use it for. The "use cases" are even overly technical things that are building blocks for ... something. What is that something?
- baq 6y ago
  judging from the comments here, it's an in-memory (in-process?) columnar database format without a DBMS attached. sounds useful when sqlite is too slow for your aggregations or if you do processing in X (C, Java) and visualization in Y (R, Python) - just guessing, haven't used it.
- asdffdsa 6y ago
  Judging from the faq (https://arrow.apache.org/faq/ https://arrow.apache.org/faq/), it seems like it's a storage format for large structured datasets, designed specifically for use in-memory and for serialization across the network. It provides a spec for the data format, as well as implementations for several programming languages. That seems pretty straightforward, what part was confusing for you?
  - michaelcampbell 6y ago
    > what part was confusing for you? The passive aggressive isn't necessary.
    - asdffdsa 6y ago
      His snark isn't necessary; the docs are fine.
- kthejoker2 6y ago
  If you are already doing Big Data / ML / deep learning work in Spark or Dask, Arrow is (with caveats) more or less a drop-in performance optimization (because it's optimized for in-memory vectors) and complexity reducer (because it does all the interop between JVM, Python, TF, etc) So by itself it doesn't necessarily introduce new use cases that weren't there before, except where performance was a bottleneck. But we're talking like 500x performance increase on workloads vs ODBC, so night and day for a lot of data scientist pipelines.

zinclozenge 6y ago

Can you talk about how you use it? At a cursory glance, it seems like it's being used to great effect in databases, or systems designed to query data. Is that what you're using for?

Q6T46nT668w6i3m 6y ago
An alternative to CSV or SQLite for interchange.
- woadwarrior01 6y ago
  From my experience, Parquet files are a better alternative to CSV or SQLite files for data at rest. I've observed that parquet files tend to be smaller than Feather files (which is essentially the Arrow format in a file) for the same data. Arrow is great for in-memory data, but even with compression, the on-disk size of Feather files tend to be quite a bit bigger than Parquet files.
  - ttymck 6y ago
    "for interchange" in data science might well mean data on-the-move rather than data at-rest: i.e. a pipeline in two stages, step A in python and step B in R. The data is exchanged (relatively) quickly via disk, and speed beats compression.
    - woadwarrior01 6y ago
      Even with fast SSDs, memcpy() is roughly an order of magnitude faster than writing to disk. Using a fast compression algorithm like lz4, zstd, etc can help reduce that gap. And this is invariant of whether the serialization format is Arrow or Parquet. It’s not either, or. If you’re writing to disk, with compression you can have both speed and reduction in the disk footprint.

michaelcampbell 6y ago

The docs are dire. Lots and lots about internals and how it works, and after a few minutes of searching, I've yet to see what it actually does or what I'd want to use it for. The "use cases" are even overly technical things that are building blocks for ... something. What is that something?

baq 6y ago
judging from the comments here, it's an in-memory (in-process?) columnar database format without a DBMS attached. sounds useful when sqlite is too slow for your aggregations or if you do processing in X (C, Java) and visualization in Y (R, Python) - just guessing, haven't used it.
asdffdsa 6y ago
Judging from the faq (https://arrow.apache.org/faq/ https://arrow.apache.org/faq/), it seems like it's a storage format for large structured datasets, designed specifically for use in-memory and for serialization across the network. It provides a spec for the data format, as well as implementations for several programming languages. That seems pretty straightforward, what part was confusing for you?
- michaelcampbell 6y ago
  > what part was confusing for you? The passive aggressive isn't necessary.
  - asdffdsa 6y ago
    His snark isn't necessary; the docs are fine.
kthejoker2 6y ago
If you are already doing Big Data / ML / deep learning work in Spark or Dask, Arrow is (with caveats) more or less a drop-in performance optimization (because it's optimized for in-memory vectors) and complexity reducer (because it does all the interop between JVM, Python, TF, etc) So by itself it doesn't necessarily introduce new use cases that weren't there before, except where performance was a bottleneck. But we're talking like 500x performance increase on workloads vs ODBC, so night and day for a lot of data scientist pipelines.