3 ms·

The docs are dire. Lots and lots about internals and how it works, and after a few minutes of searching, I've yet to see what it actually does or what I'd want

The docs are dire. Lots and lots about internals and how it works, and after a few minutes of searching, I've yet to see what it actually does or what I'd want to use it for. The "use cases" are even overly technical things that are building blocks for ... something. What is that something?

baq 6y ago
judging from the comments here, it's an in-memory (in-process?) columnar database format without a DBMS attached. sounds useful when sqlite is too slow for your aggregations or if you do processing in X (C, Java) and visualization in Y (R, Python) - just guessing, haven't used it.
asdffdsa 6y ago
Judging from the faq (https://arrow.apache.org/faq/ https://arrow.apache.org/faq/), it seems like it's a storage format for large structured datasets, designed specifically for use in-memory and for serialization across the network. It provides a spec for the data format, as well as implementations for several programming languages. That seems pretty straightforward, what part was confusing for you?
- michaelcampbell 6y ago
  > what part was confusing for you? The passive aggressive isn't necessary.
  - asdffdsa 6y ago
    His snark isn't necessary; the docs are fine.
kthejoker2 6y ago
If you are already doing Big Data / ML / deep learning work in Spark or Dask, Arrow is (with caveats) more or less a drop-in performance optimization (because it's optimized for in-memory vectors) and complexity reducer (because it does all the interop between JVM, Python, TF, etc) So by itself it doesn't necessarily introduce new use cases that weren't there before, except where performance was a bottleneck. But we're talking like 500x performance increase on workloads vs ODBC, so night and day for a lot of data scientist pipelines.
nwsm 6y ago
This is the overview; it's high level and informative. https://arrow.apache.org/overview/ https://arrow.apache.org/overview/ It boils down to a standard columnar format usable from disk or memory that is useful for data interchange between languages or frameworks, as well as any use case where a columnar format alone has benefits.

baq 6y ago

judging from the comments here, it's an in-memory (in-process?) columnar database format without a DBMS attached. sounds useful when sqlite is too slow for your aggregations or if you do processing in X (C, Java) and visualization in Y (R, Python) - just guessing, haven't used it.

asdffdsa 6y ago

Judging from the faq (https://arrow.apache.org/faq/ https://arrow.apache.org/faq/), it seems like it's a storage format for large structured datasets, designed specifically for use in-memory and for serialization across the network. It provides a spec for the data format, as well as implementations for several programming languages. That seems pretty straightforward, what part was confusing for you?

michaelcampbell 6y ago
> what part was confusing for you? The passive aggressive isn't necessary.
- asdffdsa 6y ago
  His snark isn't necessary; the docs are fine.

kthejoker2 6y ago

If you are already doing Big Data / ML / deep learning work in Spark or Dask, Arrow is (with caveats) more or less a drop-in performance optimization (because it's optimized for in-memory vectors) and complexity reducer (because it does all the interop between JVM, Python, TF, etc) So by itself it doesn't necessarily introduce new use cases that weren't there before, except where performance was a bottleneck. But we're talking like 500x performance increase on workloads vs ODBC, so night and day for a lot of data scientist pipelines.

nwsm 6y ago

This is the overview; it's high level and informative. https://arrow.apache.org/overview/ https://arrow.apache.org/overview/ It boils down to a standard columnar format usable from disk or memory that is useful for data interchange between languages or frameworks, as well as any use case where a columnar format alone has benefits.