14 ms·

Apache Arrow 1.0

361 points by dragonsh 6y ago

pyuser583 6y ago
Yay! Apache Arrow is great project. Whenever I use is, people think I’m some kind of genius. It should get a lot more press.
- zinclozenge 6y ago
  Can you talk about how you use it? At a cursory glance, it seems like it's being used to great effect in databases, or systems designed to query data. Is that what you're using for?
  - Q6T46nT668w6i3m 6y ago
    An alternative to CSV or SQLite for interchange.
    - woadwarrior01 6y ago
      From my experience, Parquet files are a better alternative to CSV or SQLite files for data at rest. I've observed that parquet files tend to be smaller than Feather files (which is essentially the Arrow format in a file) for the same data. Arrow is great for in-memory data, but even with compression, the on-disk size of Feather files tend to be quite a bit bigger than Parquet files.
      - ttymck 6y ago
        "for interchange" in data science might well mean data on-the-move rather than data at-rest: i.e. a pipeline in two stages, step A in python and step B in R. The data is exchanged (relatively) quickly via disk, and speed beats compression.
        woadwarrior01 6y ago
        Even with fast SSDs, memcpy() is roughly an order of magnitude faster than writing to disk. Using a fast compression algorithm like lz4, zstd, etc can help reduce that gap. And this is invariant of whether the serialization format is Arrow or Parquet. It’s not either, or. If you’re writing to disk, with compression you can have both speed and reduction in the disk footprint.
- michaelcampbell 6y ago
  The docs are dire. Lots and lots about internals and how it works, and after a few minutes of searching, I've yet to see what it actually does or what I'd want to use it for. The "use cases" are even overly technical things that are building blocks for ... something. What is that something?
  - baq 6y ago
    judging from the comments here, it's an in-memory (in-process?) columnar database format without a DBMS attached. sounds useful when sqlite is too slow for your aggregations or if you do processing in X (C, Java) and visualization in Y (R, Python) - just guessing, haven't used it.
  - asdffdsa 6y ago
    Judging from the faq (https://arrow.apache.org/faq/ https://arrow.apache.org/faq/), it seems like it's a storage format for large structured datasets, designed specifically for use in-memory and for serialization across the network. It provides a spec for the data format, as well as implementations for several programming languages. That seems pretty straightforward, what part was confusing for you?
    - michaelcampbell 6y ago
      > what part was confusing for you? The passive aggressive isn't necessary.
      - asdffdsa 6y ago
        His snark isn't necessary; the docs are fine.
  - kthejoker2 6y ago
    If you are already doing Big Data / ML / deep learning work in Spark or Dask, Arrow is (with caveats) more or less a drop-in performance optimization (because it's optimized for in-memory vectors) and complexity reducer (because it does all the interop between JVM, Python, TF, etc) So by itself it doesn't necessarily introduce new use cases that weren't there before, except where performance was a bottleneck. But we're talking like 500x performance increase on workloads vs ODBC, so night and day for a lot of data scientist pipelines.
tomhoule 6y ago
I checked recently, and the Rust implementation of arrow — parquet as well — is sadly still not usable on projects using a stable version of the compiler because it relies on specialization. There are some Jira issues on this, but there doesn't seem to be a consensus on the way forward. Does someone have more information, is the general idea to wait for specialization to stabilise, or is there a plan, or even an intention, to stop relying on it?
- nemothekid 6y ago
  Last I checked when I tried to the library last the blocker on stable was packed_simd, which provides vector instructions in Rust. I can imagine the arrow/datafusion-guy isn't too keen on dropping vectorized instructions as that would be letting up a huge advantage and I'm imagine it's used liberally throughout the code. As for stablizing packed_simd, It's completely unclear to me when that will land in stable rust. I recently had a project where I just ended up calling out to C code to handle vectorization. EDIT: According to, https://github.com/rust-lang/lang-team/issues/29 https://github.com/rust-lang/lang-team/issues/29, the effort looks abandonded/deprioritized, so it may be a long time before it sees stable rust
  - lordsunland 6y ago
    The main blocker is specialization as it's being used in several places including parquet and arrow crate. As this feature is unlikely to be stabilized, we'll need to replace it with something else but so far it is challenging.
    - tomhoule 6y ago
      Thanks for clarifying that the plan is to stop relying on it :) Is there a specific place/issue/ticket to discuss this? If time allows, I would be interested in helping out.
      - lordsunland 6y ago
        There is https://issues.apache.org/jira/browse/ARROW-6717 https://issues.apache.org/jira/browse/ARROW-6717 tracking moving to stable Rust, and https://issues.apache.org/jira/browse/ARROW-4678 https://issues.apache.org/jira/browse/ARROW-4678 is also relevant. Feel free to create a sub-task under the former to remove specialization :)
jakearmitage 6y ago
So, maybe I'm stupid and never faced enough big data, but what's the advantage of this versus a custom script that queries A and inserts into B? I do that kind of stuff all the time with Go and it's pretty fast with 20~40 million records, averaging 100 KB each. Are those tools oriented to billions, instead of millions? What are the benefits?
- th0ma5 6y ago
  I see Arrow as a "data frame server" and data frames have a higher level of abstraction for use in analysis. But you're right that raw processing can be done with most languages natively much faster. But Arrow can let a Python statistician and an R statistician compare apples to apples easily.
- cocobro 6y ago
  This article might help, it explains Arrow's performance benefits: https://www.dremio.com/apache-arrow-explained/ https://www.dremio.com/apache-arrow-explained/
- yokaze 6y ago
  One major use-case is exchanging data between processes (in different languages) using IPC, avoiding the overhead of serialisation. Best example is probably pyspark.
  - lenkite 6y ago
    Can't one do this using protobuf, flatbuffers, etc ? What advantage does arrow offer here ?
    - wenc 6y ago
      That's precisely it -- you don't have to use protobuf etc. because you don't have to serialize/deserialize at all. Languages that use Arrow can interact with the source data directly in-memory, in-process. No need to move any data around. The fastest serde operation is one that you don't have to do at all.
neurobashing 6y ago
given the sheer number and scope of Apache projects, it would be nice to have the title mention what it is (eg, "Apache Arrow - in memory analytics - 1.0.0") esp when the linked page doesn't directly say.
- j45 6y ago
  I find the same, but normally most Apache projects have an overview link at the top of the project.
- wenc 6y ago
  Normally I would agree but Apache Arrow already has significant name-recognition to people who are likely to use it -- data engineers/scientists etc. It's a library that provides an in-memory columnar format, and supplies a data engine currently used in Apache Spark, Pandas, and libraries like turbodbc, which helps these tools achieve high performance on operations on tabular data. Having a single high-performance in-memory format means different programs can read/write from the same source without serializing/copying/deserializing. For instance, if you wanted to pass a huge table of data from R to Java to Python (because your tools span different languages), normally you'd have to copy and serde (Protobuf? JSON?) to pass the data, which equals huge overheads. With Arrow, each of those languages can directly interact with the same copy of in-memory data, in-process -- with the highest possible performance. You also get the performance of columnar databases without implementing your own columnar data structure. But of course, no harm adding a short description to the title to broaden its audience. Arrow is truly something amazing and the more people know about it the better.[1] Folks who program against traditional databases might not know about it, and I think they should, especially if they need to generate analytics (i.e. fast filtering/aggregation for dashboarding or for data pipeline tasks). [1] Overview: https://arrow.apache.org/overview/ https://arrow.apache.org/overview/
  - ghshephard 6y ago
    I spend 8+ hours a day working with/ingesting data. Had no clue. Checked with my data-using colleagues just to see if any of them had heard of Arrow - about 50% had. These are people who spend their entire day working/ingesting data in various pipelines. So, while I won't comment on whether the subtitle would be useful, I'm pretty confident that the (perhaps not large) majority of people who would/could use Apache Arrow had not hear of it. A lot of people just stay in their lane. Having a solid description, like the one you provided, would be super useful. Perhaps a tool-tip feature of HN, even.
x87678r 6y ago
When I first saw Arrow I thought the biggest benefit was a good file format. (Feather I think is the format. ) However it seems to be designed for passing tables between languages in the same process. If I never use multiple languages in the same process can I safely ignore it or are there other benefits?
- tomrod 6y ago
  To_feather and from_feather are a wonderful addition if you use R and Python, or have a mixed language data science shop.
  - x87678r 6y ago
    Yes but is it any better than parquet or hdf? edit - this looks a good explanation, parquet is more widely used and smaller, but slower. https://stackoverflow.com/questions/48083405/what-are-the-differences-between-feather-and-parquet https://stackoverflow.com/questions/48083405/what-are-the-di...
    - kylebarron 6y ago
      The compression improvements in Parquet can be really great, especially for large, wide tables where you might only need a few columns for analysis.
    - tomrod 6y ago
      Hdf I've not used in any production workflow. Parquet I've really only used with impala. If I'm working in that system I'm in Spark already, typically.
- poorman 6y ago
  It also abstracts away optimizations for doing operations. For example tolowercase on a large number of elements.
- lmeyerov 6y ago
  In Python, we use Arrow a lot for going between diff frameworks: file -> ( pandas <> cudf <> cugraph ). When we work with DB vendors, we now push them to provide Arrow instead of whatever proprietary in-house thing they've been pushing for the same, e.g., slow JSON / ODBC, untyped CSV, or some ad-hoc wire protocol. Also, we use for going between our services within a server + across. On the nodejs services + frontend JS side, there was no real equiv tool for interop -- traditional soln is slowly round-tripping through SQL/ORM or manually doing protobuf through some sort of pubsub -- so Arrow is part of how we have been taming that mess too. It'll be a longer journey for Arrow or something like it to get adopted in JS land, but I can see folks doing TypeScript and serverless wanting it for a cleaner solution to typed data, faster serialization & streaming, etc. (There is no true JS equiv of pandas nor a typed variant.) We were a bit early here b/c we wanted streaming through WebGL + OpenCL/CUDA, and while we are fans of typed data, found protobuf tooling to be too unintegrated and manual in practice.
teleforce 6y ago
Now Arrow already stable hopefully TileDB can properly support it since they really complement each other [1]. [1]https://news.ycombinator.com/item?id=15561090 https://news.ycombinator.com/item?id=15561090
billman 6y ago
Congrats to the team for the 1.0 release.
Abishek_Muthian 6y ago
Apache Arrow is a nice in-memory data structure finding its usage in wide variety of projects; especially in data science due to its feather format which can be used to save data frame from memory to the disk. This can particularly be useful for low memory systems like ARM SBC when conducting long duration research. If you want to build Apache Arrow from source for ARM, I've written a How-To here[1]. [1]https://gist.github.com/heavyinfo/04e1326bb9bed9cecb19c2d603c8d521 https://gist.github.com/heavyinfo/04e1326bb9bed9cecb19c2d603...
snicker7 6y ago
Polyglot, zero-copy serialization protocols (arrow, flatbuffer, capn proto) is a trend I really appreciate. I like to imagine a future where data is freed from the languages/tools used to operate on them. In-memory objects would then be views into data stored in shared memory (or disk), with the data easily read/manipulated from multiple languages.
mands 6y ago
Congrats on the 1.0 release! We've been using Arrow for a few years in our startup (link in profile) - it's been great as a common format for passing tabular data between Python and Javascript. Haven't used the in-memory/zero-copy features as much, but as a binary, high-performance, typed format for the data analytics world it can't be beat. And now that the Feather format is basically the Arrow format, I expect to see it really take off as a common interchange and even storage format for medium- to even long-term projects. (Also nice to see the reduced Python wheel size - a little bonus for the 1.0 release :) )
oxfordmale 6y ago
We extensively use Apache Arrow to store data files as parquet files on S3. This is a cheap way to store data that doesn't require the query speed of a relational (or non relational database). The main advantage of Arrow is that is a columnar database, and loses no information in transit unlike the nightmare of CSV files.
- lsorber 6y ago
  Have you benchmarked this against pickling those data files? In our experience, parquet's overhead isn't worth it for smaller data files.
  - EdwardDiego 6y ago
    I'm not surprised, Parquet's columnar encoding and compression won't really kick in significantly for smaller files.
  - kylebarron 6y ago
    But with pickling you can only read the data in Python.
  - alfalfasprout 6y ago
    I just did some benchmarks and it's pretty similar for small files. The difference would only be noticeable if you're serializing a ton of small files.
    - lsorber 6y ago
      Huh, makes a pretty big difference for us. We were using pandas' built-in to_parquet though, which seems to suffer from some overhead.
  - cbsmith 6y ago
    If pickling is what is working best for you, it can't be much data.
gwittel 6y ago
Glad to see the 1.0.0 release. I'm hoping that it will help reduce resistance when I nudge people toward it. It also good to see progress in the documentation. For a long time its been very hard to dig into anything but the C++ or Python libraries. As I try to move to Arrow/Flight over using JSON or binary formats like Protobuf, one thing I see missing is tooling or schema -> code generation. It would be nice to see some progress or roadmap in that direction as well.
unclepadre 6y ago
Wes McKinney (Pandas creator and Arrow co-creator) will be giving an Arrow talk live this Thursday for those who want to learn more -> https://subsurfaceconf.com/summer2020 https://subsurfaceconf.com/summer2020
maximilianburke 6y ago
Big fan of Arrow (and Parquet) and we use it at UrbanLogiq extensively. It's a wonderful toolset and great for data interop across multiple languages and environments!
lmeyerov 6y ago
Congrats! And a big thank you to Wes for being such a devoted community leader! We've been on quite the journey here: https://www.graphistry.com/blog/graphistry-2-29-5-upload-100x-more-rapids-0-13-learnrapids-com-and-more https://www.graphistry.com/blog/graphistry-2-29-5-upload-100... . Think json -> protobuf -> arrow, and paralleled in our work on parallel js -> opencl+js -> python rapids/cuda for accelerated native compute over it. The blogpost demos the ~100X bigger & faster dataset result of supporting Arrow for our uploader & RAPIDS for our parser when you can't and want us to convert for you. Something folks miss with Arrow, IMO, is it is like google protobufs for everyone else. Arrow is not just having a nice binary format, but is also ready out-of-the-box for streaming, rich datatypes, and for larger/longer-term projects, standardized & auto self-describing schemas. If you've had to manually decipher, maintain, and update generated protobuf schemas (because you aren't google with all the internal ~protobuf tooling / integrations / infra / etc.), that should sound pretty good ;-) A lot more to do, but already way ahead of most other things, esp. in aggregate.
wesm 6y ago
hi, Wes (Apache Arrow co-creator and Python pandas creator) here! If you're wondering what this project is all about, my JupyterCon keynote (18 min long) from 3 years ago is a good summary and the vision / scope for what we've been doing since 2016 has been pretty consistent https://www.youtube.com/watch?v=wdmf1msbtVs https://www.youtube.com/watch?v=wdmf1msbtVs
- tgb 6y ago
  Thanks Wes, pandas and Arrow are great projects. Is Feather now ready for long-term storage with the V2 release? And now that it's just a renaming of the Arrow IPC format, what's its future?
  - tbenst 6y ago
    I’m also very confused about the relationship of Arrow’s stability guarantees and that of the on-disk feather. Can we safely switch from parquet to feather for long-term data storage?
    - sandGorgon 6y ago
      same question here - why Feather and why is it named differently ? Also is Parquet and Arrow the same ? df.to_parquet('df.parquet.gzip', compression='gzip') will not use arrow i presume ? i have to use a separate library to save to parquet using arrow. a bit confused.
      - lmeyerov 6y ago
        Graphistry uses parquet as a more stable and thus persistent storage format when folks save data, and arrow for ephemeral internal data where we're ok (and somewhat enjoy) version changes, as that just means code version upgrades. There are performance differences in practice such as parquet having more per-column compression modes built in, making it attractive for colder storage, and arrow for in-memory/rpc/streaming/etc for similar reasons. RE:Feather -- Arrow itself isn't necessarily a full file format -- you can imagine memory buffers being all over the heap with giant gaps inbetween b/c diff cols generated at diff times -- but in practice folks will indeed serialize to disk consolidated buffers (pa.Table -> write stream -> file). If we couldn't do that, RPC wouldn't work ;-) My understanding of Feather is it standardizes ideas around this consolidation, but we are able save to disk (within versions) without it. We found it more predictable to stick to ~Parquet for storage and Arrow buffer passing for streaming, but now that Feather networking APIs for accelerated bulk transfers may be stablizing, there may be speed advantages to using it over manual buffer streaming (and still stick w/ Parquet for persistent files). Arrow<>Parquet conversion is super fast b/c of the co-design around similar concepts: both using record batches of dense binary column buffers means implementations can pointer-copy, memory map, use bulk copy primitives, etc. for zero-copy or at least highly accelerated interop. Python RAPIDS GPU kernels can therefore selectively stream in a few parquet columns across many parquet files through a single 900GB/s GPU, compute over them, and write back out to arrow or a new parquet.
mindv0rtex 6y ago
How does this format compare to HDF5, which is common amount scientific developers? Are they meant for a similar use case or not?
- tbenst 6y ago
  HDF5 has no support for dataframes/tables, although projects have been built ontop of it like PyTables, but these are non-standard and don’t work across languages.
derriz 6y ago
Parquet/arrow is a great format for Pandas - fast and nicely compact in terms of file size for those of us who don't have the luxury of directly attached NVMe SSDs and where I/O bottlenecks are a consideration. Even after issues with its inability to map datetime64 values properly, I was reasonably happy with my design choice. I became less happy on discovering that it's very weak as an interchange format for cross-language work. In my case I wanted to use some existing JVM-based tooling. This caused huge pain. The Jvm/Java library/API is a complete mess, sorry, and if people are complaining about the Python/C++ documentation, there's basically nothing for the Java library. It's barely useable and the dependencies are horrific - the whole thing is mingled with hadoop dependencies - even the API itself. And the API is barely above exposing the file-format. Nothing like "load this parquet file" into some object which you can then query for it's contents - you're dealing with blocks and sections and other file-format level entities. The other issue is caused its flexibility - for example Panda's dataframes are written with what's effectively a bunch of "extension metadata" which means it works great for reading and writing pandas from Python but don't expect anything to be able to work with the files out-of-the-box in other languages. In the end, the only way I could get reliable reading and writing from the JVM was to only store numeric and string data from the Python side. Even then it feels flakey - with a bunch of hadoop warnings and deprecation warnings. I know the JVM has little appreciation in the data science world which is maybe a reason for the sorry state of the Java library. Edit: to be specific, I am talking about my experiences with Arrow/Parquet.
- MrPowers 6y ago
  Arrow is tightly integrated in Spark's Scala code: https://github.com/apache/spark/search?q=arrow&unscoped_q=arrow https://github.com/apache/spark/search?q=arrow&unscoped_q=ar... Andy Grove is building a "distributed compute platform implemented in Rust, using the Apache Arrow memory model": https://github.com/ballista-compute/ballista https://github.com/ballista-compute/ballista. Seems possible.
  - xiaodai 6y ago
    Rust will not succeed. Ppl can't debug spakr already and rust, even fewer ppl cna debug. Build a spark killer in Julia. Everyone can read the code.
thamer 6y ago
I used Apache Arrow to store and process a relatively large amount of data, 2-3B records with ~10 fields each (still fits on one machine, but it's large enough that I have to think about it a little bit). I was originally using a simple binary format for fast decoding, and switched to Arrow to be able to select only a few columns at a time. I was impressed by the speed gains, and the size benefits of storing data in column order. The one thing I wish Arrow had was a way to attach some metadata to its files (if there is, I haven't found it). I originally tried to write a small header before starting to write the Arrow data, but that made it impossible to read back the data: as far as I can tell Arrow looks at the whole file and stores its column definition at the end of the file and computes data offsets based on the start of the file, meaning that there's nowhere left for me to store anything. It's still a very promising library and I'll definitely be checking out the 1.0 release.
- lmeyerov 6y ago
  pyarrow.metadata in the header is for stuffing in uninterpreted bytes, so we put in semantic stringified json data there for version numbers etc. that are beyond the data rep typing . Super useful when maturing from localized notebook code to software services standardized on arrow interop.
  - thamer 6y ago
    Oh cool thanks! I'll check it out.
- infinite8s 6y ago
  Arrow has two possible formats - a stream based one where the first block is the schema/metadata followed by data blocks (ie record batches), and a file based one where the schema is also stored at the end, along with the index locations of each batch (this also allows random access): https://arrow.apache.org/docs/format/Columnar.html#ipc-streaming-format https://arrow.apache.org/docs/format/Columnar.html#ipc-strea... vs https://arrow.apache.org/docs/format/Columnar.html#ipc-file-format https://arrow.apache.org/docs/format/Columnar.html#ipc-file-...
e12e 6y ago
Nice. Is it just me, or is Julia notably missing an up to date arrow implementation?
yingw787 6y ago
I wish Excel / Google Sheets supported Parquet export. I've worked with CSV files and Parquet files and found the latter a vast improvement. Chunking data is trivial, you have fixed-size data types, and existing types are well-supported across a multitude of frameworks. In comparison, CSVs are pretty much unstructured text files with added suggestions. More pipelines should natively support Parquet files, or something like it, and have a thin CSV to Parquet conversion layer on top. P.S. I'd love to create a Parquet file visualization tool (something like https://github.com/in3rsha/sha256-animation https://github.com/in3rsha/sha256-animation) in the near future.
Niccizero 6y ago
Now that is has a 1.0 release, it might finally be allowed to have a Wikipedia page.
mamcx 6y ago
Is this format good for also CRUD-like/mixed workloads?
glogla 6y ago
What data types are available in Arrow? Avro and Parquet can't really do timestamp with timezone, which is a major pain when processing data from geographically distributed locations. But if Arrow can do it, then maybe there's hope for Parquet as well.
- andygrove 6y ago
  You should be able to find this info in the specification: https://arrow.apache.org/docs/format/Columnar.html https://arrow.apache.org/docs/format/Columnar.html
stokedmartin 6y ago
noob question -- what are some of the fundamental differences (may or may not help performance) between thrift and apache arrow?
- andygrove 6y ago
  Thrift is a serialization format. Arrow is a memory format.
endlessvoid94 6y ago
Is this like Memsql back in the day?
dreamcompiler 6y ago
"Columnar Memory Formats" Maybe I'm just getting old, but back when we had to melt our own sand to build a computer, we called these things arrays.
- erik_seaberg 6y ago
  Arrays of structs fit pretty naturally into many languages, but structs of arrays (and keeping corresponding values in order) are unusual enough to need plumbing that probably isn't there.
stilisstuk 6y ago
Say I have a 50GB (bigger than RAM) CSV file I want to analyse locally with R. Can I somehow use Apache arrow for that?
lloydatkinson 6y ago
I said this before but: Oh look, yet another Apache real time/batch/big data/stream processing/ingestion/workflow/whatever product. Apache Druid Apache Spark Apache Storm Apache Flink Apache Beam Apache Apex Apache Airavata Apache Samza Apache TEZ Apache Hama It's basically a terrible joke at this point. There's no single Apache page helping you to decide which one you want, and they all seem to have such large overlap. Most of them seem to have bad documentation, and give the appearence of not really being maintained. This puts me off even trying to use them. If there's this much scope creep/NIH/reinventing the wheel happening across the board, I can't imagine how bad each product is individually. Apache Kafka seems to be the only exception...
- wowi42 6y ago
  Apache Pulsar?

361 points by dragonsh 6y ago

Yay! Apache Arrow is great project. Whenever I use is, people think I’m some kind of genius. It should get a lot more press.

zinclozenge 6y ago
Can you talk about how you use it? At a cursory glance, it seems like it's being used to great effect in databases, or systems designed to query data. Is that what you're using for?
- Q6T46nT668w6i3m 6y ago
  An alternative to CSV or SQLite for interchange.
  - woadwarrior01 6y ago
    From my experience, Parquet files are a better alternative to CSV or SQLite files for data at rest. I've observed that parquet files tend to be smaller than Feather files (which is essentially the Arrow format in a file) for the same data. Arrow is great for in-memory data, but even with compression, the on-disk size of Feather files tend to be quite a bit bigger than Parquet files.
    - ttymck 6y ago
      "for interchange" in data science might well mean data on-the-move rather than data at-rest: i.e. a pipeline in two stages, step A in python and step B in R. The data is exchanged (relatively) quickly via disk, and speed beats compression.
      - woadwarrior01 6y ago
        Even with fast SSDs, memcpy() is roughly an order of magnitude faster than writing to disk. Using a fast compression algorithm like lz4, zstd, etc can help reduce that gap. And this is invariant of whether the serialization format is Arrow or Parquet. It’s not either, or. If you’re writing to disk, with compression you can have both speed and reduction in the disk footprint.
michaelcampbell 6y ago
The docs are dire. Lots and lots about internals and how it works, and after a few minutes of searching, I've yet to see what it actually does or what I'd want to use it for. The "use cases" are even overly technical things that are building blocks for ... something. What is that something?
- baq 6y ago
  judging from the comments here, it's an in-memory (in-process?) columnar database format without a DBMS attached. sounds useful when sqlite is too slow for your aggregations or if you do processing in X (C, Java) and visualization in Y (R, Python) - just guessing, haven't used it.
- asdffdsa 6y ago
  Judging from the faq (https://arrow.apache.org/faq/ https://arrow.apache.org/faq/), it seems like it's a storage format for large structured datasets, designed specifically for use in-memory and for serialization across the network. It provides a spec for the data format, as well as implementations for several programming languages. That seems pretty straightforward, what part was confusing for you?
  - michaelcampbell 6y ago
    > what part was confusing for you? The passive aggressive isn't necessary.
    - asdffdsa 6y ago
      His snark isn't necessary; the docs are fine.
- kthejoker2 6y ago
  If you are already doing Big Data / ML / deep learning work in Spark or Dask, Arrow is (with caveats) more or less a drop-in performance optimization (because it's optimized for in-memory vectors) and complexity reducer (because it does all the interop between JVM, Python, TF, etc) So by itself it doesn't necessarily introduce new use cases that weren't there before, except where performance was a bottleneck. But we're talking like 500x performance increase on workloads vs ODBC, so night and day for a lot of data scientist pipelines.

tomhoule 6y ago

I checked recently, and the Rust implementation of arrow — parquet as well — is sadly still not usable on projects using a stable version of the compiler because it relies on specialization. There are some Jira issues on this, but there doesn't seem to be a consensus on the way forward. Does someone have more information, is the general idea to wait for specialization to stabilise, or is there a plan, or even an intention, to stop relying on it?

nemothekid 6y ago
Last I checked when I tried to the library last the blocker on stable was packed_simd, which provides vector instructions in Rust. I can imagine the arrow/datafusion-guy isn't too keen on dropping vectorized instructions as that would be letting up a huge advantage and I'm imagine it's used liberally throughout the code. As for stablizing packed_simd, It's completely unclear to me when that will land in stable rust. I recently had a project where I just ended up calling out to C code to handle vectorization. EDIT: According to, https://github.com/rust-lang/lang-team/issues/29 https://github.com/rust-lang/lang-team/issues/29, the effort looks abandonded/deprioritized, so it may be a long time before it sees stable rust
- lordsunland 6y ago
  The main blocker is specialization as it's being used in several places including parquet and arrow crate. As this feature is unlikely to be stabilized, we'll need to replace it with something else but so far it is challenging.
  - tomhoule 6y ago
    Thanks for clarifying that the plan is to stop relying on it :) Is there a specific place/issue/ticket to discuss this? If time allows, I would be interested in helping out.
    - lordsunland 6y ago
      There is https://issues.apache.org/jira/browse/ARROW-6717 https://issues.apache.org/jira/browse/ARROW-6717 tracking moving to stable Rust, and https://issues.apache.org/jira/browse/ARROW-4678 https://issues.apache.org/jira/browse/ARROW-4678 is also relevant. Feel free to create a sub-task under the former to remove specialization :)

jakearmitage 6y ago

So, maybe I'm stupid and never faced enough big data, but what's the advantage of this versus a custom script that queries A and inserts into B? I do that kind of stuff all the time with Go and it's pretty fast with 20~40 million records, averaging 100 KB each. Are those tools oriented to billions, instead of millions? What are the benefits?

th0ma5 6y ago
I see Arrow as a "data frame server" and data frames have a higher level of abstraction for use in analysis. But you're right that raw processing can be done with most languages natively much faster. But Arrow can let a Python statistician and an R statistician compare apples to apples easily.
cocobro 6y ago
This article might help, it explains Arrow's performance benefits: https://www.dremio.com/apache-arrow-explained/ https://www.dremio.com/apache-arrow-explained/
yokaze 6y ago
One major use-case is exchanging data between processes (in different languages) using IPC, avoiding the overhead of serialisation. Best example is probably pyspark.
- lenkite 6y ago
  Can't one do this using protobuf, flatbuffers, etc ? What advantage does arrow offer here ?
  - wenc 6y ago
    That's precisely it -- you don't have to use protobuf etc. because you don't have to serialize/deserialize at all. Languages that use Arrow can interact with the source data directly in-memory, in-process. No need to move any data around. The fastest serde operation is one that you don't have to do at all.

neurobashing 6y ago

given the sheer number and scope of Apache projects, it would be nice to have the title mention what it is (eg, "Apache Arrow - in memory analytics - 1.0.0") esp when the linked page doesn't directly say.

j45 6y ago
I find the same, but normally most Apache projects have an overview link at the top of the project.
wenc 6y ago
Normally I would agree but Apache Arrow already has significant name-recognition to people who are likely to use it -- data engineers/scientists etc. It's a library that provides an in-memory columnar format, and supplies a data engine currently used in Apache Spark, Pandas, and libraries like turbodbc, which helps these tools achieve high performance on operations on tabular data. Having a single high-performance in-memory format means different programs can read/write from the same source without serializing/copying/deserializing. For instance, if you wanted to pass a huge table of data from R to Java to Python (because your tools span different languages), normally you'd have to copy and serde (Protobuf? JSON?) to pass the data, which equals huge overheads. With Arrow, each of those languages can directly interact with the same copy of in-memory data, in-process -- with the highest possible performance. You also get the performance of columnar databases without implementing your own columnar data structure. But of course, no harm adding a short description to the title to broaden its audience. Arrow is truly something amazing and the more people know about it the better.[1] Folks who program against traditional databases might not know about it, and I think they should, especially if they need to generate analytics (i.e. fast filtering/aggregation for dashboarding or for data pipeline tasks). [1] Overview: https://arrow.apache.org/overview/ https://arrow.apache.org/overview/
- ghshephard 6y ago
  I spend 8+ hours a day working with/ingesting data. Had no clue. Checked with my data-using colleagues just to see if any of them had heard of Arrow - about 50% had. These are people who spend their entire day working/ingesting data in various pipelines. So, while I won't comment on whether the subtitle would be useful, I'm pretty confident that the (perhaps not large) majority of people who would/could use Apache Arrow had not hear of it. A lot of people just stay in their lane. Having a solid description, like the one you provided, would be super useful. Perhaps a tool-tip feature of HN, even.

x87678r 6y ago

When I first saw Arrow I thought the biggest benefit was a good file format. (Feather I think is the format. ) However it seems to be designed for passing tables between languages in the same process. If I never use multiple languages in the same process can I safely ignore it or are there other benefits?

tomrod 6y ago
To_feather and from_feather are a wonderful addition if you use R and Python, or have a mixed language data science shop.
- x87678r 6y ago
  Yes but is it any better than parquet or hdf? edit - this looks a good explanation, parquet is more widely used and smaller, but slower. https://stackoverflow.com/questions/48083405/what-are-the-differences-between-feather-and-parquet https://stackoverflow.com/questions/48083405/what-are-the-di...
  - kylebarron 6y ago
    The compression improvements in Parquet can be really great, especially for large, wide tables where you might only need a few columns for analysis.
  - tomrod 6y ago
    Hdf I've not used in any production workflow. Parquet I've really only used with impala. If I'm working in that system I'm in Spark already, typically.
poorman 6y ago
It also abstracts away optimizations for doing operations. For example tolowercase on a large number of elements.
lmeyerov 6y ago
In Python, we use Arrow a lot for going between diff frameworks: file -> ( pandas <> cudf <> cugraph ). When we work with DB vendors, we now push them to provide Arrow instead of whatever proprietary in-house thing they've been pushing for the same, e.g., slow JSON / ODBC, untyped CSV, or some ad-hoc wire protocol. Also, we use for going between our services within a server + across. On the nodejs services + frontend JS side, there was no real equiv tool for interop -- traditional soln is slowly round-tripping through SQL/ORM or manually doing protobuf through some sort of pubsub -- so Arrow is part of how we have been taming that mess too. It'll be a longer journey for Arrow or something like it to get adopted in JS land, but I can see folks doing TypeScript and serverless wanting it for a cleaner solution to typed data, faster serialization & streaming, etc. (There is no true JS equiv of pandas nor a typed variant.) We were a bit early here b/c we wanted streaming through WebGL + OpenCL/CUDA, and while we are fans of typed data, found protobuf tooling to be too unintegrated and manual in practice.

teleforce 6y ago

Now Arrow already stable hopefully TileDB can properly support it since they really complement each other [1]. [1]https://news.ycombinator.com/item?id=15561090 https://news.ycombinator.com/item?id=15561090

billman 6y ago

Congrats to the team for the 1.0 release.

Abishek_Muthian 6y ago

Apache Arrow is a nice in-memory data structure finding its usage in wide variety of projects; especially in data science due to its feather format which can be used to save data frame from memory to the disk. This can particularly be useful for low memory systems like ARM SBC when conducting long duration research. If you want to build Apache Arrow from source for ARM, I've written a How-To here[1]. [1]https://gist.github.com/heavyinfo/04e1326bb9bed9cecb19c2d603c8d521 https://gist.github.com/heavyinfo/04e1326bb9bed9cecb19c2d603...

snicker7 6y ago

Polyglot, zero-copy serialization protocols (arrow, flatbuffer, capn proto) is a trend I really appreciate. I like to imagine a future where data is freed from the languages/tools used to operate on them. In-memory objects would then be views into data stored in shared memory (or disk), with the data easily read/manipulated from multiple languages.

mands 6y ago

Congrats on the 1.0 release! We've been using Arrow for a few years in our startup (link in profile) - it's been great as a common format for passing tabular data between Python and Javascript. Haven't used the in-memory/zero-copy features as much, but as a binary, high-performance, typed format for the data analytics world it can't be beat. And now that the Feather format is basically the Arrow format, I expect to see it really take off as a common interchange and even storage format for medium- to even long-term projects. (Also nice to see the reduced Python wheel size - a little bonus for the 1.0 release :) )

oxfordmale 6y ago

We extensively use Apache Arrow to store data files as parquet files on S3. This is a cheap way to store data that doesn't require the query speed of a relational (or non relational database). The main advantage of Arrow is that is a columnar database, and loses no information in transit unlike the nightmare of CSV files.

lsorber 6y ago
Have you benchmarked this against pickling those data files? In our experience, parquet's overhead isn't worth it for smaller data files.
- EdwardDiego 6y ago
  I'm not surprised, Parquet's columnar encoding and compression won't really kick in significantly for smaller files.
- kylebarron 6y ago
  But with pickling you can only read the data in Python.
- alfalfasprout 6y ago
  I just did some benchmarks and it's pretty similar for small files. The difference would only be noticeable if you're serializing a ton of small files.
  - lsorber 6y ago
    Huh, makes a pretty big difference for us. We were using pandas' built-in to_parquet though, which seems to suffer from some overhead.
- cbsmith 6y ago
  If pickling is what is working best for you, it can't be much data.

gwittel 6y ago

Glad to see the 1.0.0 release. I'm hoping that it will help reduce resistance when I nudge people toward it. It also good to see progress in the documentation. For a long time its been very hard to dig into anything but the C++ or Python libraries. As I try to move to Arrow/Flight over using JSON or binary formats like Protobuf, one thing I see missing is tooling or schema -> code generation. It would be nice to see some progress or roadmap in that direction as well.

unclepadre 6y ago

Wes McKinney (Pandas creator and Arrow co-creator) will be giving an Arrow talk live this Thursday for those who want to learn more -> https://subsurfaceconf.com/summer2020 https://subsurfaceconf.com/summer2020

maximilianburke 6y ago

Big fan of Arrow (and Parquet) and we use it at UrbanLogiq extensively. It's a wonderful toolset and great for data interop across multiple languages and environments!

lmeyerov 6y ago

Congrats! And a big thank you to Wes for being such a devoted community leader! We've been on quite the journey here: https://www.graphistry.com/blog/graphistry-2-29-5-upload-100x-more-rapids-0-13-learnrapids-com-and-more https://www.graphistry.com/blog/graphistry-2-29-5-upload-100... . Think json -> protobuf -> arrow, and paralleled in our work on parallel js -> opencl+js -> python rapids/cuda for accelerated native compute over it. The blogpost demos the ~100X bigger & faster dataset result of supporting Arrow for our uploader & RAPIDS for our parser when you can't and want us to convert for you. Something folks miss with Arrow, IMO, is it is like google protobufs for everyone else. Arrow is not just having a nice binary format, but is also ready out-of-the-box for streaming, rich datatypes, and for larger/longer-term projects, standardized & auto self-describing schemas. If you've had to manually decipher, maintain, and update generated protobuf schemas (because you aren't google with all the internal ~protobuf tooling / integrations / infra / etc.), that should sound pretty good ;-) A lot more to do, but already way ahead of most other things, esp. in aggregate.

wesm 6y ago

hi, Wes (Apache Arrow co-creator and Python pandas creator) here! If you're wondering what this project is all about, my JupyterCon keynote (18 min long) from 3 years ago is a good summary and the vision / scope for what we've been doing since 2016 has been pretty consistent https://www.youtube.com/watch?v=wdmf1msbtVs https://www.youtube.com/watch?v=wdmf1msbtVs

tgb 6y ago
Thanks Wes, pandas and Arrow are great projects. Is Feather now ready for long-term storage with the V2 release? And now that it's just a renaming of the Arrow IPC format, what's its future?
- tbenst 6y ago
  I’m also very confused about the relationship of Arrow’s stability guarantees and that of the on-disk feather. Can we safely switch from parquet to feather for long-term data storage?
  - sandGorgon 6y ago
    same question here - why Feather and why is it named differently ? Also is Parquet and Arrow the same ? df.to_parquet('df.parquet.gzip', compression='gzip') will not use arrow i presume ? i have to use a separate library to save to parquet using arrow. a bit confused.
    - lmeyerov 6y ago
      Graphistry uses parquet as a more stable and thus persistent storage format when folks save data, and arrow for ephemeral internal data where we're ok (and somewhat enjoy) version changes, as that just means code version upgrades. There are performance differences in practice such as parquet having more per-column compression modes built in, making it attractive for colder storage, and arrow for in-memory/rpc/streaming/etc for similar reasons. RE:Feather -- Arrow itself isn't necessarily a full file format -- you can imagine memory buffers being all over the heap with giant gaps inbetween b/c diff cols generated at diff times -- but in practice folks will indeed serialize to disk consolidated buffers (pa.Table -> write stream -> file). If we couldn't do that, RPC wouldn't work ;-) My understanding of Feather is it standardizes ideas around this consolidation, but we are able save to disk (within versions) without it. We found it more predictable to stick to ~Parquet for storage and Arrow buffer passing for streaming, but now that Feather networking APIs for accelerated bulk transfers may be stablizing, there may be speed advantages to using it over manual buffer streaming (and still stick w/ Parquet for persistent files). Arrow<>Parquet conversion is super fast b/c of the co-design around similar concepts: both using record batches of dense binary column buffers means implementations can pointer-copy, memory map, use bulk copy primitives, etc. for zero-copy or at least highly accelerated interop. Python RAPIDS GPU kernels can therefore selectively stream in a few parquet columns across many parquet files through a single 900GB/s GPU, compute over them, and write back out to arrow or a new parquet.

mindv0rtex 6y ago

How does this format compare to HDF5, which is common amount scientific developers? Are they meant for a similar use case or not?

tbenst 6y ago
HDF5 has no support for dataframes/tables, although projects have been built ontop of it like PyTables, but these are non-standard and don’t work across languages.

derriz 6y ago

Parquet/arrow is a great format for Pandas - fast and nicely compact in terms of file size for those of us who don't have the luxury of directly attached NVMe SSDs and where I/O bottlenecks are a consideration. Even after issues with its inability to map datetime64 values properly, I was reasonably happy with my design choice. I became less happy on discovering that it's very weak as an interchange format for cross-language work. In my case I wanted to use some existing JVM-based tooling. This caused huge pain. The Jvm/Java library/API is a complete mess, sorry, and if people are complaining about the Python/C++ documentation, there's basically nothing for the Java library. It's barely useable and the dependencies are horrific - the whole thing is mingled with hadoop dependencies - even the API itself. And the API is barely above exposing the file-format. Nothing like "load this parquet file" into some object which you can then query for it's contents - you're dealing with blocks and sections and other file-format level entities. The other issue is caused its flexibility - for example Panda's dataframes are written with what's effectively a bunch of "extension metadata" which means it works great for reading and writing pandas from Python but don't expect anything to be able to work with the files out-of-the-box in other languages. In the end, the only way I could get reliable reading and writing from the JVM was to only store numeric and string data from the Python side. Even then it feels flakey - with a bunch of hadoop warnings and deprecation warnings. I know the JVM has little appreciation in the data science world which is maybe a reason for the sorry state of the Java library. Edit: to be specific, I am talking about my experiences with Arrow/Parquet.

MrPowers 6y ago
Arrow is tightly integrated in Spark's Scala code: https://github.com/apache/spark/search?q=arrow&unscoped_q=arrow https://github.com/apache/spark/search?q=arrow&unscoped_q=ar... Andy Grove is building a "distributed compute platform implemented in Rust, using the Apache Arrow memory model": https://github.com/ballista-compute/ballista https://github.com/ballista-compute/ballista. Seems possible.
- xiaodai 6y ago
  Rust will not succeed. Ppl can't debug spakr already and rust, even fewer ppl cna debug. Build a spark killer in Julia. Everyone can read the code.

thamer 6y ago

I used Apache Arrow to store and process a relatively large amount of data, 2-3B records with ~10 fields each (still fits on one machine, but it's large enough that I have to think about it a little bit). I was originally using a simple binary format for fast decoding, and switched to Arrow to be able to select only a few columns at a time. I was impressed by the speed gains, and the size benefits of storing data in column order. The one thing I wish Arrow had was a way to attach some metadata to its files (if there is, I haven't found it). I originally tried to write a small header before starting to write the Arrow data, but that made it impossible to read back the data: as far as I can tell Arrow looks at the whole file and stores its column definition at the end of the file and computes data offsets based on the start of the file, meaning that there's nowhere left for me to store anything. It's still a very promising library and I'll definitely be checking out the 1.0 release.

lmeyerov 6y ago
pyarrow.metadata in the header is for stuffing in uninterpreted bytes, so we put in semantic stringified json data there for version numbers etc. that are beyond the data rep typing . Super useful when maturing from localized notebook code to software services standardized on arrow interop.
- thamer 6y ago
  Oh cool thanks! I'll check it out.
infinite8s 6y ago
Arrow has two possible formats - a stream based one where the first block is the schema/metadata followed by data blocks (ie record batches), and a file based one where the schema is also stored at the end, along with the index locations of each batch (this also allows random access): https://arrow.apache.org/docs/format/Columnar.html#ipc-streaming-format https://arrow.apache.org/docs/format/Columnar.html#ipc-strea... vs https://arrow.apache.org/docs/format/Columnar.html#ipc-file-format https://arrow.apache.org/docs/format/Columnar.html#ipc-file-...

e12e 6y ago

Nice. Is it just me, or is Julia notably missing an up to date arrow implementation?

yingw787 6y ago

I wish Excel / Google Sheets supported Parquet export. I've worked with CSV files and Parquet files and found the latter a vast improvement. Chunking data is trivial, you have fixed-size data types, and existing types are well-supported across a multitude of frameworks. In comparison, CSVs are pretty much unstructured text files with added suggestions. More pipelines should natively support Parquet files, or something like it, and have a thin CSV to Parquet conversion layer on top. P.S. I'd love to create a Parquet file visualization tool (something like https://github.com/in3rsha/sha256-animation https://github.com/in3rsha/sha256-animation) in the near future.

Niccizero 6y ago

Now that is has a 1.0 release, it might finally be allowed to have a Wikipedia page.

mamcx 6y ago

Is this format good for also CRUD-like/mixed workloads?

glogla 6y ago

What data types are available in Arrow? Avro and Parquet can't really do timestamp with timezone, which is a major pain when processing data from geographically distributed locations. But if Arrow can do it, then maybe there's hope for Parquet as well.

andygrove 6y ago
You should be able to find this info in the specification: https://arrow.apache.org/docs/format/Columnar.html https://arrow.apache.org/docs/format/Columnar.html

stokedmartin 6y ago

noob question -- what are some of the fundamental differences (may or may not help performance) between thrift and apache arrow?

andygrove 6y ago
Thrift is a serialization format. Arrow is a memory format.

endlessvoid94 6y ago

Is this like Memsql back in the day?

dreamcompiler 6y ago

"Columnar Memory Formats" Maybe I'm just getting old, but back when we had to melt our own sand to build a computer, we called these things arrays.

erik_seaberg 6y ago
Arrays of structs fit pretty naturally into many languages, but structs of arrays (and keeping corresponding values in order) are unusual enough to need plumbing that probably isn't there.

stilisstuk 6y ago

Say I have a 50GB (bigger than RAM) CSV file I want to analyse locally with R. Can I somehow use Apache arrow for that?

lloydatkinson 6y ago

I said this before but: Oh look, yet another Apache real time/batch/big data/stream processing/ingestion/workflow/whatever product. Apache Druid Apache Spark Apache Storm Apache Flink Apache Beam Apache Apex Apache Airavata Apache Samza Apache TEZ Apache Hama It's basically a terrible joke at this point. There's no single Apache page helping you to decide which one you want, and they all seem to have such large overlap. Most of them seem to have bad documentation, and give the appearence of not really being maintained. This puts me off even trying to use them. If there's this much scope creep/NIH/reinventing the wheel happening across the board, I can't imagine how bad each product is individually. Apache Kafka seems to be the only exception...

wowi42 6y ago
Apache Pulsar?