15 ms·

Apache Arrow 3.0

561 points by kylebarron 6y ago

Thaxll 6y ago
Last time I worked in ETL was with Hadoop, looks like a lot happened.
- macksd 6y ago
  There's actually a lot of overlap between Hadoop and Arrow's origins - a lot of the projects that integrated early and the founding contributors had been in the larger Hadoop ecosystem. It's a very good sign IMO that you can hardly tell anymore - very diverse community and wide adoption!
georgewfraser 6y ago
Arrow is the most important thing happening in the data ecosystem right now. It's going to allow you to run your choice of execution engine, on top of your choice of data store, as though they are designed to work together. It will mostly be invisible to users, the key thing that needs to happen is that all the producers and consumers of batch data need to adopt Arrow as the common interchange format. BigQuery recently implemented the storage API, which allows you to read BQ tables, in parallel, in Arrow format: https://cloud.google.com/bigquery/docs/reference/storage https://cloud.google.com/bigquery/docs/reference/storage Snowflake has adopted Arrow as the in-memory format for their JDBC driver, though to my knowledge there is still no way to access data in parallel from Snowflake, other than to export to S3. As Arrow spreads across the ecosystem, users are going to start discovering that they can store data in one system and query it in another, at full speed, and it's going to be amazing.
- tristanz 6y ago
  Agreed! Thank you Arrow community for such a great project. It's a long road but it opens up tremendous potential for efficient data systems that talk to one another. The future looks bright with so many vendors backing Arrow independently and Wes McKinney founding Ursa Labs and now Ursa Computing. https://ursalabs.org/blog/ursa-computing/ https://ursalabs.org/blog/ursa-computing/
- wesm 6y ago
  Microsoft is also on top of this with their Magpie project http://cidrdb.org/cidr2021/papers/cidr2021_paper08.pdf http://cidrdb.org/cidr2021/papers/cidr2021_paper08.pdf "A common, efficient serialized and wire format across data engines is a transformational development. Many previous systems and approaches (e.g., [26, 36, 38, 51]) have observed the prohibitive cost of data conversion and transfer, precluding optimizers from exploiting inter-DBMS performance advantages. By contrast, inmemory data transfer cost between a pair of Arrow-supporting systems is effectively zero. Many major, modern DBMSs (e.g., Spark, Kudu, AWS Data Wrangler, SciDB, TileDB) and data-processing frameworks (e.g., Pandas, NumPy, Dask) have or are in the process of incorporating support for Arrow and ArrowFlight. Exploiting this is key for Magpie, which is thereby free to combine data from different sources and cache intermediate data and results, without needing to consider data conversion overhead."
  - data_ders 6y ago
    way cool! Is magpie end-user facing anywhere yet? We were using the azureml-dataprep library for a while which seems similar but not all of magpie
  - polskibus 6y ago
    I wish MS put in some resources behind Arrow in .NET. I tried raising some remarks about it on dotnet repos (esp. within ML.NET), but to no avail. Hopefully it would change now that Arrow is more popular, and also written about by MS itself.
- nomel 6y ago
  Until arrow has proper support for multidimensional arrays [1], its not really appropriate for many use cases. Not having first class support for multidimensional arrays, in a modern framework, really surprised me and my sensor data. [1] https://lists.apache.org/x/thread.html/9b142c1709aa37dc35f1ce8db4e1ced94fcc4cdd96cc72b5772b373b@%3Cdev.arrow.apache.org%3E https://lists.apache.org/x/thread.html/9b142c1709aa37dc35f1c...
  - lmeyerov 6y ago
    FWIW, you can support tensors etc. natively today by using Arrow's structs, such as `column i_am_packed: {x: int32, y: int32, z: int32, time: datetime64[ns], temp: int32}`. This will be a dense, packed representation for whatever fixed type the column has, including dynamically generated. It ends up being quite nice to have that passed around, esp. vs say protobufs. You wouldn't be sending mixed-length values, and ~all metrics systems I've worked with would be fine there. The nullables end up a win for those. However, you'd need to do a manual (likely zero-copy) casts between the struct-type to whatever tensor-type you're using. Massively popular ml systems like huggingface do this fine afaict for their Arrow-based tensor work. Likewise, as we do a lot of GPU stuff, what's additionally common is compacting the in-memory stuff as big memory blocks ('long recordbatches') instead of CPU-land's typically more fragmented ones, and that ends up making casts even easier. Annoying to have to add an explicit cast for some interop cases, but preserves end-to-end type safety & hasn't been a deal breaker for us. Having to write the cast being annoying/difficult, esp. for whoever does it first. More frustrating for us has been sparse data and compression controls, but most formats are even worse here..
    - deleted 6y ago
      [deleted]
  - wesm 6y ago
    Almost no database systems support multidimensional arrays. So they are not appropriate for many use cases? * BigQuery: no * Redshift: no * Spark SQL: no * Snowflake: no * Clickhouse: no * Dremio: no * Impala: no * Presto: no ... list continues We've invited developers to add the extension types for tensor data, but no one has contributed them yet. I'm not seeing a lot of tabular data with embedded tensors out in the wild.
- waynesonfire 6y ago
  Uhh.. maybe. It's a serde that's trying to be cross-language / platform. I guess it also offers some APIs to process the data so you can minimize serde operations. But, I dunno. It's been hard to understand the benefit of the libabry and the posts here don't help.
  - TuringTest 6y ago
    If it works as a universal intermediate exchange language, it could help standardize connections among disparate systems. When you have N systems, it takes N^2 translators to build direct connections to transfer data between them; but it only takes N translators if all them can talk the same exchange language.
    - waynesonfire 6y ago
      can you define what at translator is? I don't understand the complexity you're constructing. I have N systems and they talk protobuf. What's the problem?
      - TuringTest 6y ago
        By a translator, I mean a library that allows accessing data from different subsystems (either languages or OS processes). In this case, the advantages are that 1) Arrow is language agnostic, so it's likely that it can be used as a native library in your program and 2) it doesn't copy data to make it accessible to another process, so it saves a lot of marshalling / unmarshalling steps (assuming both sides use data in tabular format, which is typical of data analysis contexts).
      - deleted 6y ago
        [deleted]
  - mumblemumble 6y ago
    It's not just a serde. One of its key use cases is eliminating serde.
- rubicon33 6y ago
  Wait so you're telling me I could store data as a PDF file, and access it easily / quickly as SQL?
  - sethhochberg 6y ago
    If you found/wrote a adapter to translate your structured PDF into Arrow's format, yes - the idea is that you can wire up anything that can produce Arrow data to anything that can consume Arrow data.
    - staticassertion 6y ago
      I'm kinda confused. Is that not the case for literally everything? "You can send me data of format X, all I ask is that you be able to produce format X" ? I'm assuming that I'm missing something fwiw, not trying to diminish the value.
      - deleted 6y ago
        [deleted]
      - humbleMouse 6y ago
        The difference is that arrow’s mapping behind the scenes enables automatic translation to any implemented “plugin” that is on the user’s implementation of arrow. You can extend arrows format to make it automatically map to whatever you want, basically. And it’s all stored in memory - so much faster access to complex data relationships than anything that exists to my knowledge.
        staticautomatic 6y ago
        Could I write a plug-in that mapped to Cypher? I’ve got a graph use case in mind where I want to use RedisGraph but don’t feel comfy with Redis as a primary DB and would totally consider a columnar store as a primary if I didn’t have to serialize.
- shafiemukhre 6y ago
  True. Arrow is awesome and Dremio is using it as well as their built in memory. I tried it and it is increadibily fast. The future of data ecosystem is gonna be amazing
- breck 6y ago
  Arrow is definitely one of the top 10 new things I'm most excited about in the data science space, but not sure I'd call it the most important thing. ;) It is pretty awesome, however, particularly for folks like me that are often hopping between Python/R/Javascript. I've definitely got in on the roadmap for all my data science libraries. Btw, Arquero from that UW lab looks really neat as well, and is supporting Arrow out of the gate (https://github.com/uwdata/arquero https://github.com/uwdata/arquero).
  - georgewfraser 6y ago
    Much of the value of Arrow is in the things that will get built after Arrow is widely supported by data warehouses. Much of the data ecosystem we have today was designed to avoid the cost of moving data between systems. The whole Hadoop ecosystem is written in Java and shoehorned into map-reduce for this reason. Imagine if, for example, you could use Mathematica or R to analyze data in your Snowflake cluster, with no bottleneck reading data from the warehouse even for giant datasets. This is the future that’s going to be enabled by Arrow.
    - RainBoooow 6y ago
      Is it not what Presto (now Trino) is solving as well (among other things) ? Even though it is focused only on analytics and not on ML use cases.
- deleted 6y ago
  [deleted]
- yeshengm 6y ago
  Are these execution engines internally using Arrow columnar format or are they just exposing Arrow as a client wire format? AFAIK Spark and Presto does not use Arrow as execution columnar format, but just data sources/sinks.
  - groceryheist 6y ago
    You can configure Spark to use arrow for passing data between Java and Python via spark.sql.execution.arrow.pyspark.enabled but yes, Spark uses Java datatypes internally.
- otabdeveloper4 6y ago
  It's nice that the cyclical technology pendulum is finally swinging back from XML/JSON to files-with-C-structs again, but any serious analytics data store (e.g. Clickhouse) uses its own aggressively optimized storage format, so the process of loading from Arrow files to database won't go away.
- davidkell 6y ago
  Any Snowflake developers reading this - the current snowflake-connector-python is pinned to 0.17, almost 1 year out of date now. Would be great to get that bumped to a more recent version :-)
mushufasa 6y ago
Can someone ELI5 what problems are best solved by apache arrow?
- Diederich 6y ago
  Rather curious myself. https://en.wikipedia.org/wiki/Apache_Arrow https://en.wikipedia.org/wiki/Apache_Arrow was interesting, but I think many of us would benefit from a broader, problem focused description of Arrow from someone in the know.
- adgjlsfhk1 6y ago
  The big thing is that it is one of the first standardized, cross language binary data formats. CSV is an OK text format, but parsing it is really slow because of string escaping. The files it produces are also pretty big since it's text. Arrow is really fast to parse (up to 1000x faster than CSV), supports data compression, enough data-types to be useful, and deals with metadata well. The closest competitor is probably protobuf, but protobuf is a total pain to parse.
  - makapuf 6y ago
    Seems nice. How does it compare to hdf5?
    - BadInformatics 6y ago
      HDF5 is pretty terrible as a wire format, so it's not a 1-1 comparison to Arrow. Generally people are not going to be saving Arrow data to disk either (though you can with the IPC format), but serializing to a more compact representation like Parquet.
      - gbrits 6y ago
        As I understand, arrow is particularly interesting since it’s wire format can be immediately queried/operated on without deserialization. Would saving an Arrow-structure as parquet not defeat that purpose, since your would need the costly deserialization step again on read? Honest question
deleted 6y ago
[deleted]
jrevels 6y ago
Excited to see this release's official inclusion of the pure Julia Arrow implementation [1]! It's so cool to be able mmap Arrow memory and natively manipulate it from within Julia with virtually no performance overhead. Since the Julia compiler can specialize on the layout of Arrow-backed types at runtime (just as it can with any other type), the notion of needing to build/work with a separate "compiler for fast UDFs" is rendered obsolete. It feels pretty magical when two tools like this compose so well without either being designed with the other in mind - a testament to the thoughtful design of both :) mad props to Jacob Quinn for spearheading the effort to revive/restart Arrow.jl and get the package into this release. [1] https://github.com/JuliaData/Arrow.jl https://github.com/JuliaData/Arrow.jl
- StefanKarpinski 6y ago
  Especially impressive that the first official version ships with such broad feature coverage! Really great work by Jacob.
- andyferris 6y ago
  > a separate "compiler for fast UDFs" is rendered obsolete Agreed. I am excited too. Thanks Jacob!
mbyio 6y ago
I'm surprised they are still making breaking changes, and they plan to make more (they are already working on a 4.0).
- reilly3000 6y ago
  I didn't see any breaking changes in the release notes but I may have missed them. Maybe they don't use SemVer?
  - lidavidm 6y ago
    Arrow uses SemVer, but the library and the data format are versioned separately: https://arrow.apache.org/docs/format/Versioning.html https://arrow.apache.org/docs/format/Versioning.html
jayd16 6y ago
Can someone dig into the pros and cons of the columnar aspect of Arrow? To some degree there are many other data transfer formats but this one seems to promote its columnar orientation. Things like eg. protobuffers support hierarchical data which seems like a superset of columns. Is there a benefit to a column based format? Is it an enforced simplification to ensure greater compatibility or is there some other reason?
- tomnipotent 6y ago
  This is intended for analytical workloads where you're often doing things that can benefit from vectorization (like SIMD). It's much faster to SUM(X) when all values of X are neatly laid out in-memory. It also has the added benefit of eliminating serialization and deserialization of data between processes - a Python process can now write to memory which is read by a C++ process that's doing windowed aggregations, which are then written over the network to another Arrow compatible service that just copies the data as-is from the network into local memory and resumes working.
  - waynesonfire 6y ago
    > It also has the added benefit of eliminating serialization and deserialization of data between processes Is that accurate? It still has to deserialize from apache arrow format to whatever the cpu understands.
    - ianmcook 6y ago
      The Arrow Feather format is an on-disk representation of Arrow memory. To read a Feather file, Arrow just copies it byte for byte from disk into memory. Or Arrow can memory-map a Feather file so you can operate on it without reading the whole file into memory.
      - waynesonfire 6y ago
        That's exactly how I read every data format. The advantage you describe is in the operations that can performed against the data. It would be nice to see what this API looks like and how it compares to flatbuffers / pq. To help me understand this benefit, can you talk through what it's like to add 1 to each record and write it back to disk?
skratlo 6y ago
Yay, another ad-tech support engine from Apache, great
dang 6y ago
If curious see also 2020 https://news.ycombinator.com/item?id=23965209 https://news.ycombinator.com/item?id=23965209 2018 (a bit) https://news.ycombinator.com/item?id=17383881 https://news.ycombinator.com/item?id=17383881 2017 https://news.ycombinator.com/item?id=15335462 https://news.ycombinator.com/item?id=15335462 2017 https://news.ycombinator.com/item?id=15594542 https://news.ycombinator.com/item?id=15594542 rediscussed recently https://news.ycombinator.com/item?id=25258626 https://news.ycombinator.com/item?id=25258626 2016 https://news.ycombinator.com/item?id=11118274 https://news.ycombinator.com/item?id=11118274 Also: related from a couple weeks ago https://news.ycombinator.com/item?id=25824399 https://news.ycombinator.com/item?id=25824399 related from a few months ago https://news.ycombinator.com/item?id=24534274 https://news.ycombinator.com/item?id=24534274 related from 2019 https://news.ycombinator.com/item?id=21826974 https://news.ycombinator.com/item?id=21826974
liminal 6y ago
Would really love to see first class support for Javascript/Typescript for data visualization purposes. The columnar format would naturally lend itself to an Entity-Component style architecture with TypedArrays.
- nevi-me 6y ago
  Have you seen https://github.com/finos/perspective https://github.com/finos/perspective? Though they removed the JS library a few months ago in favour of a WASM build of the C++ library.
  - juntan 6y ago
    The JS library hasn't been removed - it's the main UI and API over the WASM library that allows for Perspective to be used in the browser: https://perspective.finos.org/ https://perspective.finos.org/
    - infinite8s 6y ago
      I think the GP meant the Typescript arrow library.
- BadInformatics 6y ago
  https://arrow.apache.org/docs/js/ https://arrow.apache.org/docs/js/ has existed for a while and uses typed arrays under the hood. It's a bit of a chunky dependency, but if you're at the point where that level of throughput is required bundle size is probably not a big deal.
  - liminal 6y ago
    I was mostly looking at this: https://arrow.apache.org/docs/status.html https://arrow.apache.org/docs/status.html
    - BadInformatics 6y ago
offtop5 6y ago
Does no one do load testing anymore, anyone got a working mirror
humbleMouse 6y ago
I worked at a large company a few years ago on a team implementing this. It’s super cool and works great. Definitely where the future is headed
anonyfox 6y ago
So if I understand this correctly from an application developers perspective: - for OLTP tasks, something row based like sqlite is great. Small to medium amounts of data mixed reading/writing with transactions - for OLAP tasks, arrow looks great. Big amounts of data, faster querying (datafusion) and more compact data files with parquet. Basically prevent the operational database from growing too large, offload older data to arrow/parquet. Did I get this correct? Additionally there seem to be further benefits like sharing arrow/parquet with other consumers. Sounds convincing, I just have two very specific questions: - if I load a ~2GB collection of items into arrow and query it with datafusion, how much slower will this perform in comparison to my current rust code that holds a large Vec in memory and „queries“ via iter/filter? - if I want to move data from sqlite to a more permanent parquet „Archive“ file, is there a better way than recreating the whole file or write additional files, like, appending? Really curious, could find no hints online so far to get an idea.
- humbleMouse 6y ago
  I think the best way to think about it is, you have teams where hadoop queries take 5+ hours to run. You port that same data into arrow, now that same query takes 30 seconds.
  - anonyfox 6y ago
    This is something I understood, but I ask specifically as a person at the intersection point between those worlds, owning the operational database and want to see if I can enhance systems for both, speedy operations _and_ seamless analytics using arrow.
kats 6y ago
I'm done with Hacker News, you guys just upvote marketing and politics.
- TheGuyWhoCodes 6y ago
  I genuinely would like to know what's the issue with a post about a major release of an innovative project. What's political about it? It's not like it's a new release of an enterprise/paid product based on an open source project.
- Uberzi 6y ago
  Yet your only contribution here is fully political...
mushufasa 6y ago
has anyone had success using arrow in js to feed tabular data to the frontend from the backend, such as a pandas data frame?
MR4D 6y ago
Uh... 404 error for the link....
atian 6y ago
Has anyone had success in getting the page to load? I'm on my desktop and can't see what's behind the link.
- TrispusAttucks 6y ago
  I'm on mobile and getting a white screen of death also.
peachy_no_pie 6y ago
How is Arrow when it comes to streaming workflows? Like, could Arrow replace Storm as analytics in a pipeline from Flume?
chenster 6y ago
Why no love for PHP I wonder? Don't see a supported library there.
- mumblemumble 6y ago
  If you're successfully doing data science or data engineering in PHP, you're already a god among informaticians, and don't need any extra help.
  - rscho 6y ago
    Hiya, a bit of OT: I saw your comment about type systems in data science the other day (https://news.ycombinator.com/item?id=25923839 https://news.ycombinator.com/item?id=25923839). From what I understood, it seems you want a contract system, wouldn't you think? The reason I'm asking is that I'm fishing for opinions on building data science infra in Racket (and saw your deleted comment in https://news.ycombinator.com/item?id=26008869 https://news.ycombinator.com/item?id=26008869 so thought you'd perhaps be interested), and Racket (and R) dataframes happen to support contracts on their columns.
- zackmorris 6y ago
  I notice PHP compatibility conspicuously absent from so many libraries. Which is kind of amazing to me, since I "think" in PHP and MATLAB in a declarative and data-driven way. I do everything in sync blocking functional style and mostly just pipe data around. So I've found that the runners up (like Javascript, Python and Ruby) mostly just get in the way and force me to adopt their style. They're all roughly equivalent in power and expressivity, but nothing lets me go between a spreadsheet and the shell quite as easily as PHP.
gnefgnil 6y ago
hi
archagon 6y ago
For use as a file format, where one priority is to compress columnar data as well as possible, the practical difference between Arrow (via Feather?), Parquet, and ORC is still somewhat vague to me. During my last investigation, I got the impression that Arrow worked great as a standard, interoperable, in-memory columnar format, but didn't compress nearly as well as ORC or Parquet due to lack of RLE and other compression schemes (other than dictionary). Is this still the case? Is there a world where Arrow completely supplants Parquet and/or ORC? EDIT: Just found https://wesmckinney.com/blog/arrow-columnar-abadi https://wesmckinney.com/blog/arrow-columnar-abadi, which helps answer this question.
ryanianian 6y ago
This link is a 404. Perhaps they weren't intending this post to be public yet? At any rate, archive.org managed to grab it https://web.archive.org/web/20210203194945/https://arrow.apache.org/blog/2021/01/25/3.0.0-release/ https://web.archive.org/web/20210203194945/https://arrow.apa...
- ianmcook 6y ago
  Thanks for the heads up. The post is intended to be up but there's an intermittent error happening. It's been reported to the Apache infrastructure team.
bravura 6y ago
I got interested in Arrow recently after reading this blog post showing that Arrow (and Ray) are much faster than Pickle: https://rise.cs.berkeley.edu/blog/fast-python-serialization-ray-apache-arrow/ https://rise.cs.berkeley.edu/blog/fast-python-serialization-... I have a question about whether it would fit this use-case: * I need a SUPER fast KV-store. * I'm on a single machine. * Keys are 10-bytes if you compress (or strings with 32 characters if you don't), unfortunately I can't store it as an 8-byte int. sqlite said it supports arbitrary precision numerics, but then I got burned finding out that casts integers to arbitrary precision floats and only keeps the first 14 digits of precision :\ * Values are 4-byte ints. Maybe 3 4-byte ints. * I have maybe 10B - 100B rows. * I need super fast lookup and depending upon my machine can't always cache this in memory, might need to work from disk. Would arrow be useful for this? Currently just using sqlite.
- tyingq 6y ago
  "much faster than Pickle" That isn't saying much, though. https://www.benfrederickson.com/images/python-serialization/speed.png https://www.benfrederickson.com/images/python-serialization/...
  - jzer0cool 6y ago
    Unrelated: What is this *pickle* term origin? Python errors sometimes generate some "Could not pickle" errors and not sure what it tries to convey ...
    - tyingq 6y ago
      Pickling a cucumber stores and preserves it for later use, so the term is used for serialization. Still an odd analogy though, since you can't unpickle a pickle. I've heard freeze/thaw which is at least a little better.
      - jdub 6y ago
        It's Python. So it's Monty Python.
supunkk 6y ago
Cudf and Cylon are two execution engines natively supporting Arrow format https://github.com/rapidsai/cudf https://github.com/rapidsai/cudf https://github.com/cylondata/cylon https://github.com/cylondata/cylon
sriku 6y ago
Related - recent Apache Arrow support in Julia announcement - https://julialang.org/blog/2021/01/arrow/ https://julialang.org/blog/2021/01/arrow/

Apache Arrow 3.0

561 points by kylebarron 6y ago

Thaxll 6y ago

Last time I worked in ETL was with Hadoop, looks like a lot happened.

macksd 6y ago
There's actually a lot of overlap between Hadoop and Arrow's origins - a lot of the projects that integrated early and the founding contributors had been in the larger Hadoop ecosystem. It's a very good sign IMO that you can hardly tell anymore - very diverse community and wide adoption!

georgewfraser 6y ago

Arrow is the most important thing happening in the data ecosystem right now. It's going to allow you to run your choice of execution engine, on top of your choice of data store, as though they are designed to work together. It will mostly be invisible to users, the key thing that needs to happen is that all the producers and consumers of batch data need to adopt Arrow as the common interchange format. BigQuery recently implemented the storage API, which allows you to read BQ tables, in parallel, in Arrow format: https://cloud.google.com/bigquery/docs/reference/storage https://cloud.google.com/bigquery/docs/reference/storage Snowflake has adopted Arrow as the in-memory format for their JDBC driver, though to my knowledge there is still no way to access data in parallel from Snowflake, other than to export to S3. As Arrow spreads across the ecosystem, users are going to start discovering that they can store data in one system and query it in another, at full speed, and it's going to be amazing.

tristanz 6y ago
Agreed! Thank you Arrow community for such a great project. It's a long road but it opens up tremendous potential for efficient data systems that talk to one another. The future looks bright with so many vendors backing Arrow independently and Wes McKinney founding Ursa Labs and now Ursa Computing. https://ursalabs.org/blog/ursa-computing/ https://ursalabs.org/blog/ursa-computing/
wesm 6y ago
Microsoft is also on top of this with their Magpie project http://cidrdb.org/cidr2021/papers/cidr2021_paper08.pdf http://cidrdb.org/cidr2021/papers/cidr2021_paper08.pdf "A common, efficient serialized and wire format across data engines is a transformational development. Many previous systems and approaches (e.g., [26, 36, 38, 51]) have observed the prohibitive cost of data conversion and transfer, precluding optimizers from exploiting inter-DBMS performance advantages. By contrast, inmemory data transfer cost between a pair of Arrow-supporting systems is effectively zero. Many major, modern DBMSs (e.g., Spark, Kudu, AWS Data Wrangler, SciDB, TileDB) and data-processing frameworks (e.g., Pandas, NumPy, Dask) have or are in the process of incorporating support for Arrow and ArrowFlight. Exploiting this is key for Magpie, which is thereby free to combine data from different sources and cache intermediate data and results, without needing to consider data conversion overhead."
- data_ders 6y ago
  way cool! Is magpie end-user facing anywhere yet? We were using the azureml-dataprep library for a while which seems similar but not all of magpie
- polskibus 6y ago
  I wish MS put in some resources behind Arrow in .NET. I tried raising some remarks about it on dotnet repos (esp. within ML.NET), but to no avail. Hopefully it would change now that Arrow is more popular, and also written about by MS itself.
nomel 6y ago
Until arrow has proper support for multidimensional arrays [1], its not really appropriate for many use cases. Not having first class support for multidimensional arrays, in a modern framework, really surprised me and my sensor data. [1] https://lists.apache.org/x/thread.html/9b142c1709aa37dc35f1ce8db4e1ced94fcc4cdd96cc72b5772b373b@%3Cdev.arrow.apache.org%3E https://lists.apache.org/x/thread.html/9b142c1709aa37dc35f1c...
- lmeyerov 6y ago
  FWIW, you can support tensors etc. natively today by using Arrow's structs, such as `column i_am_packed: {x: int32, y: int32, z: int32, time: datetime64[ns], temp: int32}`. This will be a dense, packed representation for whatever fixed type the column has, including dynamically generated. It ends up being quite nice to have that passed around, esp. vs say protobufs. You wouldn't be sending mixed-length values, and ~all metrics systems I've worked with would be fine there. The nullables end up a win for those. However, you'd need to do a manual (likely zero-copy) casts between the struct-type to whatever tensor-type you're using. Massively popular ml systems like huggingface do this fine afaict for their Arrow-based tensor work. Likewise, as we do a lot of GPU stuff, what's additionally common is compacting the in-memory stuff as big memory blocks ('long recordbatches') instead of CPU-land's typically more fragmented ones, and that ends up making casts even easier. Annoying to have to add an explicit cast for some interop cases, but preserves end-to-end type safety & hasn't been a deal breaker for us. Having to write the cast being annoying/difficult, esp. for whoever does it first. More frustrating for us has been sparse data and compression controls, but most formats are even worse here..
  - deleted 6y ago
    [deleted]
- wesm 6y ago
  Almost no database systems support multidimensional arrays. So they are not appropriate for many use cases? * BigQuery: no * Redshift: no * Spark SQL: no * Snowflake: no * Clickhouse: no * Dremio: no * Impala: no * Presto: no ... list continues We've invited developers to add the extension types for tensor data, but no one has contributed them yet. I'm not seeing a lot of tabular data with embedded tensors out in the wild.
waynesonfire 6y ago
Uhh.. maybe. It's a serde that's trying to be cross-language / platform. I guess it also offers some APIs to process the data so you can minimize serde operations. But, I dunno. It's been hard to understand the benefit of the libabry and the posts here don't help.
- TuringTest 6y ago
  If it works as a universal intermediate exchange language, it could help standardize connections among disparate systems. When you have N systems, it takes N^2 translators to build direct connections to transfer data between them; but it only takes N translators if all them can talk the same exchange language.
  - waynesonfire 6y ago
    can you define what at translator is? I don't understand the complexity you're constructing. I have N systems and they talk protobuf. What's the problem?
    - TuringTest 6y ago
      By a translator, I mean a library that allows accessing data from different subsystems (either languages or OS processes). In this case, the advantages are that 1) Arrow is language agnostic, so it's likely that it can be used as a native library in your program and 2) it doesn't copy data to make it accessible to another process, so it saves a lot of marshalling / unmarshalling steps (assuming both sides use data in tabular format, which is typical of data analysis contexts).
    - deleted 6y ago
      [deleted]
- mumblemumble 6y ago
  It's not just a serde. One of its key use cases is eliminating serde.
rubicon33 6y ago
Wait so you're telling me I could store data as a PDF file, and access it easily / quickly as SQL?
- sethhochberg 6y ago
  If you found/wrote a adapter to translate your structured PDF into Arrow's format, yes - the idea is that you can wire up anything that can produce Arrow data to anything that can consume Arrow data.
  - staticassertion 6y ago
    I'm kinda confused. Is that not the case for literally everything? "You can send me data of format X, all I ask is that you be able to produce format X" ? I'm assuming that I'm missing something fwiw, not trying to diminish the value.
    - deleted 6y ago
      [deleted]
    - humbleMouse 6y ago
      The difference is that arrow’s mapping behind the scenes enables automatic translation to any implemented “plugin” that is on the user’s implementation of arrow. You can extend arrows format to make it automatically map to whatever you want, basically. And it’s all stored in memory - so much faster access to complex data relationships than anything that exists to my knowledge.
      - staticautomatic 6y ago
        Could I write a plug-in that mapped to Cypher? I’ve got a graph use case in mind where I want to use RedisGraph but don’t feel comfy with Redis as a primary DB and would totally consider a columnar store as a primary if I didn’t have to serialize.
shafiemukhre 6y ago
True. Arrow is awesome and Dremio is using it as well as their built in memory. I tried it and it is increadibily fast. The future of data ecosystem is gonna be amazing
breck 6y ago
Arrow is definitely one of the top 10 new things I'm most excited about in the data science space, but not sure I'd call it the most important thing. ;) It is pretty awesome, however, particularly for folks like me that are often hopping between Python/R/Javascript. I've definitely got in on the roadmap for all my data science libraries. Btw, Arquero from that UW lab looks really neat as well, and is supporting Arrow out of the gate (https://github.com/uwdata/arquero https://github.com/uwdata/arquero).
- georgewfraser 6y ago
  Much of the value of Arrow is in the things that will get built after Arrow is widely supported by data warehouses. Much of the data ecosystem we have today was designed to avoid the cost of moving data between systems. The whole Hadoop ecosystem is written in Java and shoehorned into map-reduce for this reason. Imagine if, for example, you could use Mathematica or R to analyze data in your Snowflake cluster, with no bottleneck reading data from the warehouse even for giant datasets. This is the future that’s going to be enabled by Arrow.
  - RainBoooow 6y ago
    Is it not what Presto (now Trino) is solving as well (among other things) ? Even though it is focused only on analytics and not on ML use cases.
deleted 6y ago
[deleted]
yeshengm 6y ago
Are these execution engines internally using Arrow columnar format or are they just exposing Arrow as a client wire format? AFAIK Spark and Presto does not use Arrow as execution columnar format, but just data sources/sinks.
- groceryheist 6y ago
  You can configure Spark to use arrow for passing data between Java and Python via spark.sql.execution.arrow.pyspark.enabled but yes, Spark uses Java datatypes internally.
otabdeveloper4 6y ago
It's nice that the cyclical technology pendulum is finally swinging back from XML/JSON to files-with-C-structs again, but any serious analytics data store (e.g. Clickhouse) uses its own aggressively optimized storage format, so the process of loading from Arrow files to database won't go away.
davidkell 6y ago
Any Snowflake developers reading this - the current snowflake-connector-python is pinned to 0.17, almost 1 year out of date now. Would be great to get that bumped to a more recent version :-)

mushufasa 6y ago

Can someone ELI5 what problems are best solved by apache arrow?

Diederich 6y ago
Rather curious myself. https://en.wikipedia.org/wiki/Apache_Arrow https://en.wikipedia.org/wiki/Apache_Arrow was interesting, but I think many of us would benefit from a broader, problem focused description of Arrow from someone in the know.
adgjlsfhk1 6y ago
The big thing is that it is one of the first standardized, cross language binary data formats. CSV is an OK text format, but parsing it is really slow because of string escaping. The files it produces are also pretty big since it's text. Arrow is really fast to parse (up to 1000x faster than CSV), supports data compression, enough data-types to be useful, and deals with metadata well. The closest competitor is probably protobuf, but protobuf is a total pain to parse.
- makapuf 6y ago
  Seems nice. How does it compare to hdf5?
  - BadInformatics 6y ago
    HDF5 is pretty terrible as a wire format, so it's not a 1-1 comparison to Arrow. Generally people are not going to be saving Arrow data to disk either (though you can with the IPC format), but serializing to a more compact representation like Parquet.
    - gbrits 6y ago
      As I understand, arrow is particularly interesting since it’s wire format can be immediately queried/operated on without deserialization. Would saving an Arrow-structure as parquet not defeat that purpose, since your would need the costly deserialization step again on read? Honest question

deleted 6y ago

[deleted]

jrevels 6y ago

Excited to see this release's official inclusion of the pure Julia Arrow implementation [1]! It's so cool to be able mmap Arrow memory and natively manipulate it from within Julia with virtually no performance overhead. Since the Julia compiler can specialize on the layout of Arrow-backed types at runtime (just as it can with any other type), the notion of needing to build/work with a separate "compiler for fast UDFs" is rendered obsolete. It feels pretty magical when two tools like this compose so well without either being designed with the other in mind - a testament to the thoughtful design of both :) mad props to Jacob Quinn for spearheading the effort to revive/restart Arrow.jl and get the package into this release. [1] https://github.com/JuliaData/Arrow.jl https://github.com/JuliaData/Arrow.jl

StefanKarpinski 6y ago
Especially impressive that the first official version ships with such broad feature coverage! Really great work by Jacob.
andyferris 6y ago
> a separate "compiler for fast UDFs" is rendered obsolete Agreed. I am excited too. Thanks Jacob!

mbyio 6y ago

I'm surprised they are still making breaking changes, and they plan to make more (they are already working on a 4.0).

reilly3000 6y ago
I didn't see any breaking changes in the release notes but I may have missed them. Maybe they don't use SemVer?
- lidavidm 6y ago
  Arrow uses SemVer, but the library and the data format are versioned separately: https://arrow.apache.org/docs/format/Versioning.html https://arrow.apache.org/docs/format/Versioning.html

jayd16 6y ago

Can someone dig into the pros and cons of the columnar aspect of Arrow? To some degree there are many other data transfer formats but this one seems to promote its columnar orientation. Things like eg. protobuffers support hierarchical data which seems like a superset of columns. Is there a benefit to a column based format? Is it an enforced simplification to ensure greater compatibility or is there some other reason?

tomnipotent 6y ago
This is intended for analytical workloads where you're often doing things that can benefit from vectorization (like SIMD). It's much faster to SUM(X) when all values of X are neatly laid out in-memory. It also has the added benefit of eliminating serialization and deserialization of data between processes - a Python process can now write to memory which is read by a C++ process that's doing windowed aggregations, which are then written over the network to another Arrow compatible service that just copies the data as-is from the network into local memory and resumes working.
- waynesonfire 6y ago
  > It also has the added benefit of eliminating serialization and deserialization of data between processes Is that accurate? It still has to deserialize from apache arrow format to whatever the cpu understands.
  - ianmcook 6y ago
    The Arrow Feather format is an on-disk representation of Arrow memory. To read a Feather file, Arrow just copies it byte for byte from disk into memory. Or Arrow can memory-map a Feather file so you can operate on it without reading the whole file into memory.
    - waynesonfire 6y ago
      That's exactly how I read every data format. The advantage you describe is in the operations that can performed against the data. It would be nice to see what this API looks like and how it compares to flatbuffers / pq. To help me understand this benefit, can you talk through what it's like to add 1 to each record and write it back to disk?

skratlo 6y ago

Yay, another ad-tech support engine from Apache, great

dang 6y ago

If curious see also 2020 https://news.ycombinator.com/item?id=23965209 https://news.ycombinator.com/item?id=23965209 2018 (a bit) https://news.ycombinator.com/item?id=17383881 https://news.ycombinator.com/item?id=17383881 2017 https://news.ycombinator.com/item?id=15335462 https://news.ycombinator.com/item?id=15335462 2017 https://news.ycombinator.com/item?id=15594542 https://news.ycombinator.com/item?id=15594542 rediscussed recently https://news.ycombinator.com/item?id=25258626 https://news.ycombinator.com/item?id=25258626 2016 https://news.ycombinator.com/item?id=11118274 https://news.ycombinator.com/item?id=11118274 Also: related from a couple weeks ago https://news.ycombinator.com/item?id=25824399 https://news.ycombinator.com/item?id=25824399 related from a few months ago https://news.ycombinator.com/item?id=24534274 https://news.ycombinator.com/item?id=24534274 related from 2019 https://news.ycombinator.com/item?id=21826974 https://news.ycombinator.com/item?id=21826974

liminal 6y ago

Would really love to see first class support for Javascript/Typescript for data visualization purposes. The columnar format would naturally lend itself to an Entity-Component style architecture with TypedArrays.

nevi-me 6y ago
Have you seen https://github.com/finos/perspective https://github.com/finos/perspective? Though they removed the JS library a few months ago in favour of a WASM build of the C++ library.
- juntan 6y ago
  The JS library hasn't been removed - it's the main UI and API over the WASM library that allows for Perspective to be used in the browser: https://perspective.finos.org/ https://perspective.finos.org/
  - infinite8s 6y ago
    I think the GP meant the Typescript arrow library.
BadInformatics 6y ago
https://arrow.apache.org/docs/js/ https://arrow.apache.org/docs/js/ has existed for a while and uses typed arrays under the hood. It's a bit of a chunky dependency, but if you're at the point where that level of throughput is required bundle size is probably not a big deal.
- liminal 6y ago
  I was mostly looking at this: https://arrow.apache.org/docs/status.html https://arrow.apache.org/docs/status.html
  - BadInformatics 6y ago

offtop5 6y ago

Does no one do load testing anymore, anyone got a working mirror

humbleMouse 6y ago

I worked at a large company a few years ago on a team implementing this. It’s super cool and works great. Definitely where the future is headed

anonyfox 6y ago

So if I understand this correctly from an application developers perspective: - for OLTP tasks, something row based like sqlite is great. Small to medium amounts of data mixed reading/writing with transactions - for OLAP tasks, arrow looks great. Big amounts of data, faster querying (datafusion) and more compact data files with parquet. Basically prevent the operational database from growing too large, offload older data to arrow/parquet. Did I get this correct? Additionally there seem to be further benefits like sharing arrow/parquet with other consumers. Sounds convincing, I just have two very specific questions: - if I load a ~2GB collection of items into arrow and query it with datafusion, how much slower will this perform in comparison to my current rust code that holds a large Vec in memory and „queries“ via iter/filter? - if I want to move data from sqlite to a more permanent parquet „Archive“ file, is there a better way than recreating the whole file or write additional files, like, appending? Really curious, could find no hints online so far to get an idea.

humbleMouse 6y ago
I think the best way to think about it is, you have teams where hadoop queries take 5+ hours to run. You port that same data into arrow, now that same query takes 30 seconds.
- anonyfox 6y ago
  This is something I understood, but I ask specifically as a person at the intersection point between those worlds, owning the operational database and want to see if I can enhance systems for both, speedy operations _and_ seamless analytics using arrow.

kats 6y ago

I'm done with Hacker News, you guys just upvote marketing and politics.

TheGuyWhoCodes 6y ago
I genuinely would like to know what's the issue with a post about a major release of an innovative project. What's political about it? It's not like it's a new release of an enterprise/paid product based on an open source project.
Uberzi 6y ago
Yet your only contribution here is fully political...

mushufasa 6y ago

has anyone had success using arrow in js to feed tabular data to the frontend from the backend, such as a pandas data frame?

MR4D 6y ago

Uh... 404 error for the link....

atian 6y ago

Has anyone had success in getting the page to load? I'm on my desktop and can't see what's behind the link.

TrispusAttucks 6y ago
I'm on mobile and getting a white screen of death also.

peachy_no_pie 6y ago

How is Arrow when it comes to streaming workflows? Like, could Arrow replace Storm as analytics in a pipeline from Flume?

chenster 6y ago

Why no love for PHP I wonder? Don't see a supported library there.

mumblemumble 6y ago
If you're successfully doing data science or data engineering in PHP, you're already a god among informaticians, and don't need any extra help.
- rscho 6y ago
  Hiya, a bit of OT: I saw your comment about type systems in data science the other day (https://news.ycombinator.com/item?id=25923839 https://news.ycombinator.com/item?id=25923839). From what I understood, it seems you want a contract system, wouldn't you think? The reason I'm asking is that I'm fishing for opinions on building data science infra in Racket (and saw your deleted comment in https://news.ycombinator.com/item?id=26008869 https://news.ycombinator.com/item?id=26008869 so thought you'd perhaps be interested), and Racket (and R) dataframes happen to support contracts on their columns.
zackmorris 6y ago
I notice PHP compatibility conspicuously absent from so many libraries. Which is kind of amazing to me, since I "think" in PHP and MATLAB in a declarative and data-driven way. I do everything in sync blocking functional style and mostly just pipe data around. So I've found that the runners up (like Javascript, Python and Ruby) mostly just get in the way and force me to adopt their style. They're all roughly equivalent in power and expressivity, but nothing lets me go between a spreadsheet and the shell quite as easily as PHP.

gnefgnil 6y ago

archagon 6y ago

For use as a file format, where one priority is to compress columnar data as well as possible, the practical difference between Arrow (via Feather?), Parquet, and ORC is still somewhat vague to me. During my last investigation, I got the impression that Arrow worked great as a standard, interoperable, in-memory columnar format, but didn't compress nearly as well as ORC or Parquet due to lack of RLE and other compression schemes (other than dictionary). Is this still the case? Is there a world where Arrow completely supplants Parquet and/or ORC? EDIT: Just found https://wesmckinney.com/blog/arrow-columnar-abadi https://wesmckinney.com/blog/arrow-columnar-abadi, which helps answer this question.

ryanianian 6y ago

This link is a 404. Perhaps they weren't intending this post to be public yet? At any rate, archive.org managed to grab it https://web.archive.org/web/20210203194945/https://arrow.apache.org/blog/2021/01/25/3.0.0-release/ https://web.archive.org/web/20210203194945/https://arrow.apa...

ianmcook 6y ago
Thanks for the heads up. The post is intended to be up but there's an intermittent error happening. It's been reported to the Apache infrastructure team.

bravura 6y ago

I got interested in Arrow recently after reading this blog post showing that Arrow (and Ray) are much faster than Pickle: https://rise.cs.berkeley.edu/blog/fast-python-serialization-ray-apache-arrow/ https://rise.cs.berkeley.edu/blog/fast-python-serialization-... I have a question about whether it would fit this use-case: * I need a SUPER fast KV-store. * I'm on a single machine. * Keys are 10-bytes if you compress (or strings with 32 characters if you don't), unfortunately I can't store it as an 8-byte int. sqlite said it supports arbitrary precision numerics, but then I got burned finding out that casts integers to arbitrary precision floats and only keeps the first 14 digits of precision :\ * Values are 4-byte ints. Maybe 3 4-byte ints. * I have maybe 10B - 100B rows. * I need super fast lookup and depending upon my machine can't always cache this in memory, might need to work from disk. Would arrow be useful for this? Currently just using sqlite.

tyingq 6y ago
"much faster than Pickle" That isn't saying much, though. https://www.benfrederickson.com/images/python-serialization/speed.png https://www.benfrederickson.com/images/python-serialization/...
- jzer0cool 6y ago
  Unrelated: What is this *pickle* term origin? Python errors sometimes generate some "Could not pickle" errors and not sure what it tries to convey ...
  - tyingq 6y ago
    Pickling a cucumber stores and preserves it for later use, so the term is used for serialization. Still an odd analogy though, since you can't unpickle a pickle. I've heard freeze/thaw which is at least a little better.
    - jdub 6y ago
      It's Python. So it's Monty Python.

supunkk 6y ago

Cudf and Cylon are two execution engines natively supporting Arrow format https://github.com/rapidsai/cudf https://github.com/rapidsai/cudf https://github.com/cylondata/cylon https://github.com/cylondata/cylon

sriku 6y ago

Related - recent Apache Arrow support in Julia announcement - https://julialang.org/blog/2021/01/arrow/ https://julialang.org/blog/2021/01/arrow/