4 ms·

One of the interesting components of Polars that I've been watching is the use of the Apache Arrow memory format, which is a standard layout for data in memory

One of the interesting components of Polars that I've been watching is the use of the Apache Arrow memory format, which is a standard layout for data in memory that enables processing (querying, iterating, calculating, etc) in a language agnostic way, in particular without having to copy/convert it into the local object format first. This enables cross-language data access by mmaping or transferring a single buffer, with zero [de]serialization overhead. Something genius and obvious in hindsight. For some history, there has been a bit of contention between the official arrow-rs implementation and the arrow2 implementation created by the polars team which includes some extra features that they find important. I think the current status is that everyone agrees that having two crates that implement the same standard is not ideal, and they are working to port any necessary features to the arrow-rs crate and plan on eventually switching to it and deprecating arrow2, but it will take some time to get there. https://github.com/apache/arrow-rs/issues/1176 https://github.com/apache/arrow-rs/issues/1176 https://github.com/jorgecarleitao/arrow2/pull/1476 https://github.com/jorgecarleitao/arrow2/pull/1476

dikei 3y ago
That's not how I read it. I think the arrow-rs team wants a merger, and the arrow2/polars team does not oppose as long as they can keep using the arrow2 API. The responsibility to port the changes over would lie with the arrow-rs team, however, as the arrow2/polars team won't spend any effort on it.
- infogulch 3y ago
  Note that there are about a dozen relevant threads across various repos, which I omitted for obvious reasons. Yes, arrow-rs contributors have driven the effort and made code changes in the arrow-rs repo, but it has absolutely been a collaboration; besides creating new useful apis, arrow2 devs have spent considerable effort to communicate and justify the api differences and the benefits thereof. And switching polars from arrow2 to arrow-rs in the end will not be zero effort. I used neutral language to describe the situation with the explicit intention to avoid spawning unnecessarily divisive commentary here. I don't see how your reading is helpful to understand the broad context.

dikei 3y ago

That's not how I read it. I think the arrow-rs team wants a merger, and the arrow2/polars team does not oppose as long as they can keep using the arrow2 API. The responsibility to port the changes over would lie with the arrow-rs team, however, as the arrow2/polars team won't spend any effort on it.

infogulch 3y ago
Note that there are about a dozen relevant threads across various repos, which I omitted for obvious reasons. Yes, arrow-rs contributors have driven the effort and made code changes in the arrow-rs repo, but it has absolutely been a collaboration; besides creating new useful apis, arrow2 devs have spent considerable effort to communicate and justify the api differences and the benefits thereof. And switching polars from arrow2 to arrow-rs in the end will not be zero effort. I used neutral language to describe the situation with the explicit intention to avoid spawning unnecessarily divisive commentary here. I don't see how your reading is helpful to understand the broad context.