4 ms·
One major use-case is exchanging data between processes (in different languages) using IPC, avoiding the overhead of serialisation. Best example is probably py
by yokaze 6y ago
One major use-case is exchanging data between processes (in different languages) using IPC, avoiding the overhead of serialisation.
Best example is probably pyspark.
- lenkite 6y agoCan't one do this using protobuf, flatbuffers, etc ? What advantage does arrow offer here ?
- wenc 6y agoThat's precisely it -- you don't have to use protobuf etc. because you don't have to serialize/deserialize at all. Languages that use Arrow can interact with the source data directly in-memory, in-process. No need to move any data around. The fastest serde operation is one that you don't have to do at all.
- lenkite 6y agoThis is already possible with flatbuffers.
- wenc 6y agoFrom the Arrow FAQ How does Arrow relate to Flatbuffers? Flatbuffers is a low-level building block for binary data serialization. It is not adapted to the representation of large, structured, homogenous data, and does not sit at the right abstraction layer for data analysis tasks. Arrow is a data layer aimed directly at the needs of data analysis, providing a comprehensive collection of data types required to analytics, built-in support for “null” values (representing missing data), and an expanding toolbox of I/O and computing facilities. The Arrow file format does use Flatbuffers under the hood to serialize schemas and other metadata needed to implement the Arrow binary IPC protocol, but the Arrow data format uses its own representation for optimal access and computation.
- FridgeSeal 6y agoZero copy serialisation and performance benefits over doing “row-at-a-time” like you would with protobuf.
- lenkite 6y agoZero copy serialisation is offered by flatbuffers - and one additionally has the advantage of IDL as a contract between different technology platforms.