3 ms·
That's precisely it -- you don't have to use protobuf etc. because you don't have to serialize/deserialize at all. Languages that use Arrow can interact with t
by wenc 6y ago
That's precisely it -- you don't have to use protobuf etc. because you don't have to serialize/deserialize at all.
Languages that use Arrow can interact with the source data directly in-memory, in-process. No need to move any data around. The fastest serde operation is one that you don't have to do at all.
- lenkite 6y agoThis is already possible with flatbuffers.
- wenc 6y agoFrom the Arrow FAQ How does Arrow relate to Flatbuffers? Flatbuffers is a low-level building block for binary data serialization. It is not adapted to the representation of large, structured, homogenous data, and does not sit at the right abstraction layer for data analysis tasks. Arrow is a data layer aimed directly at the needs of data analysis, providing a comprehensive collection of data types required to analytics, built-in support for “null” values (representing missing data), and an expanding toolbox of I/O and computing facilities. The Arrow file format does use Flatbuffers under the hood to serialize schemas and other metadata needed to implement the Arrow binary IPC protocol, but the Arrow data format uses its own representation for optimal access and computation.