16 ms·
Apologies for being a debbie downer, but it looks like the authors looked into Protocol Buffers and Flatbuffers. Did they consider CapnProto? In my limited ex
by subhobroto 9y ago
Apologies for being a debbie downer, but it looks like the authors looked into Protocol Buffers and Flatbuffers.
Did they consider CapnProto?
In my limited experience with efficient serdes for analytics, CapnProto would meet 1 - 4.
Arrow is an in-memory format and CapnProto would afford the ondisc (memmap) heavy lifting with (very) little overhead. My
CapnProto/LMDB datastructures allow me to write better than C++ code in Python with comparable runtime performance.
The only issue I have with CapnProto is getting it to run on Windows 7 x64, but it works out of the box on Linux.
- pcmoritz 9y agoAuthor here. From our perspective, CapnProto has similar characteristics as Flatbuffers and the reasons to prefer Arrow over it are the same: We would need to develop a mapping from Python types to CapnProto from scratch and Arrow has many facilities that are useful for us already built (Tensor serialization, code to deal with some Python types like datetimes, zero copy DataFrames, a larger ecosystem for interfacing with other formats like parquet, reading from HDFS, etc.). And it is designed for Big Data. So Arrow was a very natural choice (and it also supports Windows). Wes is doing some amazing work here!
- elcritch 9y agoIt looks like Arrow utilizes FlatBuffers internally [1]. Seems like using the Arrow project builds a lot of scaffolding that'd otherwise need to be built for this particular use case. 1: https://arrow.apache.org/docs/metadata.html https://arrow.apache.org/docs/metadata.html