4 ms·
Not GP post, but it might have been better stated as 'eliminating serde overhead'. Arrow's RPC serialization [1] is basically Protobuf, with a whole lot of hack
by kristjansson 6y ago
Not GP post, but it might have been better stated as 'eliminating serde overhead'. Arrow's RPC serialization [1] is basically Protobuf, with a whole lot of hacks to eliminate copies on both ends of the wire. So it's still 'serde', but markedly more efficient for large blocks of tabular-ish data.
[1]: https://arrow.apache.org/docs/format/Flight.html https://arrow.apache.org/docs/format/Flight.html
- wesm 6y ago> Arrow's serialization is Protobuf Incorrect. Only Arrow Flight embeds the Arrow wire format in a Protocol Buffer, but the Arrow protocol itself does not use Protobuf.
- kristjansson 6y agoApologies, off base there. Edited with a pointer to Flight :)