5 ms·
Arrow is columnar storage, nothing relevant here. And capnproto is basically another version of pb... (see discussion above)
by aldanor 2y ago
Arrow is columnar storage, nothing relevant here.
And capnproto is basically another version of pb... (see discussion above)
- adolph 2y agoLook again at Arrow. Part of the concept is the wire format is the same as the in memory format—no de/serialization. Arrow isn’t intended for storage; they point to parquet for that. Apache Arrow defines an inter-process communication (IPC) mechanism to transfer a collection of Arrow columnar arrays (called a “record batch”). It can be used synchronously between processes using the Arrow “stream format”, or asynchronously by first persisting data on storage using the Arrow “file format”. The Arrow IPC mechanism is based on the Arrow in-memory format, such that there is no translation necessary between the on-disk representation and the in-memory representation. [0] The Arrow spec aligns columnar data in memory to minimize cache misses and take advantage of the latest SIMD (Single input multiple data) and GPU operations on modern processors. [1] 0. https://arrow.apache.org/faq/ https://arrow.apache.org/faq/ 1. https://arrow.apache.org/docs/js/ https://arrow.apache.org/docs/js/