4 ms·
I used Apache Arrow to store and process a relatively large amount of data, 2-3B records with ~10 fields each (still fits on one machine, but it's large enough
by thamer 6y ago
I used Apache Arrow to store and process a relatively large amount of data, 2-3B records with ~10 fields each (still fits on one machine, but it's large enough that I have to think about it a little bit).
I was originally using a simple binary format for fast decoding, and switched to Arrow to be able to select only a few columns at a time. I was impressed by the speed gains, and the size benefits of storing data in column order.
The one thing I wish Arrow had was a way to attach some metadata to its files (if there is, I haven't found it). I originally tried to write a small header before starting to write the Arrow data, but that made it impossible to read back the data: as far as I can tell Arrow looks at the whole file and stores its column definition at the end of the file and computes data offsets based on the start of the file, meaning that there's nowhere left for me to store anything.
It's still a very promising library and I'll definitely be checking out the 1.0 release.
- lmeyerov 6y agopyarrow.metadata in the header is for stuffing in uninterpreted bytes, so we put in semantic stringified json data there for version numbers etc. that are beyond the data rep typing . Super useful when maturing from localized notebook code to software services standardized on arrow interop.
- thamer 6y agoOh cool thanks! I'll check it out.
- infinite8s 6y agoArrow has two possible formats - a stream based one where the first block is the schema/metadata followed by data blocks (ie record batches), and a file based one where the schema is also stored at the end, along with the index locations of each batch (this also allows random access): https://arrow.apache.org/docs/format/Columnar.html#ipc-streaming-format https://arrow.apache.org/docs/format/Columnar.html#ipc-strea... vs https://arrow.apache.org/docs/format/Columnar.html#ipc-file-format https://arrow.apache.org/docs/format/Columnar.html#ipc-file-...