4 ms·
Is Arrow data commonly written to disk? I was under the impression (as someone who doesn't use it currently) it's usually converted to Parquet for this purpose.
by anderspitman 6y ago
Is Arrow data commonly written to disk? I was under the impression (as someone who doesn't use it currently) it's usually converted to Parquet for this purpose.
- thundergolfer 6y agoI'd be interested to know any benefits of converting to Parquet if you know your reader supports Arrow.
- lmeyerov 6y agoCompression. For our app, we do multilevel caches, where warm is arrow (in-memory, inflight, streaming, ...) and cold (persistent/scaleout) is parquet (ssd, s3, ...).
- andygrove 6y agoArrow is a memory format and optimized for efficient vectorized processing in memory. Although it is possible to persist the Arrow format to disk, it is more common to use Parquet.