4 ms·
If you were ready to throw away a bit of archival quality, I wonder if writing Apache Arrow straight to disk and then just mmap your columns back in for a query
by jaxrtech 4y ago
If you were ready to throw away a bit of archival quality, I wonder if writing Apache Arrow straight to disk and then just mmap your columns back in for a query.
- teej 4y agoIt's certainly possible. But Parquet and Arrow are being built with each other in mind. I doubt the tradeoffs on storage, durability, and future-proofness are worth skipping the marginal overhead to load a Parquet file into memory.
- DasIch 4y agoIsn't that what Feather, the arrow file format, is? https://arrow.apache.org/docs/python/feather.html https://arrow.apache.org/docs/python/feather.html