4 ms·
From my experience, Parquet files are a better alternative to CSV or SQLite files for data at rest. I've observed that parquet files tend to be smaller than Fe
by woadwarrior01 6y ago
From my experience, Parquet files are a better alternative to CSV or SQLite files for data at rest. I've observed that parquet files tend to be smaller than Feather files (which is essentially the Arrow format in a file) for the same data. Arrow is great for in-memory data, but even with compression, the on-disk size of Feather files tend to be quite a bit bigger than Parquet files.
- ttymck 6y ago"for interchange" in data science might well mean data on-the-move rather than data at-rest: i.e. a pipeline in two stages, step A in python and step B in R. The data is exchanged (relatively) quickly via disk, and speed beats compression.
- woadwarrior01 6y agoEven with fast SSDs, memcpy() is roughly an order of magnitude faster than writing to disk. Using a fast compression algorithm like lz4, zstd, etc can help reduce that gap. And this is invariant of whether the serialization format is Arrow or Parquet. It’s not either, or. If you’re writing to disk, with compression you can have both speed and reduction in the disk footprint.