4 ms·

Can you talk about how you use it? At a cursory glance, it seems like it's being used to great effect in databases, or systems designed to query data. Is that w

• Q6T46nT668w6i3m 6y ago
  An alternative to CSV or SQLite for interchange.
  • woadwarrior01 6y ago
    From my experience, Parquet files are a better alternative to CSV or SQLite files for data at rest. I've observed that parquet files tend to be smaller than Feather files (which is essentially the Arrow format in a file) for the same data. Arrow is great for in-memory data, but even with compression, the on-disk size of Feather files tend to be quite a bit bigger than Parquet files.
    • ttymck 6y ago
      "for interchange" in data science might well mean data on-the-move rather than data at-rest: i.e. a pipeline in two stages, step A in python and step B in R. The data is exchanged (relatively) quickly via disk, and speed beats compression.
      • woadwarrior01 6y ago
        Even with fast SSDs, memcpy() is roughly an order of magnitude faster than writing to disk. Using a fast compression algorithm like lz4, zstd, etc can help reduce that gap. And this is invariant of whether the serialization format is Arrow or Parquet. It’s not either, or. If you’re writing to disk, with compression you can have both speed and reduction in the disk footprint.