4 ms·

Question, doesn't Parquet already do that?

• ianmcook 6y ago
  From https://arrow.apache.org/faq/ https://arrow.apache.org/faq/: "Parquet files cannot be directly operated on but must be decoded in large chunks... Arrow is an in-memory format meant for direct and efficient use for computational purposes. Arrow data is... laid out in natural format for the CPU, so that data can be accessed at arbitrary places at full speed."
• snicker7 6y ago
  Yes. But parquet is now based on Apache Arrow.
  • ianmcook 6y ago
    Parquet is not based on Arrow. The Parquet libraries are built into Arrow, but the two projects are separate and Arrow is not a dependency of Parquet.
    • mr_toad 6y ago
      Arrow has definitely influenced the design of Parquet, they’re meant to compliment each other.
      • deleted 6y ago
        [deleted]