4 ms·

Check out Apache Arrow and Feather for a "local" equivalent of Parquet https://blog.cloudera.com/blog/2016/03/feather-a-fast-on-disk-format-for-data-frames-for

• tomnipotent 10y ago
  Not sure what you mean by "local". Arrow is in-memory, Parquet is on-disk. They are complimentary techs.
  • EwanToo 10y ago
    But Feather is, as it describes itself, all about making: "pushing data frames in and out of memory as simple as possible" Parquet is fantastic, but it has a lot of functionality that's not very relevant for sub-TB of data, local use.
    • tomnipotent 10y ago
      >> pushing data frames in and out of memory as simple as possible Where do you think its pushing from and to? Parquet. Wes McKinney (of pandas fame) is the author behind Arrow/Feather as well as parquet-cpp [1]. Arrow is a compact binary representation of columnar data for in-memory processing with the HUGE benefit that data can be copied over the network with no serialization/deserialization between processes or languages. One of the primary advantages of Spark over MapReduce is that it doesn't require data to be copied to disk and incur costly SerDer overhead. Now imagine that not only can I avoid writing to disk between processing stages, I can now move GBs of data between nodes by simply copying data from the network stack to memory as-is. >> Parquet is fantastic, but it has a lot of functionality that's not very relevant for sub-TB of data, local use. Parquet files can still be on local storage, there's no dependency on HDFS or network storage. If I'm only dealing with MBs then CSV will do the trick, otherwise Parquet wins in every category when reading/writing data. [1] https://github.com/apache/parquet-cpp https://github.com/apache/parquet-cpp