3 ms·
Noob question: what is the relationship between Arrow, Parquet, and ORC? Do we need all three?
by twic 9y ago
Noob question: what is the relationship between Arrow, Parquet, and ORC? Do we need all three?
- jamesblonde 9y agoParquet and ORC are columnar on-disk data formats that power Hadoop-on-SQL engines (Impala/SparkSQL and Hive, respectively). Arrow is an in-memory representation (Parquet/Orc are on-disk). The idea is that you can have workflows in different languages or frameworks using the same in-memory representation, not having to rebuild it just because you're going from Spark to another framework