3 ms·

"If you are responsible for generating parquet from another format—say you are using PyArrow and Pandas for some large-scale migration—be conscious that simply

by tveita 8y ago

"If you are responsible for generating parquet from another format—say you are using PyArrow and Pandas for some large-scale migration—be conscious that simply creating a single parquet file gives up a major benefit of the format." I don't understand this part, it's not clear which major benefit you're giving up, or what you should do instead. Is it saying not to convert these formats to parquet? Or that you should create multiple parquet files to get the full benefits?

enigma_daniel 8y ago
Great question - and I agree that it is a bit unclear as to which benefit you are giving up. The benefit would be using multiple parquet files for increased performance.
- nevi-me 8y ago
  Or; the way that Spark writes files is to save the file in a folder, and break it up into multiple partitions (if the data is large). The way "normal" (non-HDFS) tools write data is by creating a file with an extension. Let's say you have a 2GB parquet file, if it's broken down into 4 smaller positions, your read rates will likely be quicker, especially if the file is distributed.
- rsanders 8y ago
  I believe that if you have a Parquet file meeting certain criteria, it's directly parallelizable as multiple Spark partitions without any shuffling. The splits would occur at Parquet row group boundaries, I believe. See https://stackoverflow.com/questions/27194333/how-to-split-parquet-files-into-many-partitions-in-spark/29819133#29819133 https://stackoverflow.com/questions/27194333/how-to-split-pa..., https://parquet.apache.org/documentation/latest/ https://parquet.apache.org/documentation/latest/, etc. Whether it's better to have multiple Parquet files or a single parallelizable Parquet file is dependent on your environment and application. At my company, we've tended to have a single row group per file (and one HDFS block per file), in part due to historical reasons.

by tveita 8y ago

enigma_daniel 8y ago

Great question - and I agree that it is a bit unclear as to which benefit you are giving up. The benefit would be using multiple parquet files for increased performance.

nevi-me 8y ago
Or; the way that Spark writes files is to save the file in a folder, and break it up into multiple partitions (if the data is large). The way "normal" (non-HDFS) tools write data is by creating a file with an extension. Let's say you have a 2GB parquet file, if it's broken down into 4 smaller positions, your read rates will likely be quicker, especially if the file is distributed.
rsanders 8y ago
I believe that if you have a Parquet file meeting certain criteria, it's directly parallelizable as multiple Spark partitions without any shuffling. The splits would occur at Parquet row group boundaries, I believe. See https://stackoverflow.com/questions/27194333/how-to-split-parquet-files-into-many-partitions-in-spark/29819133#29819133 https://stackoverflow.com/questions/27194333/how-to-split-pa..., https://parquet.apache.org/documentation/latest/ https://parquet.apache.org/documentation/latest/, etc. Whether it's better to have multiple Parquet files or a single parallelizable Parquet file is dependent on your environment and application. At my company, we've tended to have a single row group per file (and one HDFS block per file), in part due to historical reasons.