4 ms·

Great question - and I agree that it is a bit unclear as to which benefit you are giving up. The benefit would be using multiple parquet files for increased per

• nevi-me 8y ago
  Or; the way that Spark writes files is to save the file in a folder, and break it up into multiple partitions (if the data is large). The way "normal" (non-HDFS) tools write data is by creating a file with an extension. Let's say you have a 2GB parquet file, if it's broken down into 4 smaller positions, your read rates will likely be quicker, especially if the file is distributed.
• rsanders 8y ago
  I believe that if you have a Parquet file meeting certain criteria, it's directly parallelizable as multiple Spark partitions without any shuffling. The splits would occur at Parquet row group boundaries, I believe. See https://stackoverflow.com/questions/27194333/how-to-split-parquet-files-into-many-partitions-in-spark/29819133#29819133 https://stackoverflow.com/questions/27194333/how-to-split-pa..., https://parquet.apache.org/documentation/latest/ https://parquet.apache.org/documentation/latest/, etc. Whether it's better to have multiple Parquet files or a single parallelizable Parquet file is dependent on your environment and application. At my company, we've tended to have a single row group per file (and one HDFS block per file), in part due to historical reasons.