Or; the way that Spark writes files is to save the file in a folder, and break it up into multiple partitions (if the data is large). The way "normal" (non-HDF

Or; the way that Spark writes files is to save the file in a folder, and break it up into multiple partitions (if the data is large). The way "normal" (non-HDFS) tools write data is by creating a file with an extension. Let's say you have a 2GB parquet file, if it's broken down into 4 smaller positions, your read rates will likely be quicker, especially if the file is distributed.