3 ms·

Things I Wish I'd Known About Spark When I Started

39 points by enigma_daniel 8y ago

tveita 8y ago
"If you are responsible for generating parquet from another format—say you are using PyArrow and Pandas for some large-scale migration—be conscious that simply creating a single parquet file gives up a major benefit of the format." I don't understand this part, it's not clear which major benefit you're giving up, or what you should do instead. Is it saying not to convert these formats to parquet? Or that you should create multiple parquet files to get the full benefits?
- enigma_daniel 8y ago
  Great question - and I agree that it is a bit unclear as to which benefit you are giving up. The benefit would be using multiple parquet files for increased performance.
  - nevi-me 8y ago
    Or; the way that Spark writes files is to save the file in a folder, and break it up into multiple partitions (if the data is large). The way "normal" (non-HDFS) tools write data is by creating a file with an extension. Let's say you have a 2GB parquet file, if it's broken down into 4 smaller positions, your read rates will likely be quicker, especially if the file is distributed.
  - rsanders 8y ago
    I believe that if you have a Parquet file meeting certain criteria, it's directly parallelizable as multiple Spark partitions without any shuffling. The splits would occur at Parquet row group boundaries, I believe. See https://stackoverflow.com/questions/27194333/how-to-split-parquet-files-into-many-partitions-in-spark/29819133#29819133 https://stackoverflow.com/questions/27194333/how-to-split-pa..., https://parquet.apache.org/documentation/latest/ https://parquet.apache.org/documentation/latest/, etc. Whether it's better to have multiple Parquet files or a single parallelizable Parquet file is dependent on your environment and application. At my company, we've tended to have a single row group per file (and one HDFS block per file), in part due to historical reasons.
ataturk 8y ago
Most of the people I work with utter the phrase SparkScala as though it were a single word. I sense that it is more of an ego trip than anything real. Java is a 1st class citizen in Spark land, too, and it is easier to write tests for and hire developers for, too. Just food for thought. I have heard there is a performance penalty for Java, but I don't know if it is a big deal. Also, parquet is great in Hadoop Land, but outside of that it can be difficult to read Parquet files, say from a mounted filesystem in Kubernetes using a microservice.
nevi-me 8y ago
Interesting points! I haven't used Spark in a while until I recently got back into it. I think data engineers 'should' add Spark to their toolset, even if just for ETL. I found myself having to go through a few hundred CSV files and import them into Oracle, Spark made it fast and easy. Many of the pain points that you mention are universal whether using Scala or Python. I wish I'd have known about .par years ago. I also find checkpointing to help a lot, especially if using JDBC data sources. If in reading in data that doesn't change, I read it into parquet, and then change to reading from parquet. Arrow has made pyspark more pleasant to work with, especially if you primarily work on notebooks. The one thing which I struggled with was latency on time-critical jobs. I'd sometimes get a few minutes' pauses on jobs that should take under a minute to run. I haven't checked how that's improved since Spark 2.0 though

39 points by enigma_daniel 8y ago

"If you are responsible for generating parquet from another format—say you are using PyArrow and Pandas for some large-scale migration—be conscious that simply creating a single parquet file gives up a major benefit of the format." I don't understand this part, it's not clear which major benefit you're giving up, or what you should do instead. Is it saying not to convert these formats to parquet? Or that you should create multiple parquet files to get the full benefits?

enigma_daniel 8y ago
Great question - and I agree that it is a bit unclear as to which benefit you are giving up. The benefit would be using multiple parquet files for increased performance.
- nevi-me 8y ago
  Or; the way that Spark writes files is to save the file in a folder, and break it up into multiple partitions (if the data is large). The way "normal" (non-HDFS) tools write data is by creating a file with an extension. Let's say you have a 2GB parquet file, if it's broken down into 4 smaller positions, your read rates will likely be quicker, especially if the file is distributed.
- rsanders 8y ago
  I believe that if you have a Parquet file meeting certain criteria, it's directly parallelizable as multiple Spark partitions without any shuffling. The splits would occur at Parquet row group boundaries, I believe. See https://stackoverflow.com/questions/27194333/how-to-split-parquet-files-into-many-partitions-in-spark/29819133#29819133 https://stackoverflow.com/questions/27194333/how-to-split-pa..., https://parquet.apache.org/documentation/latest/ https://parquet.apache.org/documentation/latest/, etc. Whether it's better to have multiple Parquet files or a single parallelizable Parquet file is dependent on your environment and application. At my company, we've tended to have a single row group per file (and one HDFS block per file), in part due to historical reasons.

ataturk 8y ago

Most of the people I work with utter the phrase SparkScala as though it were a single word. I sense that it is more of an ego trip than anything real. Java is a 1st class citizen in Spark land, too, and it is easier to write tests for and hire developers for, too. Just food for thought. I have heard there is a performance penalty for Java, but I don't know if it is a big deal. Also, parquet is great in Hadoop Land, but outside of that it can be difficult to read Parquet files, say from a mounted filesystem in Kubernetes using a microservice.

nevi-me 8y ago

Interesting points! I haven't used Spark in a while until I recently got back into it. I think data engineers 'should' add Spark to their toolset, even if just for ETL. I found myself having to go through a few hundred CSV files and import them into Oracle, Spark made it fast and easy. Many of the pain points that you mention are universal whether using Scala or Python. I wish I'd have known about .par years ago. I also find checkpointing to help a lot, especially if using JDBC data sources. If in reading in data that doesn't change, I read it into parquet, and then change to reading from parquet. Arrow has made pyspark more pleasant to work with, especially if you primarily work on notebooks. The one thing which I struggled with was latency on time-critical jobs. I'd sometimes get a few minutes' pauses on jobs that should take under a minute to run. I haven't checked how that's improved since Spark 2.0 though