Interesting points! I haven't used Spark in a while until I recently got back into it. I think data engineers 'should' add Spark to their toolset, even if just

Interesting points! I haven't used Spark in a while until I recently got back into it. I think data engineers 'should' add Spark to their toolset, even if just for ETL. I found myself having to go through a few hundred CSV files and import them into Oracle, Spark made it fast and easy. Many of the pain points that you mention are universal whether using Scala or Python. I wish I'd have known about .par years ago. I also find checkpointing to help a lot, especially if using JDBC data sources. If in reading in data that doesn't change, I read it into parquet, and then change to reading from parquet. Arrow has made pyspark more pleasant to work with, especially if you primarily work on notebooks. The one thing which I struggled with was latency on time-critical jobs. I'd sometimes get a few minutes' pauses on jobs that should take under a minute to run. I haven't checked how that's improved since Spark 2.0 though