Spark seems like it will be good, but always disappoints. I doubt know much about spark, so I may get this wrong, but I feel that they need to greatly improve

Spark seems like it will be good, but always disappoints. I doubt know much about spark, so I may get this wrong, but I feel that they need to greatly improve both: 1) the automatic configuration of single node systems 2) the changing of config throughout the processing The fact that I don't even really know if this is the problem however is the real problem. I get that setting stuff up for a huge network of computers may be challenging, but standard defaults to use all processors on a single machine should be automatic. Spark pisses me off that the defaults for a 1TB RAM, 20 core machine as a single node can't handle reading a 20GB parquet file with one line. And further, checking substrings can throw OOM errors in the next step without adjusting the config of worker memory, etc. Worst of all, a 40 MB (!) file after filtering and processing can take like 10 minutes to save because of all the parallelism config. It's off course instantaneous on pandas. They really need to fix the experience for single machines (e.g. just like a small 18 core, 512 GB ram system), so that people can write packages that work for anyone, but with a small yaml flag, it can be sent to a cluster.