Let's say you're developing your analysis script, and it takes 45 minutes to run. That doesn't sound like much of a problem, but it means you only really get to

Let's say you're developing your analysis script, and it takes 45 minutes to run. That doesn't sound like much of a problem, but it means you only really get to try 10 things a day. You can kick something off and then start trying something different, but that's actually quite a burden. It's cognitively easier to run a series of parallel tasks instead of parallel serial tasks. Even if a job is taking 15-20 minutes, that can really wear you down, and give you a pretty unproductive week. It's hard to keep your place. That said: I think often these 45 minute tasks should be much faster, and folks would actually benefit from optimising their code a bit more. If I have to do a word count in 30gb of text, I'd rather implement carefully or use Hyper Log Log. I find it weird that people would rather use Spark, which I always find quite painful. I guess it comes down to familiarity.