I agree that the spin up time is not 0, but how much that matters is really influenced by the size of your jobs. I would also call out that, when developing ag

I agree that the spin up time is not 0, but how much that matters is really influenced by the size of your jobs. I would also call out that, when developing against Spark, you can and should be running your jobs locally to develop and debug them. So, I would lean on local Spark execution to keep a tight feedback loop. This is one of the big advantages of using Spark over Hadoop or similar more complex systems. Your jobs scale predictably from a local laptop to a cluster. Sure, there's a cutoff point, but within a range, this is largely a choice. We're a startup, so our traffic is always increasing. We want to worry about sizing our infrastructure as new jobs come on line. There is no perfect static size for our clusters. We've also been scaling down as well as up in the size of our infrastructure as we improve our code's performance and understand our systems better. So ephemeral just makes sense to us for the moment. Certainly we might want a persistent cluster at some point, but dynamically-scaled out clusters will likely still be valuable at that point.