This is true, but using a Distributed system like Spark itself adds a ton of complexity in having to understand and manage it. If one can do something with a se

This is true, but using a Distributed system like Spark itself adds a ton of complexity in having to understand and manage it. If one can do something with a set of stateless processes, even if it's more performant, I feel it's a bad idea to use a distributed system instead. Not always, but a good majority of cases that I've seen. I've seen projects where Celery would be enough but instead they chose to use Spark/Storm and never delivered.