I mean for the problem at hand ... not for anything, ever :) I think usually the argument for Spark and the like proceeds by just making the dataset bigger unt

I mean for the problem at hand ... not for anything, ever :) I think usually the argument for Spark and the like proceeds by just making the dataset bigger until whatever you’re comparing becomes absurd (e.g. command line vs Spark). It was the same deal in the map reduce days. Works like shit for 99% of datasets but it’s the best thing ever when there are no other alternatives :)