3 ms·

I had a job interview for a Data Science position, in which I was talking about data science stuff I'd done in my current position. I mentioned that I started w

I had a job interview for a Data Science position, in which I was talking about data science stuff I'd done in my current position. I mentioned that I started with R, but it would keep choking and running out of memory, so I moved to Python and largely fixed the issue. To which one of the interviewers asked why I didn't use a spark cluster. I was almost too stunned to answer. Their solution, was not to just use something more efficient, or even to just rent a single, more powerful EC2 instance or something, but to go to the effort of setting up and running a cluster. The dataset wasn't even that big: a few gb on disk.

lmm 9y ago
There's no a priori reason setting up a cluster has to be more effort than setting up a non-cluster. I mean, a computer is more complicated than a breadboard of custom-wired components, but most people find it easier to solve their problems with a computer. And Spark is usually much more efficient than R or Python (and gives you a nicer language to work with IMO, though that's very subjective). It's entirely possible a 1-node cluster would have outperformed your non-cluster approach, and while a 1-node cluster seems stupid it's useful if you know you're eventually going to have to scale, because it ensures that you test your scalability assumptions and don't embed any non-distributable steps into your logic.