This reminds me my experience from a company internal hackathon. My colleague started writing a Spark program that would process the data we needed (a few hundr

This reminds me my experience from a company internal hackathon. My colleague started writing a Spark program that would process the data we needed (a few hundreds GB uncompressed). Before he finished writing it, I was able to process all the data on a single machine with a unix pipeline. The computationally intensive steps were basically just grep, sort and uniq. When he finished the program, it couldn't run because of some operational issues on the cluster at the moment, so we didn't even find out the speed to compare. For me, the morale is that the cheap hardware saves money/time twice: 1. It's faster if a program can run on a single machine. 2. It's easier to write a program that runs on a single machine. With this in mind, cloud works great for analytical data processing. Just start a big enough machine, download data, do the computation, upload the result and turn the machine off. If you develop the program on a sample of the data so you can do it locally, it will be even cheap because you use only short time of the powerful server.