As someone who runs jobs on giant clusters day in day out, I just looked at my last job. It indeed had input data of ~100GB. However size of input data is misle

As someone who runs jobs on giant clusters day in day out, I just looked at my last job. It indeed had input data of ~100GB. However size of input data is misleading. Job does a lot of processing and generates ~5TB of intermediate data and it took 800+ machine hours to complete. If I'd ran that on my desktop I would be waiting for a month to finish. On cluster it took ~4 hours. I'd to smile at the statement "Is more data always better? Hardly". There is old saying the world of data scientists: There is no data like more data. Yes, the value of it may be diminishing but when your competitor is trying to squeeze out gain in second decimal, you are probably better off accepting more data. So the moral of the story is, all these really depends. People do get fired for buying clusters. Modern cluster management software track several utilization metrics and someone some day would going to look at it and point out how bad decision it was.