This is still not an uncommon problem. Interestingly, we (Cloud Dataproc team) have been trying to work in the opposite direction. A few months ago we launched

This is still not an uncommon problem. Interestingly, we (Cloud Dataproc team) have been trying to work in the opposite direction. A few months ago we launched single-node clusters so people can use Spark on one VM instead of creating crazy-huge clusters just for lightweight data processing. No sense in using a a fleet of cores for simple stuff. :) Disclaimer - work for Google Cloud (and think we need better tools for simple data processing)