3 ms·
interesting idea. I can see it being worthwhile with per-minute billing, which is not something supported by AWS. Also EMR charges a per-instance premium, does
by matt_wulfeck 9y ago
interesting idea. I can see it being worthwhile with per-minute billing, which is not something supported by AWS. Also EMR charges a per-instance premium, does anyone know if Google Cloud does similar?
I'm curious how shuffle data is handled. Does the cluster intelligently scale down and move the shuffle data, or will the entire thing keep running while waiting for a single skewed reducer to finish? Or does the entire thing run on a single instance??
- vgt 9y agoThere is a per-core surcharge of basically $0.01 per hour [9]. And sadly Dataproc doesn't have a good autoscaling story. On the flip side, you can use Preemptible VMs for Dataproc, which are 80% off, and Dataflow has an amazing auto-scaler [77] https://cloud.google.com/blog/big-data/2017/01/understanding-cost-versus-speed-tradeoffs-in-google-cloud-dataflow-batch-pipelines https://cloud.google.com/blog/big-data/2017/01/understanding... [9] https://cloud.google.com/dataproc/docs/resources/pricing https://cloud.google.com/dataproc/docs/resources/pricing
- tawkspace 9y ago(Disclaimer: work on Dataproc) Though Dataproc doesn't natively have an autoscaler, Spotify wrote one as part of Spydra: https://github.com/spotify/spydra https://github.com/spotify/spydra
- Xeago 9y agoThe autoscaler here doesn't solve the downscaling issues mentioned. We've seen it work fine for scaling up, quickly. For now we're not scaling down, as the job will finish quicker than individual tasks will time out when unlucky.