3 ms·

There is a per-core surcharge of basically $0.01 per hour [9]. And sadly Dataproc doesn't have a good autoscaling story. On the flip side, you can use Preempti

by vgt 9y ago

There is a per-core surcharge of basically $0.01 per hour [9]. And sadly Dataproc doesn't have a good autoscaling story. On the flip side, you can use Preemptible VMs for Dataproc, which are 80% off, and Dataflow has an amazing auto-scaler [77] https://cloud.google.com/blog/big-data/2017/01/understanding-cost-versus-speed-tradeoffs-in-google-cloud-dataflow-batch-pipelines https://cloud.google.com/blog/big-data/2017/01/understanding... [9] https://cloud.google.com/dataproc/docs/resources/pricing https://cloud.google.com/dataproc/docs/resources/pricing

tawkspace 9y ago
(Disclaimer: work on Dataproc) Though Dataproc doesn't natively have an autoscaler, Spotify wrote one as part of Spydra: https://github.com/spotify/spydra https://github.com/spotify/spydra
- Xeago 9y ago
  The autoscaler here doesn't solve the downscaling issues mentioned. We've seen it work fine for scaling up, quickly. For now we're not scaling down, as the job will finish quicker than individual tasks will time out when unlucky.