5 ms·
When it comes to Big Data, per-minute billing + rapid startup is significant. When you can have a 10,000-core Hadoop cluster in 45 seconds, you stop thinking ab
by vgt 11y ago
When it comes to Big Data, per-minute billing + rapid startup is significant. When you can have a 10,000-core Hadoop cluster in 45 seconds, you stop thinking about it in terms of clusters and start thinking about it in terms of jobs.
Before: Start cluster > submit many jobs to the cluster > manage resources
Now: Submit a job to Dataproc > Start a cluster for each job > shut down the cluster
And, of course, BigQuery gives you the equivalent of per-second billing and 0-second scaling to thousands of cores:
https://cloud.google.com/blog/big-data/2016/02/understanding-bigquerys-rapid-scaling-and-simple-pricing https://cloud.google.com/blog/big-data/2016/02/understanding...
- izzym 11y agoBigQuery is well and truly amazing, zero upfront setup or infrastructure. You ur Query & BigQuery that is it
- chimerasaurus 11y agoIn addition, you can also use preemptibles and custom VM machine types (specify the exact ratio of CPU/RAM you want for master + workers) for your clusters. This gives even more control for cost vs resources. Disclaimer - I work at Google, on Cloud Dataproc (and we are passionate about making the Spark/Hadoop ecosystem both super fast, but also extremely cost effective) :)