3 ms·

> On the pay-per-TB pricing for BigQuery and Snowflake. I think that's more of a marketing spin. Most companies we work with are compute-bound. So per-TB pricin

by vgt 10y ago

> On the pay-per-TB pricing for BigQuery and Snowflake. I think that's more of a marketing spin. Most companies we work with are compute-bound. So per-TB pricing helps them very little. They want to crunch their data as fast as possible. More CPUs give them more I/O. If you feel you're storage-bound - take a hard look at the data that really needs to be available in Redshift for analysis, and move everything else to S3. Do you mind elaborating on what you mean by that? I've worked a whole lot on BigQuery pricing, and am a little confused by your comment. Do you mean that separation of storage compute is an overrated concept? (work on Google Cloud)

marcinzukowski 10y ago
To clarify, Snowflake doesn't charge for processing per-TB like BigQuery, but for the (fully elastic) compute time. (disclaimer: I work at Snowflake)
- vgt 10y ago
  Thanks for the clarification! BigQuery's on-demand model charges just for the resources consumed during the job execution (via a per-TB proxy), rather than resources provisioned. This is the highest order of cloud-native pricing models, and good on Athena for doing the same! If one were to equate this to VM pricing, you're getting: - The ability to scale from 0 to thousands of cores in roughly 1 second (what it takes for Dremel to put together the execution plan) - Pipelined execution (meaning, each processing stage ONLY uses the resources required for that stage, rather than net resources. think of this as per-second autoscaling from query stage to query stage. This is part of the reason why BigQuery does very well in concurrency tests) - Equivalent of per-second billing This in practice equates to 100% resource efficiency - never pay for resources you just deploy and not use. I've written on this topic at [0] and at [1] and happy to chat further! (And BigQuery did introduce a Flat Rate pricing model last Fall for the more enterprisey crowd) [0] https://cloud.google.com/blog/big-data/2016/02/understanding-bigquerys-rapid-scaling-and-simple-pricing https://cloud.google.com/blog/big-data/2016/02/understanding... [1] https://cloud.google.com/blog/big-data/2016/02/visualizing-the-mechanics-of-on-demand-pricing-in-big-data-technologies https://cloud.google.com/blog/big-data/2016/02/visualizing-t...
  - scapecast 10y ago
    this is a very elaborate and insightful answer. appreciate the clarification.

by vgt 10y ago

marcinzukowski 10y ago

To clarify, Snowflake doesn't charge for processing per-TB like BigQuery, but for the (fully elastic) compute time. (disclaimer: I work at Snowflake)

vgt 10y ago
Thanks for the clarification! BigQuery's on-demand model charges just for the resources consumed during the job execution (via a per-TB proxy), rather than resources provisioned. This is the highest order of cloud-native pricing models, and good on Athena for doing the same! If one were to equate this to VM pricing, you're getting: - The ability to scale from 0 to thousands of cores in roughly 1 second (what it takes for Dremel to put together the execution plan) - Pipelined execution (meaning, each processing stage ONLY uses the resources required for that stage, rather than net resources. think of this as per-second autoscaling from query stage to query stage. This is part of the reason why BigQuery does very well in concurrency tests) - Equivalent of per-second billing This in practice equates to 100% resource efficiency - never pay for resources you just deploy and not use. I've written on this topic at [0] and at [1] and happy to chat further! (And BigQuery did introduce a Flat Rate pricing model last Fall for the more enterprisey crowd) [0] https://cloud.google.com/blog/big-data/2016/02/understanding-bigquerys-rapid-scaling-and-simple-pricing https://cloud.google.com/blog/big-data/2016/02/understanding... [1] https://cloud.google.com/blog/big-data/2016/02/visualizing-the-mechanics-of-on-demand-pricing-in-big-data-technologies https://cloud.google.com/blog/big-data/2016/02/visualizing-t...
- scapecast 10y ago
  this is a very elaborate and insightful answer. appreciate the clarification.