4 ms·

> ...BigQuery's per-query pricing charges you JUST for the resources you use (well, data scaned), so it SHOULD be far less expensive than a model that charges y

> ...BigQuery's per-query pricing charges you JUST for the resources you use (well, data scaned), so it SHOULD be far less expensive than a model that charges you for the luxury of having a cluster sit idle (and often at only 30% utilization), correct? > For example, if Netflix charged you a penny per minute of watchtime, you'd have no idea if it's more expensive or less expensive, but you'd be assured it's more efficient. The Netflix analogy is actually quite good. Using that, let's say that I have a family of six, I'm billed $0.01 per minute of viewership, but I'm getting the best/quality speed. Each person watches 40 minutes a day, for all 30 days, for a grand total of $72. Far greater than the fixed cost of $10.99 (with, say, significantly less quality, speed, and minutes per month of viewership). In the real world, the family of six is my team of data analysts, scientists, and BI folks who are querying my database from 9-5 every weekday. A customer of mine, a large NYC publisher, who you have heard of and probably read, evaluated BigQuery in 2017. They loaded BQ with the exact same data as their Redshift cluster, pointed their Looker instance at it, and in just one day blew through 1/4 their typical Redshift budget. All the queries were faster. Way faster than they needed to be, actually. Going back to my original post above, the issue here is that just because Google CAN throw these massive amounts of resources at my problem, doesn't mean I can afford to use that level computation for each query, or would even want to. In the Netflix example, I'm happy if Sally and Billy get less quality or limited time watching Netflix, as long as my bill stays at $10.99. For most companies, their Redshift cluster is optimized to be able to handle their peak workload WELL ENOUGH. That means that queries won't be as fast as BigQuery - and that's totally fine. And it means that the cluster will be underutilized for large portions of the night and weekends - again, totally fine. They just need their usage capped at a predetermined cost, and have their queries finishing in a reasonable amount of time. I've posed this Google employees before and I'm hit with "well, you can limit how much each person can query a day." Except that isn't a acceptable solution. I can't have analysts sitting around unable to query their database because they've exceeded their daily limit. They'd rather just fire off their Redshift query and if it takes a little bit longer, so be it.

A21z 8y ago
Netflix's analogy is great, and I would add: being charged $0.01 per minute of viewership is so stressful you would consume less, or you would be stressed about what your parents would say about the bill.
vgt 8y ago
Thanks, this is very valuable feedback, worth in gold. Please feel free to email me directly to unpack our future plans here. The Netflix analogy also applies if they bill $0.001 per minute - the number I chose is not indicative of reality, I was just making a point. My goal was to demonstrate the fact that it's more efficient per-resource.
manigandham 8y ago
To repeat my other comments, the issue is that BQ doesn't actually charge for compute but for data scanned. You already pay for storage (uncompressed but that's a separate issue) so paying again on that metric doesn't make sense for querying since you're using CPU cores, which is a time-based metric. If I use 1000 cores for 10 seconds vs 10 minutes, it makes sense to pay exactly that regardless of how much data was scanned. We've run into the same issue where being curious with BigQuery actually becomes problematic as users are perfectly fine waiting an extra minute to scan 10TBs, but are afraid of the $50 bill that comes with that, especially for every little query that might be mistyped or out of their hands when using BI tools that run queries of their own.

A21z 8y ago

Netflix's analogy is great, and I would add: being charged $0.01 per minute of viewership is so stressful you would consume less, or you would be stressed about what your parents would say about the bill.

vgt 8y ago

Thanks, this is very valuable feedback, worth in gold. Please feel free to email me directly to unpack our future plans here. The Netflix analogy also applies if they bill $0.001 per minute - the number I chose is not indicative of reality, I was just making a point. My goal was to demonstrate the fact that it's more efficient per-resource.

manigandham 8y ago

To repeat my other comments, the issue is that BQ doesn't actually charge for compute but for data scanned. You already pay for storage (uncompressed but that's a separate issue) so paying again on that metric doesn't make sense for querying since you're using CPU cores, which is a time-based metric. If I use 1000 cores for 10 seconds vs 10 minutes, it makes sense to pay exactly that regardless of how much data was scanned. We've run into the same issue where being curious with BigQuery actually becomes problematic as users are perfectly fine waiting an extra minute to scan 10TBs, but are afraid of the $50 bill that comes with that, especially for every little query that might be mistyped or out of their hands when using BI tools that run queries of their own.