3 ms·

We use BigQuery a lot for internal analytics and we've been super happy. I don't see a lot of love for BigQuery on HN and I wonder why. Tons of features, no has

We use BigQuery a lot for internal analytics and we've been super happy. I don't see a lot of love for BigQuery on HN and I wonder why. Tons of features, no hassle and easy to throw a bunch of TB at it. I guess maybe the cost?

RadiozRadioz 3y ago
Probably also because it is proprietary and only exists in one cloud platform.
- wodenokoto 3y ago
  No, it’s because it’s google and HN are certain it will get cancelled at any moment.
  - alooPotato 3y ago
    seems unlikely, I think its the most popular google cloud product
wodenokoto 3y ago
I was quite surprised that other clouds don’t have an easy to get started analytics data warehouse solution like big query.
mnahkies 3y ago
I'm a big fan of big query as well, but the cost can be problematic if you're not careful. Generally speaking I've found it manageable if you make good use of partitioning and do incremental aggregation (we use dbt, though you have to do some macro gymnastics to make the partition key filter eligible for pruning due to restrictions on use of dynamic values https://docs.getdbt.com/docs/build/incremental-models https://docs.getdbt.com/docs/build/incremental-models) It's also important to monitor your cost and watch for the point where switching from the per-tb queried pricing model to slots makes sense.
- alooPotato 3y ago
  yeah between partitioning, clustering, materialized views, and smart tuning it seems like there are enough knobs to control costs.
doo_daa 3y ago
We are lucky enough to be able to run BigQuery with flat rate billing. It's incredibly powerful and it's a really good example of SaaS and Serverless done right. It just works.
lysecret 3y ago
Yep love it too, especially with external data on GCS. Costs this way are very low. And the convenience is amazing (getting caches you can stream from for every query is a godsend)
- alooPotato 3y ago
  What do you mean by the streaming caches?

RadiozRadioz 3y ago

Probably also because it is proprietary and only exists in one cloud platform.

wodenokoto 3y ago
No, it’s because it’s google and HN are certain it will get cancelled at any moment.
- alooPotato 3y ago
  seems unlikely, I think its the most popular google cloud product

wodenokoto 3y ago

I was quite surprised that other clouds don’t have an easy to get started analytics data warehouse solution like big query.

mnahkies 3y ago

I'm a big fan of big query as well, but the cost can be problematic if you're not careful. Generally speaking I've found it manageable if you make good use of partitioning and do incremental aggregation (we use dbt, though you have to do some macro gymnastics to make the partition key filter eligible for pruning due to restrictions on use of dynamic values https://docs.getdbt.com/docs/build/incremental-models https://docs.getdbt.com/docs/build/incremental-models) It's also important to monitor your cost and watch for the point where switching from the per-tb queried pricing model to slots makes sense.

alooPotato 3y ago
yeah between partitioning, clustering, materialized views, and smart tuning it seems like there are enough knobs to control costs.

doo_daa 3y ago

We are lucky enough to be able to run BigQuery with flat rate billing. It's incredibly powerful and it's a really good example of SaaS and Serverless done right. It just works.

lysecret 3y ago

Yep love it too, especially with external data on GCS. Costs this way are very low. And the convenience is amazing (getting caches you can stream from for every query is a godsend)

alooPotato 3y ago
What do you mean by the streaming caches?