5 ms·

Bq pm here. No papers, yes, but lots of talks and blog posts. Here's a list: https://medium.com/@thetinot/bigquery-required-reading-list-71945444477b https://m

by vgt 8y ago

Bq pm here. No papers, yes, but lots of talks and blog posts. Here's a list: https://medium.com/@thetinot/bigquery-required-reading-list-71945444477b https://medium.com/@thetinot/bigquery-required-reading-list-... Happy to answer any questions!

polskibus 8y ago
How would you compare it to a traditional OLAP solutions (MS SQL Analysis Services in OLAP model (not tabular), Essbase etc.) in terms of : - multidimensional / hierarchy modeling for analytical purposes - permissioning model (roles with element-level granularity, ie. User A - allow Country=USA, User B - allow Product=Bike) - historical modeling (built-in slowly-changing dimensions support) What is the underlying storage model? Is it a column store ? Is it closer to traditional row-based stores? Are you taking advantage of GPUs or other dedicated hardware to accelerate BQ?
- kornish 8y ago
  If you haven't read it already, I strongly suggest reading the original Dremel paper [0]. It's no doubt somewhat out of date, but I believe BQ is based in Dremel. tl;dr for underlying storage model: distributed column store which pushes computation down a tree to leaf nodes to parallelize disk I/O. Parent nodes aggregate computations before returning to the client. [0]: https://ai.google/research/pubs/pub36632 https://ai.google/research/pubs/pub36632
  - vgt 8y ago
    This video gives a walkthrough of the new version of Dremel that's been running in production for past 3 years (no paper sorry!): https://www.youtube.com/watch?v=UueWySREWvk https://www.youtube.com/watch?v=UueWySREWvk
nrjames 8y ago
Did you all work together with the Firebase team on the recent schema change? We're scrambling to mitigate the ~2x cost increase that caused for us, since it blew our analytics budget out of the water.
- fhoffa 8y ago
  Can you post on https://stackoverflow.com/questions/tagged/google-bigquery https://stackoverflow.com/questions/tagged/google-bigquery? I haven't heard about this, but if you give the community good specifics, we'll work on it.
- vgt 8y ago
  I second Felipe's comment. Shoot me a note.
  - nrjames 8y ago
    It was a change that Firebase implemented that went into effect about a month ago. Check this out and the look at the "Old Export Schema" at the bottom. The export schema went from multiple events per record to one event per record, with the User Parameters repeated on each record. That bloated the exports from Firebase and increased costs. https://support.google.com/firebase/answer/7029846?hl=en https://support.google.com/firebase/answer/7029846?hl=en
jimmytucson 8y ago
This blog post from 2 years ago from AWS has Redshift beating BigQuery pretty handily on TPC-H and TPS-DS: https://aws.amazon.com/blogs/big-data/fact-or-fiction-google-big-query-outperforms-amazon-redshift-as-an-enterprise-data-warehouse/ https://aws.amazon.com/blogs/big-data/fact-or-fiction-google... Has anything changed on BigQuery since then that would warrant rerunning those benchmarks? If Amazon’s largest cluster outperforms BigQuery then is your decision about which service to use just a cost calculation of dedicated per month for Redshift vs. projected units scanned per month on BigQuery?
- fhoffa 8y ago
  Let's say you are Twitter, Reddit, or the CTO of your current company. How do you choose your next analytical platform: a) Do you look at the benchmarks that each company produces, and choose the one that publishes the numbers that make them look the best? b) Do you ask "I have n data analysts with m different questions and I want to give them the most productive platform I can". Well, this is what Twitter chose: - How Twitter Migrated its On-Prem Analytics to Google Cloud https://www.youtube.com/watch?v=N3JAwCYGHU8 https://www.youtube.com/watch?v=N3JAwCYGHU8 Or listen to Nick Caldwell, Reddit VP Engineering, moving away from AWS to BigQuery: - "2017, which effectively brought us to the present system, we began forking all of our event data into BigQuery, after considering a lot of different alternatives" https://youtu.be/tKISLQ87GO8?t=426 https://youtu.be/tKISLQ87GO8?t=426 I prefer method b) :) Disclosure: I'm https://twitter.com/felipehoffa https://twitter.com/felipehoffa and I work for GCP
- blaisio 8y ago
  I've worked with both, and BigQuery is so much easier to work with. Redshift does give you more control (to a limited extent) but it takes a lot more work and specialized knowledge to perform simple tasks well. In the article you linked, they gloss over the part that takes by far the longest when working with redshift: setting up "compression, distribution keys on large tables, and sort keys on commonly filtered columns". With BigQuery, you don't really worry about that crap, or about reloading your terabytes of data if you made a mistake somewhere in the schema. You also don't worry about vacuuming, or running out of space, or taking down the database from excessive CPU usage. Do you want to have a team of people at your company whose job is just to keep Redshift running smoothly? Or do you want another team of analysts or engineers? I could see some companies being forced to use Redshift, especially if they're using S3 a lot, and I could see people saving money in some specialized use cases, but for most people BigQuery is almost certainly going to be cheaper and faster in almost every way.
djhworld 8y ago
Thanks for the links, will definitely take a look. I suspect you might be hearing some feedback from us soon w.r.t this trial, assuming the google people we are working with relay that back to you?
- vgt 8y ago
  I'd love to hear your feedback, so yes please :) Anything you don't like is an opportunity to make our product better

by vgt 8y ago

polskibus 8y ago

How would you compare it to a traditional OLAP solutions (MS SQL Analysis Services in OLAP model (not tabular), Essbase etc.) in terms of : - multidimensional / hierarchy modeling for analytical purposes - permissioning model (roles with element-level granularity, ie. User A - allow Country=USA, User B - allow Product=Bike) - historical modeling (built-in slowly-changing dimensions support) What is the underlying storage model? Is it a column store ? Is it closer to traditional row-based stores? Are you taking advantage of GPUs or other dedicated hardware to accelerate BQ?

kornish 8y ago
If you haven't read it already, I strongly suggest reading the original Dremel paper [0]. It's no doubt somewhat out of date, but I believe BQ is based in Dremel. tl;dr for underlying storage model: distributed column store which pushes computation down a tree to leaf nodes to parallelize disk I/O. Parent nodes aggregate computations before returning to the client. [0]: https://ai.google/research/pubs/pub36632 https://ai.google/research/pubs/pub36632
- vgt 8y ago
  This video gives a walkthrough of the new version of Dremel that's been running in production for past 3 years (no paper sorry!): https://www.youtube.com/watch?v=UueWySREWvk https://www.youtube.com/watch?v=UueWySREWvk

nrjames 8y ago

Did you all work together with the Firebase team on the recent schema change? We're scrambling to mitigate the ~2x cost increase that caused for us, since it blew our analytics budget out of the water.

fhoffa 8y ago
Can you post on https://stackoverflow.com/questions/tagged/google-bigquery https://stackoverflow.com/questions/tagged/google-bigquery? I haven't heard about this, but if you give the community good specifics, we'll work on it.
vgt 8y ago
I second Felipe's comment. Shoot me a note.
- nrjames 8y ago
  It was a change that Firebase implemented that went into effect about a month ago. Check this out and the look at the "Old Export Schema" at the bottom. The export schema went from multiple events per record to one event per record, with the User Parameters repeated on each record. That bloated the exports from Firebase and increased costs. https://support.google.com/firebase/answer/7029846?hl=en https://support.google.com/firebase/answer/7029846?hl=en

jimmytucson 8y ago

This blog post from 2 years ago from AWS has Redshift beating BigQuery pretty handily on TPC-H and TPS-DS: https://aws.amazon.com/blogs/big-data/fact-or-fiction-google-big-query-outperforms-amazon-redshift-as-an-enterprise-data-warehouse/ https://aws.amazon.com/blogs/big-data/fact-or-fiction-google... Has anything changed on BigQuery since then that would warrant rerunning those benchmarks? If Amazon’s largest cluster outperforms BigQuery then is your decision about which service to use just a cost calculation of dedicated per month for Redshift vs. projected units scanned per month on BigQuery?

fhoffa 8y ago
Let's say you are Twitter, Reddit, or the CTO of your current company. How do you choose your next analytical platform: a) Do you look at the benchmarks that each company produces, and choose the one that publishes the numbers that make them look the best? b) Do you ask "I have n data analysts with m different questions and I want to give them the most productive platform I can". Well, this is what Twitter chose: - How Twitter Migrated its On-Prem Analytics to Google Cloud https://www.youtube.com/watch?v=N3JAwCYGHU8 https://www.youtube.com/watch?v=N3JAwCYGHU8 Or listen to Nick Caldwell, Reddit VP Engineering, moving away from AWS to BigQuery: - "2017, which effectively brought us to the present system, we began forking all of our event data into BigQuery, after considering a lot of different alternatives" https://youtu.be/tKISLQ87GO8?t=426 https://youtu.be/tKISLQ87GO8?t=426 I prefer method b) :) Disclosure: I'm https://twitter.com/felipehoffa https://twitter.com/felipehoffa and I work for GCP
blaisio 8y ago
I've worked with both, and BigQuery is so much easier to work with. Redshift does give you more control (to a limited extent) but it takes a lot more work and specialized knowledge to perform simple tasks well. In the article you linked, they gloss over the part that takes by far the longest when working with redshift: setting up "compression, distribution keys on large tables, and sort keys on commonly filtered columns". With BigQuery, you don't really worry about that crap, or about reloading your terabytes of data if you made a mistake somewhere in the schema. You also don't worry about vacuuming, or running out of space, or taking down the database from excessive CPU usage. Do you want to have a team of people at your company whose job is just to keep Redshift running smoothly? Or do you want another team of analysts or engineers? I could see some companies being forced to use Redshift, especially if they're using S3 a lot, and I could see people saving money in some specialized use cases, but for most people BigQuery is almost certainly going to be cheaper and faster in almost every way.

djhworld 8y ago

Thanks for the links, will definitely take a look. I suspect you might be hearing some feedback from us soon w.r.t this trial, assuming the google people we are working with relay that back to you?

vgt 8y ago
I'd love to hear your feedback, so yes please :) Anything you don't like is an opportunity to make our product better