4 ms·

I worked on the BigQuery team and happy to answer any questions you may have. There are quite a few AWS customers running on BigQuery. (See [0], [1] [7], [8] )

by vgt 10y ago

I worked on the BigQuery team and happy to answer any questions you may have. There are quite a few AWS customers running on BigQuery. (See [0], [1] [7], [8] ) I'll offer some resources: - A session from last Google NEXT, where Yahoo, NY Times, and Blue Apron detail their experience with migrating to BigQuery [0] - A session from previous Google NEXT, where Kabam Games detailed their experience with migrating to BigQuery (and running the rest of their workload on AWS) [1] - A blog "15 things you probably didn't know about BigQuery" [2] - A blog detailing what BigQuery means by "fully managed" [3] - A blog offering some details on most recent iteration of Dremel "In-memory query execution in Google BigQuery" [4] - A blog describing Capacitor, the opinionated storage format that replaced ColumnIO (on which Parquet is based) [5] - "BigQuery Under The Hood" [6] - Since you specifically asked about Load Performance, [9] ( A wholly redundant disclosure that I work at Google Cloud) [0] https://www.youtube.com/watch?v=TLpfGaYWshw https://www.youtube.com/watch?v=TLpfGaYWshw [1] https://www.youtube.com/watch?v=6Nv18xmJirs https://www.youtube.com/watch?v=6Nv18xmJirs [2] https://medium.com/google-cloud/15-awesome-things-you-probably-didnt-know-about-google-bigquery-6654841fa2dc https://medium.com/google-cloud/15-awesome-things-you-probab... [3] https://cloud.google.com/blog/big-data/2016/08/google-bigquery-continues-to-define-what-it-means-to-be-fully-managed https://cloud.google.com/blog/big-data/2016/08/google-bigque... [4] https://cloud.google.com/blog/big-data/2016/08/in-memory-query-execution-in-google-bigquery https://cloud.google.com/blog/big-data/2016/08/in-memory-que... [5] https://cloud.google.com/blog/big-data/2016/04/inside-capacitor-bigquerys-next-generation-columnar-storage-format https://cloud.google.com/blog/big-data/2016/04/inside-capaci... [6] https://cloud.google.com/blog/big-data/2016/01/bigquery-under-the-hood https://cloud.google.com/blog/big-data/2016/01/bigquery-unde... [7] https://cloud.google.com/customers/sharethis/ https://cloud.google.com/customers/sharethis/ [8] https://storify.com/felipehoffa/dow-jones-on-bigquery https://storify.com/felipehoffa/dow-jones-on-bigquery [9] https://medium.com/google-cloud/paying-it-forward-how-bigquerys-data-ingest-breaks-tech-norms-8bfe2341f5eb https://medium.com/google-cloud/paying-it-forward-how-bigque...

paladin314159 10y ago
Thanks for the wealth of resources. After reading through, the only big concern I have with BigQuery is pricing. I like the model of pay-per-query, but it seems like costs can get out of hand easily. These two issues particularly worry me: - No non-date partitioning, which means you'll often end up (in my particular use case) scanning 10-100x as much data as you actually want to. - Pricing is based on uncompressed data size, which is at least several times more expensive than the alternative (e.g. Athena). Any thoughts on these? We'll have several hundreds of TB of data, so cost matters.
- vgt 10y ago
  Happy to help! On partitioning - sounds like a great candidate for sharding. BigQuery standard SQL makes it very easy to run unions on shards (ex. Select * from tables_*) On pricing - BigQuery is a very different product functionally, and I encourage you to compare it via a poc. That said, definitely reach out to me on price.

by vgt 10y ago

paladin314159 10y ago

Thanks for the wealth of resources. After reading through, the only big concern I have with BigQuery is pricing. I like the model of pay-per-query, but it seems like costs can get out of hand easily. These two issues particularly worry me: - No non-date partitioning, which means you'll often end up (in my particular use case) scanning 10-100x as much data as you actually want to. - Pricing is based on uncompressed data size, which is at least several times more expensive than the alternative (e.g. Athena). Any thoughts on these? We'll have several hundreds of TB of data, so cost matters.

vgt 10y ago
Happy to help! On partitioning - sounds like a great candidate for sharding. BigQuery standard SQL makes it very easy to run unions on shards (ex. Select * from tables_*) On pricing - BigQuery is a very different product functionally, and I encourage you to compare it via a poc. That said, definitely reach out to me on price.