4 ms·

Google BigQuery has pure separation of storage and compute, which allows for Public Datasets [0] to exist in ready-to-query highly optimized format. Run SQL imm

by vgt 10y ago

Google BigQuery has pure separation of storage and compute, which allows for Public Datasets [0] to exist in ready-to-query highly optimized format. Run SQL immediately! BigQuery has a perpetual free query tier of 1 Terabyte per month ($5). In addition, you can a get $300 in Google Cloud credits for two months to do more work [1]. (work on Google Cloud and used to work on BigQuery) Edit: spaces, how do they work?!?! [0] https://cloud.google.com/bigquery/public-data/ https://cloud.google.com/bigquery/public-data/ [1] https://cloud.google.com/free-trial/ https://cloud.google.com/free-trial/

dsl 10y ago
Are there any plans to bring predictable pricing (i.e. reserved slots) to the sub $40k/mo user? I'm trying to bootstrap an idea for a startup dealing with a massive dataset, but the fear of future costs is a major blocker of building around BQ. The pricing model for Azure SQL Data Warehouse is amazing. You buy a fixed number of slots and can find your own sweet spot of response time/cost, but still has a $1k/mo barrier to entry.
- vgt 10y ago
  feel free to pm me.
  - ddorian43 10y ago
    Damn, and the whole dataset for 1 query is in memory. Talk about fast network. Any other juice for the internals ?
    - mikecb 10y ago
      Juice: [1] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43119.pdf https://static.googleusercontent.com/media/research.google.c... [2] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/37217.pdf https://static.googleusercontent.com/media/research.google.c... [3] https://cloud.google.com/blog/big-data/2016/01/bigquery-under-the-hood https://cloud.google.com/blog/big-data/2016/01/bigquery-unde... [4] https://cloud.google.com/blog/big-data/2016/08/in-memory-query-execution-in-google-bigquery https://cloud.google.com/blog/big-data/2016/08/in-memory-que... [5] https://cloud.google.com/blog/big-data/2016/04/inside-capacitor-bigquerys-next-generation-columnar-storage-format https://cloud.google.com/blog/big-data/2016/04/inside-capaci... I think there are more, but these are the ones I have on pinboard.
      - vgt 10y ago
        Some more: [0] https://cloud.google.com/blog/big-data/2016/01/anatomy-of-a-bigquery-query https://cloud.google.com/blog/big-data/2016/01/anatomy-of-a-... [1] https://cloud.google.com/blog/big-data/2016/08/google-bigquery-continues-to-define-what-it-means-to-be-fully-managed https://cloud.google.com/blog/big-data/2016/08/google-bigque... [2] https://medium.com/@thetinot/paying-it-forward-how-bigquerys-data-ingest-breaks-tech-norms-8bfe2341f5eb https://medium.com/@thetinot/paying-it-forward-how-bigquerys... [3] https://medium.com/google-cloud/15-awesome-things-you-probably-didnt-know-about-google-bigquery-6654841fa2dc https://medium.com/google-cloud/15-awesome-things-you-probab... [4] https://cloud.google.com/blog/big-data/2016/02/visualizing-the-mechanics-of-on-demand-pricing-in-big-data-technologies https://cloud.google.com/blog/big-data/2016/02/visualizing-t... [5] https://cloud.google.com/blog/big-data/2016/02/understanding-bigquerys-rapid-scaling-and-simple-pricing https://cloud.google.com/blog/big-data/2016/02/understanding...

by vgt 10y ago

dsl 10y ago

Are there any plans to bring predictable pricing (i.e. reserved slots) to the sub $40k/mo user? I'm trying to bootstrap an idea for a startup dealing with a massive dataset, but the fear of future costs is a major blocker of building around BQ. The pricing model for Azure SQL Data Warehouse is amazing. You buy a fixed number of slots and can find your own sweet spot of response time/cost, but still has a $1k/mo barrier to entry.

vgt 10y ago
feel free to pm me.
- ddorian43 10y ago
  Damn, and the whole dataset for 1 query is in memory. Talk about fast network. Any other juice for the internals ?
  - mikecb 10y ago
    Juice: [1] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43119.pdf https://static.googleusercontent.com/media/research.google.c... [2] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/37217.pdf https://static.googleusercontent.com/media/research.google.c... [3] https://cloud.google.com/blog/big-data/2016/01/bigquery-under-the-hood https://cloud.google.com/blog/big-data/2016/01/bigquery-unde... [4] https://cloud.google.com/blog/big-data/2016/08/in-memory-query-execution-in-google-bigquery https://cloud.google.com/blog/big-data/2016/08/in-memory-que... [5] https://cloud.google.com/blog/big-data/2016/04/inside-capacitor-bigquerys-next-generation-columnar-storage-format https://cloud.google.com/blog/big-data/2016/04/inside-capaci... I think there are more, but these are the ones I have on pinboard.
    - vgt 10y ago
      Some more: [0] https://cloud.google.com/blog/big-data/2016/01/anatomy-of-a-bigquery-query https://cloud.google.com/blog/big-data/2016/01/anatomy-of-a-... [1] https://cloud.google.com/blog/big-data/2016/08/google-bigquery-continues-to-define-what-it-means-to-be-fully-managed https://cloud.google.com/blog/big-data/2016/08/google-bigque... [2] https://medium.com/@thetinot/paying-it-forward-how-bigquerys-data-ingest-breaks-tech-norms-8bfe2341f5eb https://medium.com/@thetinot/paying-it-forward-how-bigquerys... [3] https://medium.com/google-cloud/15-awesome-things-you-probably-didnt-know-about-google-bigquery-6654841fa2dc https://medium.com/google-cloud/15-awesome-things-you-probab... [4] https://cloud.google.com/blog/big-data/2016/02/visualizing-the-mechanics-of-on-demand-pricing-in-big-data-technologies https://cloud.google.com/blog/big-data/2016/02/visualizing-t... [5] https://cloud.google.com/blog/big-data/2016/02/understanding-bigquerys-rapid-scaling-and-simple-pricing https://cloud.google.com/blog/big-data/2016/02/understanding...