4 ms·

Google BigQuery is a paid service, Dremel under the hood, used internally at Google by ~80% of Googlers every week. It's simply not going away because Google re

by vgt 11y ago

Google BigQuery is a paid service, Dremel under the hood, used internally at Google by ~80% of Googlers every week. It's simply not going away because Google relies on it heavily. Externally, companies like Spotify and Kabam Games rely heavily on BigQuery as well: https://www.youtube.com/watch?v=LTVFg6YOjWo&list=PLIivdWyY5sqLpj5W-tdqT3O2NQ-_LgiPL&index=7 https://www.youtube.com/watch?v=LTVFg6YOjWo&list=PLIivdWyY5s... https://www.youtube.com/watch?v=6Nv18xmJirs&list=PLIivdWyY5sqLpj5W-tdqT3O2NQ-_LgiPL&index=6 https://www.youtube.com/watch?v=6Nv18xmJirs&list=PLIivdWyY5s... So the story of Google Cloud products, which are paid, supported, and Enterprise-grade, and which are covered by a clear 1 year deprecation policy, are very different from the story of free services. Public Datasets is just a function of BigQuery's unique ACL functionality.

awqrre 11y ago
Google employees using something internally doesn't mean that it won't go away for everyone else... Also, they could decide to honor existing contracts and stop issuing new ones...
- vgt 11y ago
  BigQuery is a truly fully-managed service. This means that SREs are on call 24/7. It's not software that one downloads. Hence it really doesn't make a lot of sense to have it available to just a few folks, incremental overhead is low. I'd love to hear industry standards around deprecation of paid services. This is really a way of doing business in the entire world, and I'm not sure why Google is being held to an unreasonable and impossible standard of "never deprecate anything ever!". For example, here is Microsoft's: https://en.wikipedia.org/wiki/Category:Discontinued_Microsoft_software https://en.wikipedia.org/wiki/Category:Discontinued_Microsof...
  - awqrre 11y ago
    I was not blaming anybody, it's possible that a service would be more valuable to a company if it's kept internally.
    - vgt 11y ago
      Ah yes, point well taken. I think we've seen this historically at Google, but things have changed in the past few years.
      - ignoramous 11y ago
        Primarily as a reaction to AWS and Azure? Google announced Colossus in 2011, is that going to make it to GCP?
        vgt 11y ago
        You can use Colossus today! It's a very low level service, leveraged by other GCP services. For example, Google Cloud Storage and BigQuery Storage both sit on Colossus. Read more on BQ here: https://cloud.google.com/blog/big-data/2016/01/bigquery-under-the-hood https://cloud.google.com/blog/big-data/2016/01/bigquery-unde...
- kyrra 11y ago
  BigQuery is cover by the Google Cloud deprecation policy: https://cloud.google.com/terms/deprecation https://cloud.google.com/terms/deprecation Google is required to give at least a 1 year warning on shutting down released Cloud products.
duncanawoods 11y ago
>> Externally, companies like Spotify I hate people bringing them up - its just being a sucker for marketing. Spotify and Netflix are clearly brands recruited to promote their respective platforms. The only question is how large a discount they are getting among other special treatment in return for their name. I wouldn't be surprised if its conditional on a gag criticising the platform too. These are the sort of situations that are revealed years later to have required a totally isolated and custom solution that bore no resemblance to the service it was intended to promote. Special partnerships prove nothing because the economics and service you get won't be remotely similar.
- vgt 11y ago
  You can listen and read Spotify engineers talking about their experiences with Google, including running 2,000,000 qps load tests without ever telling anyone at Google. If you read some of their blogs, you can see how brutally honest they are about the pros and cons of our services: https://labs.spotify.com/2016/03/10/spotifys-event-delivery-the-road-to-the-cloud-part-iii/ https://labs.spotify.com/2016/03/10/spotifys-event-delivery-... https://www.youtube.com/watch?v=LTVFg6YOjWo&index=7&list=PLIivdWyY5sqLpj5W-tdqT3O2NQ-_LgiPL https://www.youtube.com/watch?v=LTVFg6YOjWo&index=7&list=PLI... https://www.youtube.com/watch?v=xT6tQAIywFQ&index=1&list=PLIivdWyY5sqLpj5W-tdqT3O2NQ-_LgiPL https://www.youtube.com/watch?v=xT6tQAIywFQ&index=1&list=PLI... Finally, does this sound like an engineer who got paid to speak? https://twitter.com/mpjme/status/702167231623516160 https://twitter.com/mpjme/status/702167231623516160
incepted 11y ago
> It's simply not going away because Google relies on it heavily. Still doesn't mean Google will maintain the public access to this database. They might just as well shut it down and limit its access to Googlers. Like they already did a few years ago.
- lbn 11y ago
  When you use these datasets you pay for how much the query has to scan. For example, if you do a select on a single 8 byte column with 1 billion rows in the table you are paying for 8GB. Every month the first TB is free but if you go over that you have to pay $5 per TB. They are not exactly doing this for free.

by vgt 11y ago

awqrre 11y ago

Google employees using something internally doesn't mean that it won't go away for everyone else... Also, they could decide to honor existing contracts and stop issuing new ones...

vgt 11y ago
BigQuery is a truly fully-managed service. This means that SREs are on call 24/7. It's not software that one downloads. Hence it really doesn't make a lot of sense to have it available to just a few folks, incremental overhead is low. I'd love to hear industry standards around deprecation of paid services. This is really a way of doing business in the entire world, and I'm not sure why Google is being held to an unreasonable and impossible standard of "never deprecate anything ever!". For example, here is Microsoft's: https://en.wikipedia.org/wiki/Category:Discontinued_Microsoft_software https://en.wikipedia.org/wiki/Category:Discontinued_Microsof...
- awqrre 11y ago
  I was not blaming anybody, it's possible that a service would be more valuable to a company if it's kept internally.
  - vgt 11y ago
    Ah yes, point well taken. I think we've seen this historically at Google, but things have changed in the past few years.
    - ignoramous 11y ago
      Primarily as a reaction to AWS and Azure? Google announced Colossus in 2011, is that going to make it to GCP?
      - vgt 11y ago
        You can use Colossus today! It's a very low level service, leveraged by other GCP services. For example, Google Cloud Storage and BigQuery Storage both sit on Colossus. Read more on BQ here: https://cloud.google.com/blog/big-data/2016/01/bigquery-under-the-hood https://cloud.google.com/blog/big-data/2016/01/bigquery-unde...
kyrra 11y ago
BigQuery is cover by the Google Cloud deprecation policy: https://cloud.google.com/terms/deprecation https://cloud.google.com/terms/deprecation Google is required to give at least a 1 year warning on shutting down released Cloud products.

duncanawoods 11y ago

>> Externally, companies like Spotify I hate people bringing them up - its just being a sucker for marketing. Spotify and Netflix are clearly brands recruited to promote their respective platforms. The only question is how large a discount they are getting among other special treatment in return for their name. I wouldn't be surprised if its conditional on a gag criticising the platform too. These are the sort of situations that are revealed years later to have required a totally isolated and custom solution that bore no resemblance to the service it was intended to promote. Special partnerships prove nothing because the economics and service you get won't be remotely similar.

vgt 11y ago
You can listen and read Spotify engineers talking about their experiences with Google, including running 2,000,000 qps load tests without ever telling anyone at Google. If you read some of their blogs, you can see how brutally honest they are about the pros and cons of our services: https://labs.spotify.com/2016/03/10/spotifys-event-delivery-the-road-to-the-cloud-part-iii/ https://labs.spotify.com/2016/03/10/spotifys-event-delivery-... https://www.youtube.com/watch?v=LTVFg6YOjWo&index=7&list=PLIivdWyY5sqLpj5W-tdqT3O2NQ-_LgiPL https://www.youtube.com/watch?v=LTVFg6YOjWo&index=7&list=PLI... https://www.youtube.com/watch?v=xT6tQAIywFQ&index=1&list=PLIivdWyY5sqLpj5W-tdqT3O2NQ-_LgiPL https://www.youtube.com/watch?v=xT6tQAIywFQ&index=1&list=PLI... Finally, does this sound like an engineer who got paid to speak? https://twitter.com/mpjme/status/702167231623516160 https://twitter.com/mpjme/status/702167231623516160

incepted 11y ago

> It's simply not going away because Google relies on it heavily. Still doesn't mean Google will maintain the public access to this database. They might just as well shut it down and limit its access to Googlers. Like they already did a few years ago.

lbn 11y ago
When you use these datasets you pay for how much the query has to scan. For example, if you do a select on a single 8 byte column with 1 billion rows in the table you are paying for 8GB. Every month the first TB is free but if you go over that you have to pay $5 per TB. They are not exactly doing this for free.