8 ms·

I remember the earlier Google attempt at the same thing. I was in undergrad and an engineer came to my Uni to give a presentation on the topic. A big area of c

by pbh101 11y ago

I remember the earlier Google attempt at the same thing. I was in undergrad and an engineer came to my Uni to give a presentation on the topic. A big area of concern was "Why won't Google just shut this down when the going gets rough?" The question was mostly batted away along the lines of "this is lunch money to Google anyway." But just a couple years later, it did indeed bite the dust. Covered in Wired: http://www.wired.com/2008/01/google-to-provi/ http://www.wired.com/2008/01/google-to-provi/ Update: And the shutdown article, 11 months later: http://www.wired.com/2008/12/googlescienceda/ http://www.wired.com/2008/12/googlescienceda/

vgt 11y ago
Google BigQuery is a paid service, Dremel under the hood, used internally at Google by ~80% of Googlers every week. It's simply not going away because Google relies on it heavily. Externally, companies like Spotify and Kabam Games rely heavily on BigQuery as well: https://www.youtube.com/watch?v=LTVFg6YOjWo&list=PLIivdWyY5sqLpj5W-tdqT3O2NQ-_LgiPL&index=7 https://www.youtube.com/watch?v=LTVFg6YOjWo&list=PLIivdWyY5s... https://www.youtube.com/watch?v=6Nv18xmJirs&list=PLIivdWyY5sqLpj5W-tdqT3O2NQ-_LgiPL&index=6 https://www.youtube.com/watch?v=6Nv18xmJirs&list=PLIivdWyY5s... So the story of Google Cloud products, which are paid, supported, and Enterprise-grade, and which are covered by a clear 1 year deprecation policy, are very different from the story of free services. Public Datasets is just a function of BigQuery's unique ACL functionality.
- awqrre 11y ago
  Google employees using something internally doesn't mean that it won't go away for everyone else... Also, they could decide to honor existing contracts and stop issuing new ones...
  - vgt 11y ago
    BigQuery is a truly fully-managed service. This means that SREs are on call 24/7. It's not software that one downloads. Hence it really doesn't make a lot of sense to have it available to just a few folks, incremental overhead is low. I'd love to hear industry standards around deprecation of paid services. This is really a way of doing business in the entire world, and I'm not sure why Google is being held to an unreasonable and impossible standard of "never deprecate anything ever!". For example, here is Microsoft's: https://en.wikipedia.org/wiki/Category:Discontinued_Microsoft_software https://en.wikipedia.org/wiki/Category:Discontinued_Microsof...
    - awqrre 11y ago
      I was not blaming anybody, it's possible that a service would be more valuable to a company if it's kept internally.
      - vgt 11y ago
        Ah yes, point well taken. I think we've seen this historically at Google, but things have changed in the past few years.
        ignoramous 11y ago
        Primarily as a reaction to AWS and Azure? Google announced Colossus in 2011, is that going to make it to GCP?
        vgt 11y ago
        You can use Colossus today! It's a very low level service, leveraged by other GCP services. For example, Google Cloud Storage and BigQuery Storage both sit on Colossus. Read more on BQ here: https://cloud.google.com/blog/big-data/2016/01/bigquery-under-the-hood https://cloud.google.com/blog/big-data/2016/01/bigquery-unde...
  - kyrra 11y ago
    BigQuery is cover by the Google Cloud deprecation policy: https://cloud.google.com/terms/deprecation https://cloud.google.com/terms/deprecation Google is required to give at least a 1 year warning on shutting down released Cloud products.
- duncanawoods 11y ago
  >> Externally, companies like Spotify I hate people bringing them up - its just being a sucker for marketing. Spotify and Netflix are clearly brands recruited to promote their respective platforms. The only question is how large a discount they are getting among other special treatment in return for their name. I wouldn't be surprised if its conditional on a gag criticising the platform too. These are the sort of situations that are revealed years later to have required a totally isolated and custom solution that bore no resemblance to the service it was intended to promote. Special partnerships prove nothing because the economics and service you get won't be remotely similar.
  - vgt 11y ago
    You can listen and read Spotify engineers talking about their experiences with Google, including running 2,000,000 qps load tests without ever telling anyone at Google. If you read some of their blogs, you can see how brutally honest they are about the pros and cons of our services: https://labs.spotify.com/2016/03/10/spotifys-event-delivery-the-road-to-the-cloud-part-iii/ https://labs.spotify.com/2016/03/10/spotifys-event-delivery-... https://www.youtube.com/watch?v=LTVFg6YOjWo&index=7&list=PLIivdWyY5sqLpj5W-tdqT3O2NQ-_LgiPL https://www.youtube.com/watch?v=LTVFg6YOjWo&index=7&list=PLI... https://www.youtube.com/watch?v=xT6tQAIywFQ&index=1&list=PLIivdWyY5sqLpj5W-tdqT3O2NQ-_LgiPL https://www.youtube.com/watch?v=xT6tQAIywFQ&index=1&list=PLI... Finally, does this sound like an engineer who got paid to speak? https://twitter.com/mpjme/status/702167231623516160 https://twitter.com/mpjme/status/702167231623516160
- incepted 11y ago
  > It's simply not going away because Google relies on it heavily. Still doesn't mean Google will maintain the public access to this database. They might just as well shut it down and limit its access to Googlers. Like they already did a few years ago.
  - lbn 11y ago
    When you use these datasets you pay for how much the query has to scan. For example, if you do a select on a single 8 byte column with 1 billion rows in the table you are paying for 8GB. Every month the first TB is free but if you go over that you have to pay $5 per TB. They are not exactly doing this for free.
fhoffa 11y ago
2008. I remember those days. GMail was still in beta and ' a big area of concern was "Why won't Google just shut this down when the going gets rough?" ' 8 years later GMail has a billion users, and not going anywhere but up. I haven't been to the future (yet), but I'll happily take any bet you have against BigQuery. Disclaimer: I'm Felipe Hoffa, and I work at Google. (https://twitter.com/felipehoffa https://twitter.com/felipehoffa)
- philovivero 11y ago
  Can you give me an update on Wave and Orkut? I never use this Reader thing, but I hear others do.
  - deleted 11y ago
    [deleted]
  - deleted 11y ago
    [deleted]
  - vgt 11y ago
    I'm happy to respond to every single person that makes this argument in perpetuity :) There's a difference between experimental free services and Enterprise-grade SLA'd SLO'd fully-supported paid services with a very clear and recorded deprecation policy and an army of customers with contracts and full support from Google CEO and chairman.
    - dingaling 11y ago
      Indeed, but who is paying to maintain 546GB of online storage ( plus backups ) for Reddit comments in BigQuery, for example? If the answer is "Google" then I think people are still right to be cautious. Or to invert the question; could I put 1TB of my own 'interesting' data into BigQuery and have Google maintain it in perpetuity for free? If not, then why are any of these datasets considered safer?
      - ucaetano 11y ago

by pbh101 11y ago

vgt 11y ago

Google BigQuery is a paid service, Dremel under the hood, used internally at Google by ~80% of Googlers every week. It's simply not going away because Google relies on it heavily. Externally, companies like Spotify and Kabam Games rely heavily on BigQuery as well: https://www.youtube.com/watch?v=LTVFg6YOjWo&list=PLIivdWyY5sqLpj5W-tdqT3O2NQ-_LgiPL&index=7 https://www.youtube.com/watch?v=LTVFg6YOjWo&list=PLIivdWyY5s... https://www.youtube.com/watch?v=6Nv18xmJirs&list=PLIivdWyY5sqLpj5W-tdqT3O2NQ-_LgiPL&index=6 https://www.youtube.com/watch?v=6Nv18xmJirs&list=PLIivdWyY5s... So the story of Google Cloud products, which are paid, supported, and Enterprise-grade, and which are covered by a clear 1 year deprecation policy, are very different from the story of free services. Public Datasets is just a function of BigQuery's unique ACL functionality.

awqrre 11y ago
Google employees using something internally doesn't mean that it won't go away for everyone else... Also, they could decide to honor existing contracts and stop issuing new ones...
- vgt 11y ago
  BigQuery is a truly fully-managed service. This means that SREs are on call 24/7. It's not software that one downloads. Hence it really doesn't make a lot of sense to have it available to just a few folks, incremental overhead is low. I'd love to hear industry standards around deprecation of paid services. This is really a way of doing business in the entire world, and I'm not sure why Google is being held to an unreasonable and impossible standard of "never deprecate anything ever!". For example, here is Microsoft's: https://en.wikipedia.org/wiki/Category:Discontinued_Microsoft_software https://en.wikipedia.org/wiki/Category:Discontinued_Microsof...
  - awqrre 11y ago
    I was not blaming anybody, it's possible that a service would be more valuable to a company if it's kept internally.
    - vgt 11y ago
      Ah yes, point well taken. I think we've seen this historically at Google, but things have changed in the past few years.
      - ignoramous 11y ago
        Primarily as a reaction to AWS and Azure? Google announced Colossus in 2011, is that going to make it to GCP?
        vgt 11y ago
        You can use Colossus today! It's a very low level service, leveraged by other GCP services. For example, Google Cloud Storage and BigQuery Storage both sit on Colossus. Read more on BQ here: https://cloud.google.com/blog/big-data/2016/01/bigquery-under-the-hood https://cloud.google.com/blog/big-data/2016/01/bigquery-unde...
- kyrra 11y ago
  BigQuery is cover by the Google Cloud deprecation policy: https://cloud.google.com/terms/deprecation https://cloud.google.com/terms/deprecation Google is required to give at least a 1 year warning on shutting down released Cloud products.
duncanawoods 11y ago
>> Externally, companies like Spotify I hate people bringing them up - its just being a sucker for marketing. Spotify and Netflix are clearly brands recruited to promote their respective platforms. The only question is how large a discount they are getting among other special treatment in return for their name. I wouldn't be surprised if its conditional on a gag criticising the platform too. These are the sort of situations that are revealed years later to have required a totally isolated and custom solution that bore no resemblance to the service it was intended to promote. Special partnerships prove nothing because the economics and service you get won't be remotely similar.
- vgt 11y ago
  You can listen and read Spotify engineers talking about their experiences with Google, including running 2,000,000 qps load tests without ever telling anyone at Google. If you read some of their blogs, you can see how brutally honest they are about the pros and cons of our services: https://labs.spotify.com/2016/03/10/spotifys-event-delivery-the-road-to-the-cloud-part-iii/ https://labs.spotify.com/2016/03/10/spotifys-event-delivery-... https://www.youtube.com/watch?v=LTVFg6YOjWo&index=7&list=PLIivdWyY5sqLpj5W-tdqT3O2NQ-_LgiPL https://www.youtube.com/watch?v=LTVFg6YOjWo&index=7&list=PLI... https://www.youtube.com/watch?v=xT6tQAIywFQ&index=1&list=PLIivdWyY5sqLpj5W-tdqT3O2NQ-_LgiPL https://www.youtube.com/watch?v=xT6tQAIywFQ&index=1&list=PLI... Finally, does this sound like an engineer who got paid to speak? https://twitter.com/mpjme/status/702167231623516160 https://twitter.com/mpjme/status/702167231623516160
incepted 11y ago
> It's simply not going away because Google relies on it heavily. Still doesn't mean Google will maintain the public access to this database. They might just as well shut it down and limit its access to Googlers. Like they already did a few years ago.
- lbn 11y ago
  When you use these datasets you pay for how much the query has to scan. For example, if you do a select on a single 8 byte column with 1 billion rows in the table you are paying for 8GB. Every month the first TB is free but if you go over that you have to pay $5 per TB. They are not exactly doing this for free.

fhoffa 11y ago

2008. I remember those days. GMail was still in beta and ' a big area of concern was "Why won't Google just shut this down when the going gets rough?" ' 8 years later GMail has a billion users, and not going anywhere but up. I haven't been to the future (yet), but I'll happily take any bet you have against BigQuery. Disclaimer: I'm Felipe Hoffa, and I work at Google. (https://twitter.com/felipehoffa https://twitter.com/felipehoffa)

philovivero 11y ago
Can you give me an update on Wave and Orkut? I never use this Reader thing, but I hear others do.
- deleted 11y ago
  [deleted]
- deleted 11y ago
  [deleted]
- vgt 11y ago
  I'm happy to respond to every single person that makes this argument in perpetuity :) There's a difference between experimental free services and Enterprise-grade SLA'd SLO'd fully-supported paid services with a very clear and recorded deprecation policy and an army of customers with contracts and full support from Google CEO and chairman.
  - dingaling 11y ago
    Indeed, but who is paying to maintain 546GB of online storage ( plus backups ) for Reddit comments in BigQuery, for example? If the answer is "Google" then I think people are still right to be cautious. Or to invert the question; could I put 1TB of my own 'interesting' data into BigQuery and have Google maintain it in perpetuity for free? If not, then why are any of these datasets considered safer?
    - ucaetano 11y ago