4 ms·

The most disturbing part of this incident--to me--is that Google Search/YouTube/GMail/etc never went down. Not even a blip. That means that Google is not dogfo

The most disturbing part of this incident--to me--is that Google Search/YouTube/GMail/etc never went down. Not even a blip. That means that Google is not dogfooding GCE to the degree that I would hope and expect for me to risk my business on using the service. Disappointing, to say the least. Say what you will about AWS (and I've said many critical things, and will continue to do so, publicly and privately), but when AWS has a major outage it also affects Amazon digital products. They have major skin in the game, while it appears that GCE is a special snowflake service completely separate from the important, money-making services at Google.

manigandham 10y ago
This always comes up but Google has been pretty explicit in saying that they share some internal infrastructure but that lots of existing stuff isnt running on GCP. They have decades of customized software/hardware running already. GCP is pretty new and there are plenty of big customers running on GCP that you can reference like Spotify and Snapchat. Also it is a rather important and money-making service, on track to potentially eclipse their entire ad business.
- bkeroack 10y ago
  Just because they say it doesn't mean it's good, or even acceptable. "[O]n track to potentially eclipse..." sounds like the worst form of quarterly-report spin.
  - manigandham 10y ago
    What? Good/acceptable in what sense? This is just the size of the market, cloud computing is already a major industry and has just started. The potential upside for a major cloud player dwarfs the entirety of digital advertising.
Bartweiss 10y ago
At a certain point, I think Google could probably defend this even to outsiders with "we're too important". It might be good if Youtube or something else big-but-tangential was on the line when GCE changes went out. Search, though? It dropped globally for 2 minutes in 2013, and took out 40% of internet traffic in the process. There's a real argument that it's critical infrastructure like little else, since Google and Bing are the way most users have of reaching sites other than the core Facebook-Amazon-Yahoo locations. More dogfooding wouldn't be a bad change overall, but people counting on Search and Gmail would probably prefer a minimal-downtime solution no matter how it's implemented.
- bkeroack 10y ago
  Oh I totally get why they do it. When you're an exec in charge of a ~$75B revenue stream[0] when the GCE people come to you I totally get saying "Nah, let's leave it alone for now". But like Steve Yegge said in his classic rant[1], when you're behind in the market you have to launch your cloud product (which is really just a subset of the "platform" problem Yegge was actually writing about) "...all at once, for real, no cheating". If you don't have the organizational courage to take big risks, well I guess you deserve your 2.5% market share[2]. 0. http://marketingland.com/google-revenues-beat-expectations-with-21-3-billion-in-q4-and-75-billion-in-2015-162152 http://marketingland.com/google-revenues-beat-expectations-w... 1. https://gist.github.com/chitchcock/1281611 https://gist.github.com/chitchcock/1281611 2. http://www.datacenterknowledge.com/wp-content/uploads/2016/06/IaaS-market-piechart-2015-Structure.jpg http://www.datacenterknowledge.com/wp-content/uploads/2016/0...
stickfigure 10y ago
Search/YouTube/GMail/etc predate GCE by many years. In the case of Search, almost two decades. You can't expect them to rewrite everything every time a new tool comes out.
boulos 10y ago
There was internal loss, but it was the combo of these: > resulted in announcing some Google Cloud Platform IP addresses from a single point of presence in the southwestern US. > not yet configured to handle Cloud Platform traffic and applied an overly-restrictive packet filter. that caused the real pain. Other traffic that ended up in the Southwest would have routed crazily, but wouldn't get stopped (filter). If it had just been a route thing for GCE (and related services) we would have just seen slowdown, etc. like any other routing mistake. Disclosure (and Disclaimer): I work on Compute Engine, but I'm not on our networking teams.

manigandham 10y ago

This always comes up but Google has been pretty explicit in saying that they share some internal infrastructure but that lots of existing stuff isnt running on GCP. They have decades of customized software/hardware running already. GCP is pretty new and there are plenty of big customers running on GCP that you can reference like Spotify and Snapchat. Also it is a rather important and money-making service, on track to potentially eclipse their entire ad business.

bkeroack 10y ago
Just because they say it doesn't mean it's good, or even acceptable. "[O]n track to potentially eclipse..." sounds like the worst form of quarterly-report spin.
- manigandham 10y ago
  What? Good/acceptable in what sense? This is just the size of the market, cloud computing is already a major industry and has just started. The potential upside for a major cloud player dwarfs the entirety of digital advertising.

Bartweiss 10y ago

At a certain point, I think Google could probably defend this even to outsiders with "we're too important". It might be good if Youtube or something else big-but-tangential was on the line when GCE changes went out. Search, though? It dropped globally for 2 minutes in 2013, and took out 40% of internet traffic in the process. There's a real argument that it's critical infrastructure like little else, since Google and Bing are the way most users have of reaching sites other than the core Facebook-Amazon-Yahoo locations. More dogfooding wouldn't be a bad change overall, but people counting on Search and Gmail would probably prefer a minimal-downtime solution no matter how it's implemented.

bkeroack 10y ago
Oh I totally get why they do it. When you're an exec in charge of a ~$75B revenue stream[0] when the GCE people come to you I totally get saying "Nah, let's leave it alone for now". But like Steve Yegge said in his classic rant[1], when you're behind in the market you have to launch your cloud product (which is really just a subset of the "platform" problem Yegge was actually writing about) "...all at once, for real, no cheating". If you don't have the organizational courage to take big risks, well I guess you deserve your 2.5% market share[2]. 0. http://marketingland.com/google-revenues-beat-expectations-with-21-3-billion-in-q4-and-75-billion-in-2015-162152 http://marketingland.com/google-revenues-beat-expectations-w... 1. https://gist.github.com/chitchcock/1281611 https://gist.github.com/chitchcock/1281611 2. http://www.datacenterknowledge.com/wp-content/uploads/2016/06/IaaS-market-piechart-2015-Structure.jpg http://www.datacenterknowledge.com/wp-content/uploads/2016/0...

stickfigure 10y ago

Search/YouTube/GMail/etc predate GCE by many years. In the case of Search, almost two decades. You can't expect them to rewrite everything every time a new tool comes out.

boulos 10y ago

There was internal loss, but it was the combo of these: > resulted in announcing some Google Cloud Platform IP addresses from a single point of presence in the southwestern US. > not yet configured to handle Cloud Platform traffic and applied an overly-restrictive packet filter. that caused the real pain. Other traffic that ended up in the Southwest would have routed crazily, but wouldn't get stopped (filter). If it had just been a route thing for GCE (and related services) we would have just seen slowdown, etc. like any other routing mistake. Disclosure (and Disclaimer): I work on Compute Engine, but I'm not on our networking teams.