43 ms·
Google Compute Engine Incident #16015
- petewailes 10y agoThere's a refreshing humility in admitting how it went wrong, and what they've done to fix it, and not mentioning any names that I admire here. No attempting to lay the blame at someone's feet, just admitting that a process failed, and they've amended it so it doesn't happen again. Kudos.
- nine_k 10y agoGoogle's culture of blameless post-mortems is commendable. In internal post-mortems names are named when necessary for disambiguation, without consequences to the persons named. This results in all participants being candid, and the real source of the problem more easily uncovered. (I wrote two post-mortems for small incidents I triggered.)
- BinaryIdiot 10y ago> without consequences to the persons named I would hope so (everyone makes mistakes some of which may lead to HUGE issues) and not giving them a chance to learn from it is a bit unrealistic. At the same time, however, I have been part of organizations where something has gone wrong and us engineers basically all had to "take the blame" otherwise the single person likely would have been fired. I'm confident Google is not like that but it's always something I think about.
- ben_jones 10y agoWe live in a culture where we invoke a draconian system of punishment as a "justice" system. The idea, which does not translate to software, is that fear will keep the local systems in line. Fear of this battle station. So the fact that certain companies recognize a huge societal flaw and actively operate against it is rare (imo).
- 0xmohit 10y agoYes, you do expect transparency from such service providers. It sucks when somebody starts to point you at their SLA when you ask them about outages. I'm looking at you, AWS.
- hehheh 10y agoI've seen Google ask for an NDA before acknowledging a problem existed and was fixed. They are at best selectively transparent.
- josephb 10y agoSome detailed info which is great, however I am more interested in knowing 1) why, or what, led to the wrong decision being made, and 2) why it took so long to notice / revert.
- deleted 10y ago[deleted]
- d4l3k 10y agoThese configurations are absurdly finicky. If you remember a couple of years ago, Pakistan took down youtube to a similar border gateway protocol misconfiguration. These things are pretty quick to be caught, but the configuration changes take time to propagate (in this case 30m). http://www.cnet.com/news/how-pakistan-knocked-youtube-offline-and-how-to-make-sure-it-never-happens-again/ http://www.cnet.com/news/how-pakistan-knocked-youtube-offlin...
- zzzcpan 10y agoNetworking is so fragile. I feel like DNS failover to another AS in another datacenter is the only solution for web services to actually have resilience against such failures.
- toomuchtodo 10y agoOps engineer here. You are correct.
- diafygi 10y agoI'm a little confused about the mechanics of how dns failover works. Would you care to explain a bit more?
- spotman 10y agoThere is a few things possible here. You can do global load balancing which will serve a DNS entry for a hostname based on business rules. Those business rules can contain things like geo targeting, and/or they can also contain health checks. So you could direct traffic via dns based on if a region is online at all, or more granular things like is it meeting performance criteria, etc. Even more simple, is DNS round robin (just putting more than one entry in the reply for a dns request). So in terms of failover, if one of these servers in the reply stops working, generally another one is, and browsers and other devices generally understand that they can try all addresses listed for a given host.
- deleted 10y ago[deleted]
- ben_jones 10y agoIsn't DNS cached at some level on the users machines such that if a particular hostname went down clients which cached that hostname will still be directed to it? FWIW I'm making it a thing today to read up on DNS as this post made me woefully aware of how little I understand it. So feel free to skip this question and I'll probably answer it myself tonight.
- jread 10y agoAccording to network checks we have running against VMs in each GCE region, this event resulted in about 1.6 hours of concurrent ICMP timeouts for every region (except us-west1 for only 10 minutes). We use Panopta for monitoring which verifies outages from multiple external network paths. When outages trigger we also use Ripe Atlas to confirm them using 100s of last mile network paths of which 85-95% resulted in timeouts. This is the second global GCE networking outage this year. These outages are particularly problematic because even multi-region load balancing will not avert downtime. https://cloudharmony.com/status-for-google https://cloudharmony.com/status-for-google Prior global outage - April 11: https://status.cloud.google.com/incident/compute/16007 https://status.cloud.google.com/incident/compute/16007 Disclaimer: I am the founder of CloudHarmony Edit: Outages triggered due to ICMP timeouts
- sshykes 10y agoNot accusing you of being intentionally misleading, but it would be nice if you put a clear disclaimer pointing out that you work for CloudHarmony.
- jread 10y agoI founded CloudHarmony and now work for Garter which acquired CloudHarmony in 2015.
- simonebrunozzi 10y agoI think it would still be nice to mention that you work there.
- deleted 10y ago[deleted]
- jread 10y agoIs there a reason for the downvotes?
- slau 10y ago
- itaysk 10y agoShit happens for everyone, and this kind of error could have happened for other clouds as well. It's the responsibility towards customers and lessons learned that matter.
- piinbinary 10y agoIt seems like most work done to make distributed systems reliable is aimed at handling machines or groups of machines going down (e.g. the leader node in one region goes down at the same time as an entire other region). This half seems to be a solved problem. The postmortems published by Google, Amazon, and Azure (as well as postmortems internal to the company I work for) are nearly always due to some type of change (code or configuration) being rolled out. It seems to me that we need some help from computers to make these systems reliable. - something like static type checking in a programming language, but applied to a distributed system. Perhaps your architecture wouldn't "compile" if the network traffic will go the wrong place, or if a rate limit is above the capacity something is expected to handle, or if the change would impact too many servers at once.
- foolfoolz 10y agoat the platform-wide incident level its more often configuration than code
- BinaryIdiot 10y ago> Perhaps your architecture wouldn't "compile" if the network traffic will go the wrong place, or if a rate limit is above the capacity something is expected to handle, or if the change would impact too many servers at once. This is an interesting idea. I'm always terrified when I have to deploy a minor configuration change into a production system that gets distributed out; if there was an easy way to apply some sort of check against all of it (you know without having to build something explicitly to do this) that would be awesome. I wonder if the future of something like this might be using containers. Each piece that gets deployed is in its own container networked together and you stand it up in a staging environment, ensure it's talking together correctly through some sort of set of integration tests then push it into the production environment.
- lima 10y agoIn many cases you can do staged or partial rollouts, observe error rates or whatever suitable metrics you have, and then continue with the next bunch of servers.
- bkeroack 10y agoThe most disturbing part of this incident--to me--is that Google Search/YouTube/GMail/etc never went down. Not even a blip. That means that Google is not dogfooding GCE to the degree that I would hope and expect for me to risk my business on using the service. Disappointing, to say the least. Say what you will about AWS (and I've said many critical things, and will continue to do so, publicly and privately), but when AWS has a major outage it also affects Amazon digital products. They have major skin in the game, while it appears that GCE is a special snowflake service completely separate from the important, money-making services at Google.
- manigandham 10y agoThis always comes up but Google has been pretty explicit in saying that they share some internal infrastructure but that lots of existing stuff isnt running on GCP. They have decades of customized software/hardware running already. GCP is pretty new and there are plenty of big customers running on GCP that you can reference like Spotify and Snapchat. Also it is a rather important and money-making service, on track to potentially eclipse their entire ad business.
- bkeroack 10y agoJust because they say it doesn't mean it's good, or even acceptable. "[O]n track to potentially eclipse..." sounds like the worst form of quarterly-report spin.
- manigandham 10y agoWhat? Good/acceptable in what sense? This is just the size of the market, cloud computing is already a major industry and has just started. The potential upside for a major cloud player dwarfs the entirety of digital advertising.
- Bartweiss 10y agoAt a certain point, I think Google could probably defend this even to outsiders with "we're too important". It might be good if Youtube or something else big-but-tangential was on the line when GCE changes went out. Search, though? It dropped globally for 2 minutes in 2013, and took out 40% of internet traffic in the process. There's a real argument that it's critical infrastructure like little else, since Google and Bing are the way most users have of reaching sites other than the core Facebook-Amazon-Yahoo locations. More dogfooding wouldn't be a bad change overall, but people counting on Search and Gmail would probably prefer a minimal-downtime solution no matter how it's implemented.
- karmicthreat 10y agoSo in instances like this where everything goes wrong. Does google have the equivalent of a revert button to undo whatever infrastructure changes were done?
- asuffield 10y ago(Tedious disclaimer: my opinion only, not speaking for anybody else. I'm an SRE at Google. My team is oncall for this service and I know exactly what happened here; I probably can't answer most questions you might have.) Let's go with "yes", as the most accurate answer. As soon as I or whoever is oncall has figured out what change was responsible, we can usually revert it quickly and easily. Usually, if I'm oncall and I have reason to even suspect a recent change might be the cause, I'll revert it and see if the problem goes away. The difficulty becomes more apparent when you realise the sheer number of infrastructure changes being made every hour, some of which will be fixes to other outages, and some of which will be things you can't revert because they are of the form "that location has fallen offline; probably lost networking" or "we are now at peak time and there are more users online". So if your question is "can we just roll the whole world back one day" - no, too much has changed in that time.
- karmicthreat 10y agoI know it late but thanks for this. It kind of reinforces the amazing size of the system and the number of people making changes to it.
- RantyDave 10y agoI'm constantly amazed by the competence of these 'big cloud' teams, I can't imagine the amount of preparatory work they put in. It does, however, make it clear how badly the Internet is starting to creak.
- pjlegato 10y agoAttention startups: THIS is what your public postmortems should look like. This level of detail and honesty, this level of specific steps that are being taken to prevent recurrence. 75% of startups try to sweep downtime under the rug and don't even acknowledge problems happened at all -- or only do so privately in a personal support email after hours of interrogation. Another 20% just write "we had some minor issues but everything is fine now." Be like Google.