6 ms·
I can assure you that Google has a procedure in place for that.
by l9i 5y ago
I can assure you that Google has a procedure in place for that.
- knorker 5y agoFor those who don't know who he is: l9i would know this. Just clarifying that this is not an Internet nobody guessing.
- sam_lowry_ 5y agoHe is still an anonymous dude to me.
- deleted 5y ago[deleted]
- danhak 5y agoHN Profile -> Personal Website -> LinkedIn -> Over 10 years experience as Google Site Reliability Engineer
- ant6n 5y agoIs the LinkedIn profile linking back to the hn account?
- sam_lowry_ 5y agoSecurity Engineer asking?
- ant6n 5y agoHa, no. It just occured to me that any random hacker news account could link to somebody's personal account and claim authority on some subject.
- e1g 5y agoGoogle SRE for 10 years, ending as the Principal Site Reliability Engineer (L8).
- sulam 5y agos/the// Google has more than 1 L8 SRE.
- jaywalk 5y agoI don't know who either he or you are, so...
- knorker 5y agoI was clarifying his comment, since he didn't mention that this is not a guess, but inside knowledge. I was not trying to establish a trust chain. Take from it what you will.
- astrange 5y agoWhy does it matter if he's guessing or not?
- fragmede 5y agoBecause, it may shock you to know, but sometimes people just go on the Internet and tell lies. No shit Google has plans in place for outages. But what are these plans, are they any good... a respected industry figure who's CV includes being at Google for 10 years doesn't need to go into detail describing the IRC fallback to be believed and trusted that there is such a thing.
- new_guy 5y agoThat's just an 'appeal to authority'. No-one knows or cares who made the statement, it may as well have been 'water is wet', it was useless and adds nothing but noise.
- heartbreak 5y agoThat’s…not what “appeal to authority” means.
- l9i 5y agoI found a comment that was factually incorrect and I felt competent to comment on that. Regrettably, I wrote just one sentence and clicked reply without providing any credentials to back up my claim. Not that I try to hide my identity, as danhak pointed out in https://news.ycombinator.com/item?id=28751644 https://news.ycombinator.com/item?id=28751644, my full name and URL of my personal website are only a click away. I have replied to my initial comment with provide some additonal context: https://news.ycombinator.com/edit?id=28752431 https://news.ycombinator.com/edit?id=28752431. Hope that helps.
- astrange 5y agoI've found that when I post things I learned on the job here it actually causes people to tell me I'm wrong or made it up even more often…
- ric2b 5y agoYup, they make a new chat app if the previous one is down.
- gadnuk 5y agoGoogle Talk, Google Voice, Google Buzz, Google+ Messenger, Hangouts, Spaces, Allo, Hangouts Chat, and Google Messages. At some point, they must run out of names, right?
- andrepd 5y agoYou forgot google meet!
- darkhorn 5y agoAnd Google Wave.
- londons_explore 5y agoYou forgot the chat boxes inside other apps like Google docs, Gmail, YouTube, etc.
- scatters 5y agoAnd Google Pay, apparently.
- mr_toad 5y ago> Yup, they make a new chat app if the previous one is down. Continuous Deployment.
- still_grokking 5y agoI've read here on HN that exactly this was the issue as they had one of the bigger outages (I think it was due to some auth service failure) and GMail didn't accept incoming mail.
- l9i 5y agoA Gmail outage would be barely an inconvenience as Gmail plays a minor role in Google's disaster response. Disclaimer: Ex-Googler who used to work on disaster reponse. Opinions are my own.
- l9i 5y agoI unfortunately cannot edit the parent comment anymore but several people pointed out that I didn't back up my claim or provided any credentials so here they are: Google has multiple independent procedures for coordination during disasters. A global DNS outage (mentioned in https://news.ycombinator.com/item?id=28751140 https://news.ycombinator.com/item?id=28751140) was considered and has been taken into account. I do not attempt to hide my identity here, quite the opposite: my HN profile contains my real name. Until recently a part of my job was to ensure that Google is prepared for various disasterous scenarios and that Googlers can coordinate the response independently from Google's infrastructure. I authored one of the fallback communication procedures that would likely be exercised today if Google's network experienced a global outage. Of course Google has a whole team of fantastic human beings who are deeply involved in disaster preparedness (miss you!). I am pretty sure they are going to analyze what happened to Facebook today in light of Google's emergency plans. While this topic is really fascinating, I am unfortunately not at liberty to disclose the details as they belong to my previous employer. But when I stumble upon factually incorrect comments on HN that I am in a position to correct, why not do that?
- grayfaced 5y agoIn future news: Waymo outage results in engineers unable to get to data center. Engineers don't even know where their servers are.
- shemnon42 5y agoGive us the dirt on how google does it's disaster planning exercises please! Do you do these exercises all at once or slowly over the year?
- donalhunt 5y agoWhy not do both? ;)
- l9i 5y agoInteresting that you are asking for the dirt given that DiRT stands for Disaster and Recovery Testing, at least at Google. Every year there is a DiRT week where hundreds of tests are run. That obviously requires a ton of planning that starts well in advance. The objective is, of course, that despite all the testing nobody outside Google notices anything special. Given the volume and intrusiveness of these tests, the DiRT team is doing quite an impressive job. While the DiRT week is the most intense testing period, disaster preparedness is not limited to just one event per year. There are also plenty tests conducted througout the year, some planned centrally, some done by individual teams. That's in addition to the regular training and exercises that SRE teams are doing periodically. If you are interested in reading more about Google's approach to distaster planning and preparedness, you may be interested in reading the DiRT, or how to get dirty section from Shrinking the time to mitigate production incidents—CRE life lessons (https://cloud.google.com/blog/products/management-tools/shrinking-the-time-to-mitigate-production-incidents https://cloud.google.com/blog/products/management-tools/shri...) and Weathering the Unexpected (https://queue.acm.org/detail.cfm?id=2371516 https://queue.acm.org/detail.cfm?id=2371516).