5 ms·

I have always had this fantasy thinking of what happens when outages of one of these major service never come back online i.e. in this outage Slack loses info o

I have always had this fantasy thinking of what happens when outages of one of these major service never come back online i.e. in this outage Slack loses info of all the accounts, users, messages etc. How would people react? What would engineers do to recover? I always found that idea fascinating. Imagine Google saying tomorrow that they lost all the accounts and emails. What kind of impact the world will have?

MattGaiser 6y ago
Google would be catastrophic because so much is stored there. Slack is mostly real time communication, at least for me. There are a few bits and bobs that really should be documented that are in the messages though.
- blntechie 6y ago
  Yeah, Google would easily top the list of companies which can have catastrophic impact. Microsoft, Apple, Salesforce, Dropbox would be the next in the list I guess if we leave out the utility companies and internet providers etc.
  - jon-wood 6y ago
    Just look at the impact a 40 minute outage of Google Auth had last month, I wouldn't be surprised if the global productivity hit during that outage was in the billions of dollars, and that was for a relatively short outage without any data loss.
  - sjg007 6y ago
    AWS outages have basically crippled a few businesses. The longest I know of was 8-10 hours the day before Thanksgiving. Some Bay Area food company got hit by it and couldn’t deliver thanksgiving dinners.
- thesuitonym 6y ago
  If this thread is to be believed, apparently a lot of engineers use slack for alerting and don't know how to check their monitoring software manually.
TwoPhotons 6y ago
That wouldn't be ideal.
JadoJodo 6y ago
My tangential thought in that regard is what if this is a really bad outage that causes Slack to tank (i.e. A large number of companies switch to Microsoft, Zulip, etc). Equally interesting a thought.
codingdave 6y ago
That scenario is what Disaster Recovery plans are for. Every large company I've worked for has had recovery plans in place, including scenarios as disturbing as "All data centers and offices explode simultaneously, and all staff who know how it all works are killed in the blasts." You not only have backups in place, you have documentation in place, including a back-up vendor who has copies of the documentation and can staff up workers to get it up and running again without any help from existing staff. And we tested those scenarios. I'm not sure which dry runs were less fun - when you got paged at 3 AM to go to the DR site and restore the entire infrastructure from scratch... or when you got paged at 3 AM and were instructed to stay home and not communicate with anyone for 24 hours to prove it can be done with out you. (OK, so staying home was definitely more fun, but disturbing.)
- blntechie 6y ago
  That’s some really good thoughts on DR planning. I have never thought DR to be to such an extent. How many companies really plan for an event where their entire infrastructure goes offline and their entire team gets killed? Does even companies like Google plan for this kind of event?
  - codingdave 6y ago
    The two I've worked for that took it that far were a Federal bank, and an energy company. I have no idea how far Google or other large software companies take their plans. But based on my experience, the initial recovery planning is the hard part. The documentation to tell a new team how to do it isn't so painful once the base plan exists, although you do need to think ahead to make sure somebody at your back-up vendor has an account with enough access to set up all the other accounts that will need to be created, including authorization to spend money to make it happen.
  - noir_lord 6y ago
    The last company I worked for where I was (de facto) in charge of IT (small company so I wore lots of hats) could have recovered if both sites burnt down and I got hit by a bus since I made sure that all code, data and instructions to re-up everything existed off site, that both most senior managers understood how to access everything and enough to hand it to a competent firm with a memory stick and a password. In some ways losing your ERP and it's backups would be harder to recover from than both sites burning down, insurance would cover that at least.
tclancy 6y ago
We might start to see actual legislation around implied SLAs in the US which would cause Google to rethink everyone's 20% project being rolled out for 2 years.
ab_io 6y ago
It would be a mass customer extinction event for said service, and would effectively result in a windfall for competing services
- blntechie 6y ago
  Services like Slack are replaceable to most extent. How does even replace a service like Google easily? There are like to like services available for Google but the data is where it becomes tricky. Almost 1bn people losing their email addresses could cause massive issues.
lsaferite 6y ago
This should actually be part of your Disaster Recovery plan. You should have at least some plan for the loss of all of your service providers. Even if that plan is to sit in the corner and cry (j/k).
teagee 6y ago
This reminds me of what happened to the financial services Cantor Fitzgerald after 9/11, just replacing a system with hundreds of lost employees: https://www.nytimes.com/2014/11/19/magazine/the-secret-life-of-passwords.html https://www.nytimes.com/2014/11/19/magazine/the-secret-life-...
- nobody9999 6y ago
  I was at CF (at new offices, obviously) briefly a couple weeks after 9/11. They had backups and were able to recover data and systems. By the time I got there, they were somewhat functional. The biggest problems were the lack of knowledgeable personnel, not lost data or systems.
  - teagee 6y ago
    Thanks for sharing, for some reason I think about this story a lot. It must have been such an emotionally difficult time for everyone involved in piecing back together their processes.
    - nobody9999 6y ago
      >Thanks for sharing, for some reason I think about this story a lot. It must have been such an emotionally difficult time for everyone involved in piecing back together their processes. I was there as a consultant and didn't know anyone there when I went. I won't provide any details out of respect for those fine people, but the grief was so thick, you could have cut it with a knife. As I said, I didn't know anyone who was there (or wasn't there) but after a day, I wanted to cry.
alexsey 6y ago
Being in DR,I live my life wondering about that too. I spend alot of extra time checking accounts and making sure that I print (yes, sneakernet) out important data as well as have manual copies of passwords. Its old school, but it removes the risk to my business in case of a total loss of a global service and lowers the risk of a heart attack and related stress. The rest of the world may not be so energetic re: their accounts and data, so it would be painful for many, it depends on their how much risk they are willing to experience. Being in DR, it is very difficult for businesses to allocate the time and resources to good planning - for many, DR is an insurance policy. Staff: engineering and development are focused on putting out fires however, a real DR is more than most companies can handle if they have not planned accordingly or practiced through testing failover/normalization processes as well as performing component-level testing.
signed0 6y ago
In 2011 a small amount (0.02%) of Gmail users had all their emails deleted due to a bug: https://gmail.googleblog.com/2011/02/gmail-back-soon-for-everyone.html https://gmail.googleblog.com/2011/02/gmail-back-soon-for-eve... They ended up having to restore them from tape backup, which took several days. Affected users also had all their incoming mail bounce for 20 hours.
Macha 6y ago
Happened to Ma.gnolia, which was the number 2 bookmarking site behind Del.icio.us in that era: https://en.wikipedia.org/wiki/Gnolia https://en.wikipedia.org/wiki/Gnolia HN comments at the time: https://news.ycombinator.com/item?id=487497 https://news.ycombinator.com/item?id=487497 The site relaunched a month later and shut down for good a year after that

MattGaiser 6y ago

Google would be catastrophic because so much is stored there. Slack is mostly real time communication, at least for me. There are a few bits and bobs that really should be documented that are in the messages though.

blntechie 6y ago
Yeah, Google would easily top the list of companies which can have catastrophic impact. Microsoft, Apple, Salesforce, Dropbox would be the next in the list I guess if we leave out the utility companies and internet providers etc.
- jon-wood 6y ago
  Just look at the impact a 40 minute outage of Google Auth had last month, I wouldn't be surprised if the global productivity hit during that outage was in the billions of dollars, and that was for a relatively short outage without any data loss.
- sjg007 6y ago
  AWS outages have basically crippled a few businesses. The longest I know of was 8-10 hours the day before Thanksgiving. Some Bay Area food company got hit by it and couldn’t deliver thanksgiving dinners.
thesuitonym 6y ago
If this thread is to be believed, apparently a lot of engineers use slack for alerting and don't know how to check their monitoring software manually.

TwoPhotons 6y ago

That wouldn't be ideal.

JadoJodo 6y ago

My tangential thought in that regard is what if this is a really bad outage that causes Slack to tank (i.e. A large number of companies switch to Microsoft, Zulip, etc). Equally interesting a thought.

codingdave 6y ago

That scenario is what Disaster Recovery plans are for. Every large company I've worked for has had recovery plans in place, including scenarios as disturbing as "All data centers and offices explode simultaneously, and all staff who know how it all works are killed in the blasts." You not only have backups in place, you have documentation in place, including a back-up vendor who has copies of the documentation and can staff up workers to get it up and running again without any help from existing staff. And we tested those scenarios. I'm not sure which dry runs were less fun - when you got paged at 3 AM to go to the DR site and restore the entire infrastructure from scratch... or when you got paged at 3 AM and were instructed to stay home and not communicate with anyone for 24 hours to prove it can be done with out you. (OK, so staying home was definitely more fun, but disturbing.)

blntechie 6y ago
That’s some really good thoughts on DR planning. I have never thought DR to be to such an extent. How many companies really plan for an event where their entire infrastructure goes offline and their entire team gets killed? Does even companies like Google plan for this kind of event?
- codingdave 6y ago
  The two I've worked for that took it that far were a Federal bank, and an energy company. I have no idea how far Google or other large software companies take their plans. But based on my experience, the initial recovery planning is the hard part. The documentation to tell a new team how to do it isn't so painful once the base plan exists, although you do need to think ahead to make sure somebody at your back-up vendor has an account with enough access to set up all the other accounts that will need to be created, including authorization to spend money to make it happen.
- noir_lord 6y ago
  The last company I worked for where I was (de facto) in charge of IT (small company so I wore lots of hats) could have recovered if both sites burnt down and I got hit by a bus since I made sure that all code, data and instructions to re-up everything existed off site, that both most senior managers understood how to access everything and enough to hand it to a competent firm with a memory stick and a password. In some ways losing your ERP and it's backups would be harder to recover from than both sites burning down, insurance would cover that at least.

tclancy 6y ago

We might start to see actual legislation around implied SLAs in the US which would cause Google to rethink everyone's 20% project being rolled out for 2 years.

ab_io 6y ago

It would be a mass customer extinction event for said service, and would effectively result in a windfall for competing services

blntechie 6y ago
Services like Slack are replaceable to most extent. How does even replace a service like Google easily? There are like to like services available for Google but the data is where it becomes tricky. Almost 1bn people losing their email addresses could cause massive issues.

lsaferite 6y ago

This should actually be part of your Disaster Recovery plan. You should have at least some plan for the loss of all of your service providers. Even if that plan is to sit in the corner and cry (j/k).

teagee 6y ago

This reminds me of what happened to the financial services Cantor Fitzgerald after 9/11, just replacing a system with hundreds of lost employees: https://www.nytimes.com/2014/11/19/magazine/the-secret-life-of-passwords.html https://www.nytimes.com/2014/11/19/magazine/the-secret-life-...

nobody9999 6y ago
I was at CF (at new offices, obviously) briefly a couple weeks after 9/11. They had backups and were able to recover data and systems. By the time I got there, they were somewhat functional. The biggest problems were the lack of knowledgeable personnel, not lost data or systems.
- teagee 6y ago
  Thanks for sharing, for some reason I think about this story a lot. It must have been such an emotionally difficult time for everyone involved in piecing back together their processes.
  - nobody9999 6y ago
    >Thanks for sharing, for some reason I think about this story a lot. It must have been such an emotionally difficult time for everyone involved in piecing back together their processes. I was there as a consultant and didn't know anyone there when I went. I won't provide any details out of respect for those fine people, but the grief was so thick, you could have cut it with a knife. As I said, I didn't know anyone who was there (or wasn't there) but after a day, I wanted to cry.

alexsey 6y ago

Being in DR,I live my life wondering about that too. I spend alot of extra time checking accounts and making sure that I print (yes, sneakernet) out important data as well as have manual copies of passwords. Its old school, but it removes the risk to my business in case of a total loss of a global service and lowers the risk of a heart attack and related stress. The rest of the world may not be so energetic re: their accounts and data, so it would be painful for many, it depends on their how much risk they are willing to experience. Being in DR, it is very difficult for businesses to allocate the time and resources to good planning - for many, DR is an insurance policy. Staff: engineering and development are focused on putting out fires however, a real DR is more than most companies can handle if they have not planned accordingly or practiced through testing failover/normalization processes as well as performing component-level testing.

signed0 6y ago

In 2011 a small amount (0.02%) of Gmail users had all their emails deleted due to a bug: https://gmail.googleblog.com/2011/02/gmail-back-soon-for-everyone.html https://gmail.googleblog.com/2011/02/gmail-back-soon-for-eve... They ended up having to restore them from tape backup, which took several days. Affected users also had all their incoming mail bounce for 20 hours.

Macha 6y ago

Happened to Ma.gnolia, which was the number 2 bookmarking site behind Del.icio.us in that era: https://en.wikipedia.org/wiki/Gnolia https://en.wikipedia.org/wiki/Gnolia HN comments at the time: https://news.ycombinator.com/item?id=487497 https://news.ycombinator.com/item?id=487497 The site relaunched a month later and shut down for good a year after that