4 ms·

how do people at Slack communicate to fix an outage when Slack is down?

by mepiethree 6y ago

leeoniya 6y ago
Discord?
slajax 6y ago
Freenode
forgot_old_user 6y ago
pigeons? But seriously though, just cell phone I guess
- Threeve303 6y ago
  Yeah, The procedure is outlined in RFC 2549
flatiron 6y ago
My guess is a DR XMPP implementation.
ggregoire 6y ago
staging environment?
vlan0 6y ago
irc :)
deleted 6y ago
[deleted]
manquer 6y ago
Ideally your DR and emergency response systems should be not hosted on your infra and have circular dependencies . i.e.Amazon should not be hosting status page assets on S3 as they were. At the very least they should be hosted isolated of your production service components at all layers, so both don't go down at the same time. For slack my guess is their internal slack is not hosted on slack prod and hopefully also does not share components below L4 as well ( DNS, IP , domain, routing etc )
- lucis 6y ago
  https://www.theregister.co.uk/2017/03/01/aws_s3_outage/ https://www.theregister.co.uk/2017/03/01/aws_s3_outage/
- jeffbee 6y ago
  If the disaster involves your connectivity from within your infra to without, it could be advantageous to have backup communications hosted internally. For e.g. an internal IRC for panic situations. The advantages of this might vary depending on whether your response happens from the office, home, or wherever. Pandemic might have changed the equation.
n_jd 6y ago
From https://increment.com/on-call/when-the-pager-goes-off/ https://increment.com/on-call/when-the-pager-goes-off/ : Slack uses its own product for all coordination during incidents and outages, so what happens when it goes down? “To the best of our ability, we use Slack,” says Richard Crowley, Director of Operations at Slack, “[but] If we’re really having a bad day, we use Skype.”
- partiallypro 6y ago
  Would be interesting to see Slack subscribe to MS Teams as an internal fail over and vice-versa
  - CubsFan1060 6y ago
    I believe that’s exactly what cell providers do.
deleted 6y ago
[deleted]
deleted 6y ago
[deleted]
romwell 6y ago
There are seven responses to this comment at the moment, and none of them is email. Fair, I guess.
markshepard 6y ago
Teams (or) airsend? :)
sethammons 6y ago
At SendGrid, years ago, we suffered an outage and did not realize as fast as we should have because our alerting provider used us for notifying about outages :p. This situation can no longer happen, but it was eye opening.
- BluePen7 6y ago
  That's a funny anecdote and a good step, but you guys still have a long way to go in regards to outage alerting at SendGrid. I was annoyed when after a several hour long outage SendGrid still had not updated their status page to reflect an issue. But I was even more annoyed when 5+ hours into the outage, a different support person told me they aren't aware of any outage because their status page does not reflect one. If I sound bitter it's because I am, I was the on call person who got woken up in the wee hours of the morning to deal with this. I'm referring to the outage you guys had in December if you're wondering.
  - sethammons 6y ago
    Real sorry to hear that, and surprised. I’ll look into this. If you wouldn’t mind emailing me more details, that would help.
    - zenexer 6y ago
      I had the exact same experience. I was on vacation with family, but was on call. There was a bug on SendGrid’s end that only affected a small percentage of emails; in particular, they had the wrong DKIM signature, if I remember correctly. I spent the entire vacation trying to convince SendGrid engineers that it wasn’t a configuration issue on our end. They were rather rude and kept insisting that it wasn’t an issue on their end. Eventually we gave up and migrated away from SendGrid. If I remember correctly, they did fix it, at least partially. I think it had something to do with one of their prod servers running a dev config, resulting in DKIM signatures for a SendGrid-owned test domain—something along those lines. That may have been a different incident, though. That was just the straw that broke the camel’s back. There were often issues that only affected one or two servers—we were often told that a server didn’t have an up-to-date configuration or something like that. Each time, it was the same process: convince them it wasn’t an issue on our end, escalate it repeatedly until it lands on an engineer who understands the issue, they fix it in an hour after wasting days of my time insisting that yes, I know what a DNS record is. And the whole time, some (small) percentage of emails are failing. I can only provide so many logs, DNS records, and email headers to demonstrate the issue; at some point, someone has to be able to look at it and go, “well, yes, that’s very obviously the wrong value, and nothing a customer sets on their end should ever have that effect.” Sure, Slack dropped the ball here, but at least they updated their status page. Even for small issues, when I contact them, they either get it resolved or tell me it’s on their to-do list—no need to convince people that I understand basic concepts.

how do people at Slack communicate to fix an outage when Slack is down?

leeoniya 6y ago

Discord?

slajax 6y ago

Freenode

forgot_old_user 6y ago

pigeons? But seriously though, just cell phone I guess

Threeve303 6y ago
Yeah, The procedure is outlined in RFC 2549

flatiron 6y ago

My guess is a DR XMPP implementation.

ggregoire 6y ago

staging environment?

vlan0 6y ago

irc :)

deleted 6y ago

[deleted]

manquer 6y ago

Ideally your DR and emergency response systems should be not hosted on your infra and have circular dependencies . i.e.Amazon should not be hosting status page assets on S3 as they were. At the very least they should be hosted isolated of your production service components at all layers, so both don't go down at the same time. For slack my guess is their internal slack is not hosted on slack prod and hopefully also does not share components below L4 as well ( DNS, IP , domain, routing etc )

lucis 6y ago
https://www.theregister.co.uk/2017/03/01/aws_s3_outage/ https://www.theregister.co.uk/2017/03/01/aws_s3_outage/
jeffbee 6y ago
If the disaster involves your connectivity from within your infra to without, it could be advantageous to have backup communications hosted internally. For e.g. an internal IRC for panic situations. The advantages of this might vary depending on whether your response happens from the office, home, or wherever. Pandemic might have changed the equation.

n_jd 6y ago

From https://increment.com/on-call/when-the-pager-goes-off/ https://increment.com/on-call/when-the-pager-goes-off/ : Slack uses its own product for all coordination during incidents and outages, so what happens when it goes down? “To the best of our ability, we use Slack,” says Richard Crowley, Director of Operations at Slack, “[but] If we’re really having a bad day, we use Skype.”

partiallypro 6y ago
Would be interesting to see Slack subscribe to MS Teams as an internal fail over and vice-versa
- CubsFan1060 6y ago
  I believe that’s exactly what cell providers do.

deleted 6y ago

[deleted]

deleted 6y ago

[deleted]

romwell 6y ago

There are seven responses to this comment at the moment, and none of them is email. Fair, I guess.

markshepard 6y ago

Teams (or) airsend? :)

sethammons 6y ago

At SendGrid, years ago, we suffered an outage and did not realize as fast as we should have because our alerting provider used us for notifying about outages :p. This situation can no longer happen, but it was eye opening.

BluePen7 6y ago
That's a funny anecdote and a good step, but you guys still have a long way to go in regards to outage alerting at SendGrid. I was annoyed when after a several hour long outage SendGrid still had not updated their status page to reflect an issue. But I was even more annoyed when 5+ hours into the outage, a different support person told me they aren't aware of any outage because their status page does not reflect one. If I sound bitter it's because I am, I was the on call person who got woken up in the wee hours of the morning to deal with this. I'm referring to the outage you guys had in December if you're wondering.
- sethammons 6y ago
  Real sorry to hear that, and surprised. I’ll look into this. If you wouldn’t mind emailing me more details, that would help.
  - zenexer 6y ago
    I had the exact same experience. I was on vacation with family, but was on call. There was a bug on SendGrid’s end that only affected a small percentage of emails; in particular, they had the wrong DKIM signature, if I remember correctly. I spent the entire vacation trying to convince SendGrid engineers that it wasn’t a configuration issue on our end. They were rather rude and kept insisting that it wasn’t an issue on their end. Eventually we gave up and migrated away from SendGrid. If I remember correctly, they did fix it, at least partially. I think it had something to do with one of their prod servers running a dev config, resulting in DKIM signatures for a SendGrid-owned test domain—something along those lines. That may have been a different incident, though. That was just the straw that broke the camel’s back. There were often issues that only affected one or two servers—we were often told that a server didn’t have an up-to-date configuration or something like that. Each time, it was the same process: convince them it wasn’t an issue on our end, escalate it repeatedly until it lands on an engineer who understands the issue, they fix it in an hour after wasting days of my time insisting that yes, I know what a DNS record is. And the whole time, some (small) percentage of emails are failing. I can only provide so many logs, DNS records, and email headers to demonstrate the issue; at some point, someone has to be able to look at it and go, “well, yes, that’s very obviously the wrong value, and nothing a customer sets on their end should ever have that effect.” Sure, Slack dropped the ball here, but at least they updated their status page. Even for small issues, when I contact them, they either get it resolved or tell me it’s on their to-do list—no need to convince people that I understand basic concepts.