6 ms·

I am genuinely surprised that Slack wasn't ready for people to come back from holiday, to view increased queues of unread messages, to have to manually login vs

I am genuinely surprised that Slack wasn't ready for people to come back from holiday, to view increased queues of unread messages, to have to manually login vs. having auth tokens or cookies, etc. Either that, or they had a cosmically coincidental outage on a really bad Monday to have it. It's bad enough team comms go over Slack so much now, at least we have email fallback. What scares me is for the teams that use Slack for system alerting.

djxfade 6y ago
We get our primary alerts through Slack. However we also have SMS and phone call backups through PagerDuty
- djsumdog 6y ago
  We have an alert channel in Slack, but it's mostly ignored. Our primary alerts come via SMS/VictorOps. At one of my old jobs, we had SMS via two physical/hardware devices in our data center. One had a Telstra SIM card and the other had an Optus SIM card. (They were plugged into the same machine, but we had plans to put a second one in another data center before I left). If you really care about alerts, you should have physical hardware doing your SMS messages via two different point-of-presences.
- CSDude 6y ago
  (from Opsgenie) I would imagine it would be the other way around for most people.
dysfunction 6y ago
My coworker's theory was someone was waiting for the holiday's end to deploy something risky. And I'm in that boat of depending on Slack for alerting... in fact my team was also waiting over the holidays to deploy more robust non-Slack-based alerting (in our defense the product is only a few months old and only now starting to scale to any real volume).
- hinkley 6y ago
  Not a bad theory. I used to work for a place that had a FY that ended in summer. We had a lot less problems with stuff being shoveled out the door at Thanksgiving and Christmas because nobody was trying to finish their year-end performance goals over the Holidays. I think what I'm implying is that management creates this issue, but we are complicit.
- spelunker 6y ago
  Yeah, I think it's this rather than load. Slack should be able to handle load fine (probably), but since this is the first weekday post-holidays I imagine some deployment broke something.
- bitbuilder 6y ago
  I wouldn't be surprised if it's actually a combination of a new feature being recently rolled out, along with the sudden spike in load this morning. The holidays are actually the perfect time for Slack to roll out a risky deployment, as it has to be their lowest usage time. So it would make sense if something was pushed out last week or the week before. And everything probably seemed fine. And then this morning they suddenly realize this new feature does not perform under load. And to make matters worse, the new feature has been out long enough to make any sort of rollback very tricky, if not impossible. Which means they'd need engineers to desperately hack out, test and deploy a code fix. If this is the scenario, I do not envy them at all.
  - rrrrrrrrrrrryan 6y ago
    Holidays are a good time for a company to do a risky deployment, but a bad time for an individual employee to do a risky deployment, assuming one doesn't want to work overtime over the holiday fixing things.
    - ellisv 6y ago
      This depends on how easy/difficult the rollback strategy.
    - 6y ago
hinkley 6y ago
The two cliched sources of this problem are 1) someone pushed something out over the holidays that could have waited until January, or 2) peak capacity was negatively affected since the last time a spike happened, nobody had a way to monitor it, and so this has been broken since the end of May. On further reflection, someone will admit that they noticed a notch-up in response times and did not connect the dots.
raverbashing 6y ago
It might have been the increased usage due to the pandemic (since it didn't happen from 2019 to 2020) + the sudden inflow of people at the same time.
- majewsky 6y ago
  This would be a good explanation for an outage in March or April 2020, not so much in January 2021.
  - T-hawk 6y ago
    There may well have been a bigger delta in usage from Sunday Jan 3 to Monday Jan 4 2021, than for any particular pair of days in March-April 2020. Of course 2020 saw an increase, but it was smeared over a week or a month rather than being a big jump in a single day after everyone's holidays.
deleted 6y ago
[deleted]
tqi 6y ago
Do you know that was the root cause or are you making an assumption and running with it?
t-writescode 6y ago
Slack has been in business for several years and has survived several December to January transitions, including several people stopping using their product before Christmas and then returning early January. It seems a bit presumptuous to assume that's at fault here, given their age.
- rrrrrrrrrrrryan 6y ago
  Does it? Don't you think their users might be leaning on it more heavily this year due to working from home?
twblalock 6y ago
Now is a good time to recommend to your engineering org that they should have multiple alerting methods, e.g. Slack plus Pagerduty, or Slack plus email. Hopefully email won't be your backup. I've seen that done. Alerts get filtered and ignored, often by accident.

djxfade 6y ago

We get our primary alerts through Slack. However we also have SMS and phone call backups through PagerDuty

djsumdog 6y ago
We have an alert channel in Slack, but it's mostly ignored. Our primary alerts come via SMS/VictorOps. At one of my old jobs, we had SMS via two physical/hardware devices in our data center. One had a Telstra SIM card and the other had an Optus SIM card. (They were plugged into the same machine, but we had plans to put a second one in another data center before I left). If you really care about alerts, you should have physical hardware doing your SMS messages via two different point-of-presences.
CSDude 6y ago
(from Opsgenie) I would imagine it would be the other way around for most people.

dysfunction 6y ago

My coworker's theory was someone was waiting for the holiday's end to deploy something risky. And I'm in that boat of depending on Slack for alerting... in fact my team was also waiting over the holidays to deploy more robust non-Slack-based alerting (in our defense the product is only a few months old and only now starting to scale to any real volume).

hinkley 6y ago
Not a bad theory. I used to work for a place that had a FY that ended in summer. We had a lot less problems with stuff being shoveled out the door at Thanksgiving and Christmas because nobody was trying to finish their year-end performance goals over the Holidays. I think what I'm implying is that management creates this issue, but we are complicit.
spelunker 6y ago
Yeah, I think it's this rather than load. Slack should be able to handle load fine (probably), but since this is the first weekday post-holidays I imagine some deployment broke something.
bitbuilder 6y ago
I wouldn't be surprised if it's actually a combination of a new feature being recently rolled out, along with the sudden spike in load this morning. The holidays are actually the perfect time for Slack to roll out a risky deployment, as it has to be their lowest usage time. So it would make sense if something was pushed out last week or the week before. And everything probably seemed fine. And then this morning they suddenly realize this new feature does not perform under load. And to make matters worse, the new feature has been out long enough to make any sort of rollback very tricky, if not impossible. Which means they'd need engineers to desperately hack out, test and deploy a code fix. If this is the scenario, I do not envy them at all.
- rrrrrrrrrrrryan 6y ago
  Holidays are a good time for a company to do a risky deployment, but a bad time for an individual employee to do a risky deployment, assuming one doesn't want to work overtime over the holiday fixing things.
  - ellisv 6y ago
    This depends on how easy/difficult the rollback strategy.
  - 6y ago

hinkley 6y ago

The two cliched sources of this problem are 1) someone pushed something out over the holidays that could have waited until January, or 2) peak capacity was negatively affected since the last time a spike happened, nobody had a way to monitor it, and so this has been broken since the end of May. On further reflection, someone will admit that they noticed a notch-up in response times and did not connect the dots.

raverbashing 6y ago

It might have been the increased usage due to the pandemic (since it didn't happen from 2019 to 2020) + the sudden inflow of people at the same time.

majewsky 6y ago
This would be a good explanation for an outage in March or April 2020, not so much in January 2021.
- T-hawk 6y ago
  There may well have been a bigger delta in usage from Sunday Jan 3 to Monday Jan 4 2021, than for any particular pair of days in March-April 2020. Of course 2020 saw an increase, but it was smeared over a week or a month rather than being a big jump in a single day after everyone's holidays.

deleted 6y ago

[deleted]

tqi 6y ago

Do you know that was the root cause or are you making an assumption and running with it?

t-writescode 6y ago

Slack has been in business for several years and has survived several December to January transitions, including several people stopping using their product before Christmas and then returning early January. It seems a bit presumptuous to assume that's at fault here, given their age.

rrrrrrrrrrrryan 6y ago
Does it? Don't you think their users might be leaning on it more heavily this year due to working from home?

twblalock 6y ago

Now is a good time to recommend to your engineering org that they should have multiple alerting methods, e.g. Slack plus Pagerduty, or Slack plus email. Hopefully email won't be your backup. I've seen that done. Alerts get filtered and ignored, often by accident.