3 ms·

Having been in this situation before, with a totally-down-and-not-coming-back-up outage of a payments system, I really feel for their incident response team. I

Having been in this situation before, with a totally-down-and-not-coming-back-up outage of a payments system, I really feel for their incident response team. I'll take this moment to remind everyone of their human tendency to read meaning into random events. There's no evidence to suggest New Year traffic has caused this, and outages like this can happen in spite of professional and competent preparation. Hugops for their team, I hope they get it back soon.

yjftsjthsd-h 6y ago
> I'll take this moment to remind everyone of their human tendency to read meaning into random events. There's no evidence to suggest New Year traffic has caused this, and outages like this can happen in spite of professional and competent preparation. On the one hand, sure we don't specifically know what's going on. On the other hand, it's the first Monday in the new year and they went down shortly after the start of the business day Eastern time; it could be coincidence, but it would be a remarkable coincidence.
- lawrjone 6y ago
  There are a load of ways NY might have contributed to this, but it may not be a direct cause. What's more likely, Slack forgetting to scale their deployment back up after too much mulled wine, or a number of people on holiday meaning a simple failure has developed into something more serious? It could be anything really- my post was more about how situations like this can happen to even the most prepared. The assumption it has something to do with NY tends to assume very trivial, silly mistakes. Especially with no information, that seems a bit uncharitable.
- taneliv 6y ago
  It seemingly worked ok in UTC-2 in the morning and early afternoon, then started having issues and is now a bit intermittent (or fixed, there's not much traffic on my channels, as it's evening already). Do they have that much more traffic on US east coast than in Europe?
  - martinald 6y ago
    Probably, but it was only 2-3pm UK time when it started falling over so there would be all the Europe traffic plus the East Coast traffic starting to sign in.
- xeromal 6y ago
  Another likely scenario is that they a deployment that was risky that they waited to push until after the holidays.

yjftsjthsd-h 6y ago

> I'll take this moment to remind everyone of their human tendency to read meaning into random events. There's no evidence to suggest New Year traffic has caused this, and outages like this can happen in spite of professional and competent preparation. On the one hand, sure we don't specifically know what's going on. On the other hand, it's the first Monday in the new year and they went down shortly after the start of the business day Eastern time; it could be coincidence, but it would be a remarkable coincidence.

lawrjone 6y ago
There are a load of ways NY might have contributed to this, but it may not be a direct cause. What's more likely, Slack forgetting to scale their deployment back up after too much mulled wine, or a number of people on holiday meaning a simple failure has developed into something more serious? It could be anything really- my post was more about how situations like this can happen to even the most prepared. The assumption it has something to do with NY tends to assume very trivial, silly mistakes. Especially with no information, that seems a bit uncharitable.
taneliv 6y ago
It seemingly worked ok in UTC-2 in the morning and early afternoon, then started having issues and is now a bit intermittent (or fixed, there's not much traffic on my channels, as it's evening already). Do they have that much more traffic on US east coast than in Europe?
- martinald 6y ago
  Probably, but it was only 2-3pm UK time when it started falling over so there would be all the Europe traffic plus the East Coast traffic starting to sign in.
xeromal 6y ago
Another likely scenario is that they a deployment that was risky that they waited to push until after the holidays.