5 ms·

then wouldn't this happen every monday morning?

• unethical_ban 6y ago
  Perhaps it is a large number of people checking into channels that are backlogged with lots of bot message notifications.
• floatingatoll 6y ago
  A lot of organizations essentially took the last two weeks off from work, which is long enough for a 10-day autoscale window to spin down servers, and then get confronted by a load spike that wasn't pre-spun for.
  • derwiki 6y ago
    I would be shocked if Slack operations wasn’t aware of this return to work spike and didn’t pre-scale in anticipation.
    • floatingatoll 6y ago
      I wouldn't, since my personal theory is that the outage is due to AWS and GCP autoscale capacity exhaustion. We'll find out soon enough! EDIT: And down goes Notion, too: https://news.ycombinator.com/item?id=25634159 https://news.ycombinator.com/item?id=25634159
      • yreg 6y ago
        >AWS and GCP autoscale capacity What does this mean? What do cloud providers do when customers scale down their services? Do the providers literally power down servers? Do they sell the capacity to new customers?
        • delfinom 6y ago
          They rate limit how fast you can auto scale which is dependent on a slew of factors.
        • rocho 6y ago
          They sell unused capacity at a much lower price (spot instances on AWS, preemptible VMs on GCP). I don't know if they power down some servers if usage stays low for a very long time.
    • ABeeSea 6y ago
      That doesn’t mean they chose the right number to scale to. See for example, Amazon Prime day: https://www.cnbc.com/2018/07/19/amazon-internal-documents-what-caused-prime-day-crash-company-scramble.html https://www.cnbc.com/2018/07/19/amazon-internal-documents-wh...
• spicybright 6y ago
  This is after many had a week vacation. I'm sure most weekends some people pop in and out, and logins are more staggered on a typical monday morning. Just a theory though.