4 ms·

I don't think anything like this has ever happened before to a system with this scale, so who can say? But just as an engineer at another company that probably

I don't think anything like this has ever happened before to a system with this scale, so who can say? But just as an engineer at another company that probably has a vaguely similar architecture it's certainly in a place of extreme risk. It just depends on if they can get people in place to hold together the critical services. Also things are much more likely to break it they start trying to change things, it will take longer for a problem to happen if there is little new feature work.

cratermoon 4y ago
I have a hunch that over Thanksgiving the increased traffic will break something, and there will be nobody around to fix it, so more stuff will break. I have a hunch that most of SRE people that would normally be watching the metrics ready to jump on anything starting to look dicey are gone.
- erk__ 4y ago
  Not just thanksgiving, by the world cup is starting next week as well and that usually drives a lot of traffic
  - Ekaros 4y ago
    And World Cup traffic will be bursty. Big game, big event in game. Could really peak message volume and this is even geographically distributed. So fun times for those left there to fight the fires.
    - _moof 4y ago
      During the 2010 World Cup, Twitter crashed every time a goal was scored. It was a major motivation for the rearchitecture that changed it from a monorail to services. The rock-solid stability during the 2014 World Cup vindicated that effort. With that context, the performance during this World Cup will be especially meaningful.
ruh-roh 4y ago
The "once they start trying to change things" aspect is where I'm most keen to watch. Surely we have all been involved with large-to-small systems where we were curious how well/long they would stand up on their own without intervention. It's kind of fascinating to watch a real-world example, even though we likely won't get a lot of detail. But still, :popcorn_emoji:. But what's getting somewhat-ignored in this discourse is they aren't changing anything. All deploys are frozen. And the new CEO has made a ton of promises about new features and policies. Eventually they need to roll those out, and once they do, damn... Even if they have strong automated tests, those generally only tell you that new code is functionally ok. Anything that crosses interfaces, deals with end-to-end integration, or adds new tables/dbs/jobs/etc that would eventually hit their scaling solutions - all could have catastrophic effects. (And that ignores the consent decree stuff on changes too.)

cratermoon 4y ago

I have a hunch that over Thanksgiving the increased traffic will break something, and there will be nobody around to fix it, so more stuff will break. I have a hunch that most of SRE people that would normally be watching the metrics ready to jump on anything starting to look dicey are gone.

erk__ 4y ago
Not just thanksgiving, by the world cup is starting next week as well and that usually drives a lot of traffic
- Ekaros 4y ago
  And World Cup traffic will be bursty. Big game, big event in game. Could really peak message volume and this is even geographically distributed. So fun times for those left there to fight the fires.
  - _moof 4y ago
    During the 2010 World Cup, Twitter crashed every time a goal was scored. It was a major motivation for the rearchitecture that changed it from a monorail to services. The rock-solid stability during the 2014 World Cup vindicated that effort. With that context, the performance during this World Cup will be especially meaningful.

ruh-roh 4y ago

The "once they start trying to change things" aspect is where I'm most keen to watch. Surely we have all been involved with large-to-small systems where we were curious how well/long they would stand up on their own without intervention. It's kind of fascinating to watch a real-world example, even though we likely won't get a lot of detail. But still, :popcorn_emoji:. But what's getting somewhat-ignored in this discourse is they aren't changing anything. All deploys are frozen. And the new CEO has made a ton of promises about new features and policies. Eventually they need to roll those out, and once they do, damn... Even if they have strong automated tests, those generally only tell you that new code is functionally ok. Anything that crosses interfaces, deals with end-to-end integration, or adds new tables/dbs/jobs/etc that would eventually hit their scaling solutions - all could have catastrophic effects. (And that ignores the consent decree stuff on changes too.)