4 ms·

This is actually funny. Just this weekend I was talking to another engineer friend of mine, and we were taking bets as to how long it would be before twitter w

This is actually funny. Just this weekend I was talking to another engineer friend of mine, and we were taking bets as to how long it would be before twitter went down for the last time. We felt bad for the engineers that are still there, and were actually surprised it's stayed up this long, with all the knowledge that walked out the door. Basic stuff like "where are our servers actually located". I guess the folks that left loved Twitter enough to leave behind good notes, because even the most well architected systems still need some institutional knowledge when they go down.

renewiltord 4y ago
Where did you land on the bets? It appears that when Twitter goes down, its engineers were indispensable but when it's up, it's because the engineers foresaw this. Given those factors, how long it stays up must be an interesting combination of how indispensable the engineers made themselves vs how much foresight they had.
- jedberg 4y ago
  I know some of the former Twitter engineers, and they were great engineers. I'm sure they built a fairly resilient system. But at the scale and complexity, eventually you hit an unforeseen error that requires some institutional knowledge to fix, and it's doubtful they still have that, unless the ex-engineers loved the product so much they are willing to take phone calls from the people that are left and point them in the right direction. I don't think we landed on a time frame. I'm actually surprised it stayed up this long.
  - TigeriusKirk 4y ago
    Do you actually think it will go down for good? That seems extremely unlikely to me.
    - jedberg 4y ago
      No, eventually they would figure out how to rewrite it if they can't get it working again. But I could definitely see a multi-week outage.
      - MrOwnPut 4y ago
        Seems to be back up now. Both logged in and logged out states are working. This thread is just an hour old btw, not sure when the issue started. Seems like a huge reaction from HN for an hour long downtime given that Twitter has gone down for longer in the past.
        not_a_shill 4y ago
        They do this every time. Last thread I saw was Twitter had an outage for only 10k people for a short amount of time and HN was having a victory lap. I personally don't think that 100% uptime for everyone around the globe is even a worthwhile goal for Twitter. The incremental costs for that last 0.5% aren't worth the incremental benefits. HA is most necessary for what, journalists? (Who are already antagonized under Musk) Everyone else can wait the hour.
        Daishiman 4y ago
        It's down for me and hasn't worked for the entire weekend.
  - not_a_shill 4y ago
    >I don't think we landed on a time frame. I'm actually surprised it stayed up this long. How much to get in on this? Seems like free money if you thought the time frame was that early.
    - renewiltord 4y ago
      The problem with bets with strangers is counterparty risk. The counterparty cannot prove that they will fulfill in the loss scenario, which makes it a guaranteed loss. Bets with friends don't have that problem, since the bet is usually either cash-free or for such a small value that retaining the friendship is more important.
    - deleted 4y ago
      [deleted]
- vanjajaja1 4y ago
  Exactly right, which is why this whole saga is so irritating to watch. The commentary is heavily biased and its exhausting to filter back to neutral. I just want to know the impact of many engineers leaving
  - mjr00 4y ago
    You can never know the true impact because people will spin the narrative to say what they want. This hour-long outage occurred because Twitter laid off their engineers. Meanwhile, AWS had a major outage that lasted over half a day and took down Slack among other major internet infrastructure in December 2021[0], but we can't blame that on layoffs so we just ignore it, I guess. There's really no correlation between layoffs and outages, I suspect. [0] https://aws.amazon.com/message/12721/ https://aws.amazon.com/message/12721/
    - grumple 4y ago
      There's definitely a correlation. But not having layoffs just doesn't mean no outages.
      - mjr00 4y ago
        > There's definitely a correlation. Where is the evidence of this?
        jeromegv 4y ago
        What evidence do you expect? It's not like Twitter is going to show us a post morterm from their outage. Anyone who knows and uses twitter quite a bit can see that things have been wonky on the sites. Big users have mentioned tons of issues (their mentions being empty, twitter threads going broken, some tweets never reaching their audience not even a bit), etc, etc. It's all anecdotal of course, we can't build a case of evidence, this is a private company and all the shit internally is private. But things are getting worse, not better. Even those big initiatives announced are not actually happening, that paid and free API plan that was supposed to be released in 7 days? It's been a month and nothing. Things are not "normal" and clearly the lack of engineering talent is slowing a lot of things down (both new development and infrastructure resilience).
- grumple 4y ago
  We always build systems hoping they never need any maintenance. The truth is, they often do. Especially when there are changes happening. I once took a week off and we went down on day 2. Other times, I'm convinced I could've gone 6 months without anything breaking. At Twitter's scale, I'm sure there are systems where everything is automated and self-healing. But there are probably other systems that are more fragile. And then there's the eternal race against scale; eventually a system with no maintenance will get buried under the weight of its own data or the scope of its usage. That always requires some work.
doomleika 4y ago
I would say it’s just H1B, hard to jump ship when you are on such a tight window
- vkou 4y ago
  Harder in the current economy, when everyone else is doing layoffs.
- SauciestGNU 4y ago
  Just had the wild idea that Elon fires anyone who's away from their workstation long enough to be potentially interviewing elsewhere. Just crazy enough an idea to be something he'd pull.
  - dmonitor 4y ago
    you have chosen an interesting choice of topic for fanfiction

renewiltord 4y ago

Where did you land on the bets? It appears that when Twitter goes down, its engineers were indispensable but when it's up, it's because the engineers foresaw this. Given those factors, how long it stays up must be an interesting combination of how indispensable the engineers made themselves vs how much foresight they had.

jedberg 4y ago
I know some of the former Twitter engineers, and they were great engineers. I'm sure they built a fairly resilient system. But at the scale and complexity, eventually you hit an unforeseen error that requires some institutional knowledge to fix, and it's doubtful they still have that, unless the ex-engineers loved the product so much they are willing to take phone calls from the people that are left and point them in the right direction. I don't think we landed on a time frame. I'm actually surprised it stayed up this long.
- TigeriusKirk 4y ago
  Do you actually think it will go down for good? That seems extremely unlikely to me.
  - jedberg 4y ago
    No, eventually they would figure out how to rewrite it if they can't get it working again. But I could definitely see a multi-week outage.
    - MrOwnPut 4y ago
      Seems to be back up now. Both logged in and logged out states are working. This thread is just an hour old btw, not sure when the issue started. Seems like a huge reaction from HN for an hour long downtime given that Twitter has gone down for longer in the past.
      - not_a_shill 4y ago
        They do this every time. Last thread I saw was Twitter had an outage for only 10k people for a short amount of time and HN was having a victory lap. I personally don't think that 100% uptime for everyone around the globe is even a worthwhile goal for Twitter. The incremental costs for that last 0.5% aren't worth the incremental benefits. HA is most necessary for what, journalists? (Who are already antagonized under Musk) Everyone else can wait the hour.
      - Daishiman 4y ago
        It's down for me and hasn't worked for the entire weekend.
- not_a_shill 4y ago
  >I don't think we landed on a time frame. I'm actually surprised it stayed up this long. How much to get in on this? Seems like free money if you thought the time frame was that early.
  - renewiltord 4y ago
    The problem with bets with strangers is counterparty risk. The counterparty cannot prove that they will fulfill in the loss scenario, which makes it a guaranteed loss. Bets with friends don't have that problem, since the bet is usually either cash-free or for such a small value that retaining the friendship is more important.
  - deleted 4y ago
    [deleted]
vanjajaja1 4y ago
Exactly right, which is why this whole saga is so irritating to watch. The commentary is heavily biased and its exhausting to filter back to neutral. I just want to know the impact of many engineers leaving
- mjr00 4y ago
  You can never know the true impact because people will spin the narrative to say what they want. This hour-long outage occurred because Twitter laid off their engineers. Meanwhile, AWS had a major outage that lasted over half a day and took down Slack among other major internet infrastructure in December 2021[0], but we can't blame that on layoffs so we just ignore it, I guess. There's really no correlation between layoffs and outages, I suspect. [0] https://aws.amazon.com/message/12721/ https://aws.amazon.com/message/12721/
  - grumple 4y ago
    There's definitely a correlation. But not having layoffs just doesn't mean no outages.
    - mjr00 4y ago
      > There's definitely a correlation. Where is the evidence of this?
      - jeromegv 4y ago
        What evidence do you expect? It's not like Twitter is going to show us a post morterm from their outage. Anyone who knows and uses twitter quite a bit can see that things have been wonky on the sites. Big users have mentioned tons of issues (their mentions being empty, twitter threads going broken, some tweets never reaching their audience not even a bit), etc, etc. It's all anecdotal of course, we can't build a case of evidence, this is a private company and all the shit internally is private. But things are getting worse, not better. Even those big initiatives announced are not actually happening, that paid and free API plan that was supposed to be released in 7 days? It's been a month and nothing. Things are not "normal" and clearly the lack of engineering talent is slowing a lot of things down (both new development and infrastructure resilience).
grumple 4y ago
We always build systems hoping they never need any maintenance. The truth is, they often do. Especially when there are changes happening. I once took a week off and we went down on day 2. Other times, I'm convinced I could've gone 6 months without anything breaking. At Twitter's scale, I'm sure there are systems where everything is automated and self-healing. But there are probably other systems that are more fragile. And then there's the eternal race against scale; eventually a system with no maintenance will get buried under the weight of its own data or the scope of its usage. That always requires some work.

doomleika 4y ago

I would say it’s just H1B, hard to jump ship when you are on such a tight window

vkou 4y ago
Harder in the current economy, when everyone else is doing layoffs.
SauciestGNU 4y ago
Just had the wild idea that Elon fires anyone who's away from their workstation long enough to be potentially interviewing elsewhere. Just crazy enough an idea to be something he'd pull.
- dmonitor 4y ago
  you have chosen an interesting choice of topic for fanfiction