3 ms·

Exactly right, which is why this whole saga is so irritating to watch. The commentary is heavily biased and its exhausting to filter back to neutral. I just wan

Exactly right, which is why this whole saga is so irritating to watch. The commentary is heavily biased and its exhausting to filter back to neutral. I just want to know the impact of many engineers leaving

mjr00 4y ago
You can never know the true impact because people will spin the narrative to say what they want. This hour-long outage occurred because Twitter laid off their engineers. Meanwhile, AWS had a major outage that lasted over half a day and took down Slack among other major internet infrastructure in December 2021[0], but we can't blame that on layoffs so we just ignore it, I guess. There's really no correlation between layoffs and outages, I suspect. [0] https://aws.amazon.com/message/12721/ https://aws.amazon.com/message/12721/
- grumple 4y ago
  There's definitely a correlation. But not having layoffs just doesn't mean no outages.
  - mjr00 4y ago
    > There's definitely a correlation. Where is the evidence of this?
    - jeromegv 4y ago
      What evidence do you expect? It's not like Twitter is going to show us a post morterm from their outage. Anyone who knows and uses twitter quite a bit can see that things have been wonky on the sites. Big users have mentioned tons of issues (their mentions being empty, twitter threads going broken, some tweets never reaching their audience not even a bit), etc, etc. It's all anecdotal of course, we can't build a case of evidence, this is a private company and all the shit internally is private. But things are getting worse, not better. Even those big initiatives announced are not actually happening, that paid and free API plan that was supposed to be released in 7 days? It's been a month and nothing. Things are not "normal" and clearly the lack of engineering talent is slowing a lot of things down (both new development and infrastructure resilience).
      - mjr00 4y ago
        Assuming that you are correct that things are breaking more at Twitter (which I'm not sure is accurate to begin with), how do you know that is due to "lack of engineering talent" and not due to higher pressure to deliver features? As anyone who has operated a software system knows, the #1 reason things break in production is new code, and from a third-party observer with no inside knowledge it certainly seems like more changes are happening at Twitter than ever. This is orthogonal to layoffs, though.
    - grumple 4y ago
      https://trends.google.com/trends/explore?date=today%205-y&geo=US&q=Twitter%20down https://trends.google.com/trends/explore?date=today%205-y&ge... Best source I could find. Currentlydown didn't even register yesterday's outage, and downdetector doesn't share historical data.
vkou 4y ago
You're not going to know that impact, because most of the technical problems that Twitter as a company can have aren't even visible to us. Sure, images and hyperlinks breaking and not paying the Slack bill and Musk having another mental break makes the front page of hacker news, but will a spam and abuse detection pipeline being silently busted for 3 months do that? A reporting backend that was providing advertisers with incorrect data on their ad spend? Some customer-relations workflow that ruined the week of a group of TAMs, and had revenue impact? Twitter is a big ship. There are more places things can break than anyone, inside or outside it can imagine. But as an outside observer, the only signal that we can act on is 'Has the ship sunk? Or is it floating?' Observing that binary state answers a question, but not the question of 'Is the ship working well?'
- mattkrause 4y ago
  And that's what might eventually kill it. It seems really unlikely to die from a single spectacular failure, where a key bit of infrastructure dies and just cannot be brought back online for any amount of love or money. A steady drip of annoying but individually minor issues will gradually chase people away: more spam and less relevant recommendations discourage users; bad spend data and fewer account reps make it a lower priority for advertisers. Twitter's main value prop is its network effect, and if a few people leave (or even engage less) every issue, that value will wither away....

mjr00 4y ago

You can never know the true impact because people will spin the narrative to say what they want. This hour-long outage occurred because Twitter laid off their engineers. Meanwhile, AWS had a major outage that lasted over half a day and took down Slack among other major internet infrastructure in December 2021[0], but we can't blame that on layoffs so we just ignore it, I guess. There's really no correlation between layoffs and outages, I suspect. [0] https://aws.amazon.com/message/12721/ https://aws.amazon.com/message/12721/

grumple 4y ago
There's definitely a correlation. But not having layoffs just doesn't mean no outages.
- mjr00 4y ago
  > There's definitely a correlation. Where is the evidence of this?
  - jeromegv 4y ago
    What evidence do you expect? It's not like Twitter is going to show us a post morterm from their outage. Anyone who knows and uses twitter quite a bit can see that things have been wonky on the sites. Big users have mentioned tons of issues (their mentions being empty, twitter threads going broken, some tweets never reaching their audience not even a bit), etc, etc. It's all anecdotal of course, we can't build a case of evidence, this is a private company and all the shit internally is private. But things are getting worse, not better. Even those big initiatives announced are not actually happening, that paid and free API plan that was supposed to be released in 7 days? It's been a month and nothing. Things are not "normal" and clearly the lack of engineering talent is slowing a lot of things down (both new development and infrastructure resilience).
    - mjr00 4y ago
      Assuming that you are correct that things are breaking more at Twitter (which I'm not sure is accurate to begin with), how do you know that is due to "lack of engineering talent" and not due to higher pressure to deliver features? As anyone who has operated a software system knows, the #1 reason things break in production is new code, and from a third-party observer with no inside knowledge it certainly seems like more changes are happening at Twitter than ever. This is orthogonal to layoffs, though.
  - grumple 4y ago
    https://trends.google.com/trends/explore?date=today%205-y&geo=US&q=Twitter%20down https://trends.google.com/trends/explore?date=today%205-y&ge... Best source I could find. Currentlydown didn't even register yesterday's outage, and downdetector doesn't share historical data.

vkou 4y ago

You're not going to know that impact, because most of the technical problems that Twitter as a company can have aren't even visible to us. Sure, images and hyperlinks breaking and not paying the Slack bill and Musk having another mental break makes the front page of hacker news, but will a spam and abuse detection pipeline being silently busted for 3 months do that? A reporting backend that was providing advertisers with incorrect data on their ad spend? Some customer-relations workflow that ruined the week of a group of TAMs, and had revenue impact? Twitter is a big ship. There are more places things can break than anyone, inside or outside it can imagine. But as an outside observer, the only signal that we can act on is 'Has the ship sunk? Or is it floating?' Observing that binary state answers a question, but not the question of 'Is the ship working well?'

mattkrause 4y ago
And that's what might eventually kill it. It seems really unlikely to die from a single spectacular failure, where a key bit of infrastructure dies and just cannot be brought back online for any amount of love or money. A steady drip of annoying but individually minor issues will gradually chase people away: more spam and less relevant recommendations discourage users; bad spend data and fewer account reps make it a lower priority for advertisers. Twitter's main value prop is its network effect, and if a few people leave (or even engage less) every issue, that value will wither away....