5 ms·

Non-political crazy things uncovered in Twitter whistleblower report

46 points by savanaly 4y ago

johng 4y ago
How parag isn’t being prosecuted boggles my mind.
- anigbrowl 4y ago
  It's amazing to me that you haven't been prosecuted. Edit: Some people really failing to see the point here, which is that it's meaningless to suggest someone should be prosecuted without specifying why.
smoldesu 4y ago
> Twitter does not have separate development, test, staging, and production environments. It's legitimately difficult to imagine any software architecture running smoothly like this, much less a highly successful product with hundreds of engineers working on it... It's necessarily criminal, but it seems everyone's suspicions about Twitter's codebase is being proven right in media res.
- viraptor 4y ago
  I'm suspicious of this one. It may be misinterpreting what's happening, but we need more details. Specifically, there's obviously some dev environment - people weren't literally editing files directly in production. Then, at some scale you may actually reject the idea of staging and instead concentrate on canary deployments, monitoring, and fast rollbacks. I'm not saying the situation wasn't bad, but the quote doesn't tell us much without explaining how did it work instead. There's a lot of huge projects with only production environment. Probably more than people are comfortable thinking of.
- adamsb6 4y ago
  With a service of this size creating a copy of prod is a non-trivial problem, and there are classes of bugs that will only manifest against prod data. A pristine, miniature pretend-prod just isn't that useful. What I've seen work in the past is that you develop your changes locally, run a local instance of whatever service you're changing, but local instance is backed by stateful layers in prod. Feature flags are ingrained in the culture, and you'll be expected to do things like gate your code to some small set of users at first. Employees are a good option, or test users created in prod if you're especially wary of your changes. I only know of this leading to one SEV, where the feature flags broke in such a way that everything was suddenly enabled for everyone.
- eep_social 4y ago
  I hear Google is a fairly successful company so I think this is a limitation of your imagination. In your defense, I have argued this with very intelligent people who share the same limitation. At some point, creating a second copy of prod is infeasible or becomes so expensive that one must revisit and discard assumptions about what is actually possible. Duplicating a small to medium set of micro services with a bunch of SaaS integrations isn’t typically a big deal. Much can be done by leveraging a cloud provider’s flexible capacity. At some point that stops working, e.g. when you are the cloud provider. AWS isn’t outsourcing reliability to GCP or vice versa. Other comments have already mentioned outages driven by differences between prod and staging/testing/etc — those issues only get worse at scale.
- yucky 4y ago
  There is zero chance this is true though. None. You will of course end up with some bugs that can't (for mysterious reasons) be exactly replicated in each environment. That is not the same thing.
AustinDev 4y ago
Edit: Wrong thread. My bad
- theCrowing 4y ago
  Wrong Thread?
  - AustinDev 4y ago
    Yep wrong thread thanks! Mind throwing me a downvote? I can't delete due to reply.
    - anigbrowl 4y ago
      Flagging is the best option in this situation. I flagged your original post (and upvoted this one). That eventually makes it disappear along with the replies if there is not much traffic.
      - AustinDev 4y ago
        Did not know that. I don't have the karma to flag comments it seems.
ZeroGravitas 4y ago
> Disengaged CEO: CEO Jack Dorsey had recruited Mudge personally. They got along well, and Mudge has never suspected Dorsey of harboring bad intent. But Dorsey, the high-profile CEO of one of the most prominent companies on earth, was experiencing a drastic loss of focus in 2021. Dorsey attended meetings sporadically, and when he did, he was extremely disengaged.” In some meetings —even after he was briefed on complex corporate issues — Dorsey did not speak a word. Mudge heard from his colleagues that Dorsey would remain silent for days or weeks. Worried about Dorsey's health, the senior team mostly tried to cover up for him,” but even mid- and lower-level staff could tell that the ship was rudderless
- anigbrowl 4y ago
  Worried about Dorsey's health, the senior team mostly tried to cover up for him This seems like a failure of the corporate model in general. 'Everyone knows' the CEO/dictator/monarch is unable to function, but senior leaders cover it up (poorly), the board looks the other way, shareholders are deceived or at least kept ignorance, and the rabble (that's you) are confused at best and dismissed as crazy if they intuit the state of affairs correctly. Why do we institutionalize such systems when the patterns of failure are so glaringly obvious and have been endlessly repeated?

46 points by savanaly 4y ago

johng 4y ago

How parag isn’t being prosecuted boggles my mind.

anigbrowl 4y ago
It's amazing to me that you haven't been prosecuted. Edit: Some people really failing to see the point here, which is that it's meaningless to suggest someone should be prosecuted without specifying why.

smoldesu 4y ago

> Twitter does not have separate development, test, staging, and production environments. It's legitimately difficult to imagine any software architecture running smoothly like this, much less a highly successful product with hundreds of engineers working on it... It's necessarily criminal, but it seems everyone's suspicions about Twitter's codebase is being proven right in media res.

viraptor 4y ago
I'm suspicious of this one. It may be misinterpreting what's happening, but we need more details. Specifically, there's obviously some dev environment - people weren't literally editing files directly in production. Then, at some scale you may actually reject the idea of staging and instead concentrate on canary deployments, monitoring, and fast rollbacks. I'm not saying the situation wasn't bad, but the quote doesn't tell us much without explaining how did it work instead. There's a lot of huge projects with only production environment. Probably more than people are comfortable thinking of.
adamsb6 4y ago
With a service of this size creating a copy of prod is a non-trivial problem, and there are classes of bugs that will only manifest against prod data. A pristine, miniature pretend-prod just isn't that useful. What I've seen work in the past is that you develop your changes locally, run a local instance of whatever service you're changing, but local instance is backed by stateful layers in prod. Feature flags are ingrained in the culture, and you'll be expected to do things like gate your code to some small set of users at first. Employees are a good option, or test users created in prod if you're especially wary of your changes. I only know of this leading to one SEV, where the feature flags broke in such a way that everything was suddenly enabled for everyone.
eep_social 4y ago
I hear Google is a fairly successful company so I think this is a limitation of your imagination. In your defense, I have argued this with very intelligent people who share the same limitation. At some point, creating a second copy of prod is infeasible or becomes so expensive that one must revisit and discard assumptions about what is actually possible. Duplicating a small to medium set of micro services with a bunch of SaaS integrations isn’t typically a big deal. Much can be done by leveraging a cloud provider’s flexible capacity. At some point that stops working, e.g. when you are the cloud provider. AWS isn’t outsourcing reliability to GCP or vice versa. Other comments have already mentioned outages driven by differences between prod and staging/testing/etc — those issues only get worse at scale.
yucky 4y ago
There is zero chance this is true though. None. You will of course end up with some bugs that can't (for mysterious reasons) be exactly replicated in each environment. That is not the same thing.

AustinDev 4y ago

Edit: Wrong thread. My bad

theCrowing 4y ago
Wrong Thread?
- AustinDev 4y ago
  Yep wrong thread thanks! Mind throwing me a downvote? I can't delete due to reply.
  - anigbrowl 4y ago
    Flagging is the best option in this situation. I flagged your original post (and upvoted this one). That eventually makes it disappear along with the replies if there is not much traffic.
    - AustinDev 4y ago
      Did not know that. I don't have the karma to flag comments it seems.

ZeroGravitas 4y ago

> Disengaged CEO: CEO Jack Dorsey had recruited Mudge personally. They got along well, and Mudge has never suspected Dorsey of harboring bad intent. But Dorsey, the high-profile CEO of one of the most prominent companies on earth, was experiencing a drastic loss of focus in 2021. Dorsey attended meetings sporadically, and when he did, he was extremely disengaged.” In some meetings —even after he was briefed on complex corporate issues — Dorsey did not speak a word. Mudge heard from his colleagues that Dorsey would remain silent for days or weeks. Worried about Dorsey's health, the senior team mostly tried to cover up for him,” but even mid- and lower-level staff could tell that the ship was rudderless

anigbrowl 4y ago
Worried about Dorsey's health, the senior team mostly tried to cover up for him This seems like a failure of the corporate model in general. 'Everyone knows' the CEO/dictator/monarch is unable to function, but senior leaders cover it up (poorly), the board looks the other way, shareholders are deceived or at least kept ignorance, and the rabble (that's you) are confused at best and dismissed as crazy if they intuit the state of affairs correctly. Why do we institutionalize such systems when the patterns of failure are so glaringly obvious and have been endlessly repeated?