2 ms·

> Twitter does not have separate development, test, staging, and production environments. It's legitimately difficult to imagine any software architecture runn

> Twitter does not have separate development, test, staging, and production environments. It's legitimately difficult to imagine any software architecture running smoothly like this, much less a highly successful product with hundreds of engineers working on it... It's necessarily criminal, but it seems everyone's suspicions about Twitter's codebase is being proven right in media res.

viraptor 4y ago
I'm suspicious of this one. It may be misinterpreting what's happening, but we need more details. Specifically, there's obviously some dev environment - people weren't literally editing files directly in production. Then, at some scale you may actually reject the idea of staging and instead concentrate on canary deployments, monitoring, and fast rollbacks. I'm not saying the situation wasn't bad, but the quote doesn't tell us much without explaining how did it work instead. There's a lot of huge projects with only production environment. Probably more than people are comfortable thinking of.
adamsb6 4y ago
With a service of this size creating a copy of prod is a non-trivial problem, and there are classes of bugs that will only manifest against prod data. A pristine, miniature pretend-prod just isn't that useful. What I've seen work in the past is that you develop your changes locally, run a local instance of whatever service you're changing, but local instance is backed by stateful layers in prod. Feature flags are ingrained in the culture, and you'll be expected to do things like gate your code to some small set of users at first. Employees are a good option, or test users created in prod if you're especially wary of your changes. I only know of this leading to one SEV, where the feature flags broke in such a way that everything was suddenly enabled for everyone.
eep_social 4y ago
I hear Google is a fairly successful company so I think this is a limitation of your imagination. In your defense, I have argued this with very intelligent people who share the same limitation. At some point, creating a second copy of prod is infeasible or becomes so expensive that one must revisit and discard assumptions about what is actually possible. Duplicating a small to medium set of micro services with a bunch of SaaS integrations isn’t typically a big deal. Much can be done by leveraging a cloud provider’s flexible capacity. At some point that stops working, e.g. when you are the cloud provider. AWS isn’t outsourcing reliability to GCP or vice versa. Other comments have already mentioned outages driven by differences between prod and staging/testing/etc — those issues only get worse at scale.
yucky 4y ago
There is zero chance this is true though. None. You will of course end up with some bugs that can't (for mysterious reasons) be exactly replicated in each environment. That is not the same thing.

viraptor 4y ago

I'm suspicious of this one. It may be misinterpreting what's happening, but we need more details. Specifically, there's obviously some dev environment - people weren't literally editing files directly in production. Then, at some scale you may actually reject the idea of staging and instead concentrate on canary deployments, monitoring, and fast rollbacks. I'm not saying the situation wasn't bad, but the quote doesn't tell us much without explaining how did it work instead. There's a lot of huge projects with only production environment. Probably more than people are comfortable thinking of.

adamsb6 4y ago

With a service of this size creating a copy of prod is a non-trivial problem, and there are classes of bugs that will only manifest against prod data. A pristine, miniature pretend-prod just isn't that useful. What I've seen work in the past is that you develop your changes locally, run a local instance of whatever service you're changing, but local instance is backed by stateful layers in prod. Feature flags are ingrained in the culture, and you'll be expected to do things like gate your code to some small set of users at first. Employees are a good option, or test users created in prod if you're especially wary of your changes. I only know of this leading to one SEV, where the feature flags broke in such a way that everything was suddenly enabled for everyone.

eep_social 4y ago

I hear Google is a fairly successful company so I think this is a limitation of your imagination. In your defense, I have argued this with very intelligent people who share the same limitation. At some point, creating a second copy of prod is infeasible or becomes so expensive that one must revisit and discard assumptions about what is actually possible. Duplicating a small to medium set of micro services with a bunch of SaaS integrations isn’t typically a big deal. Much can be done by leveraging a cloud provider’s flexible capacity. At some point that stops working, e.g. when you are the cloud provider. AWS isn’t outsourcing reliability to GCP or vice versa. Other comments have already mentioned outages driven by differences between prod and staging/testing/etc — those issues only get worse at scale.

yucky 4y ago

There is zero chance this is true though. None. You will of course end up with some bugs that can't (for mysterious reasons) be exactly replicated in each environment. That is not the same thing.