4 ms·

What does that mean though? Twitter must have test environments for developers and automatic tests. Do they just not have a full staging environment? That wou

by smt88 4y ago

What does that mean though? Twitter must have test environments for developers and automatic tests. Do they just not have a full staging environment? That would be understandable. It's impossible to create a production-like test environment for a product like Twitter. You're better off just doing canary releases or feature flags with slow rollouts.

michaelt 4y ago
> It's impossible to create a production-like test environment for a product like Twitter. What makes you think that? I mean, I know it costs money and needs people who understand how the product works and how customers behave - but why on earth would it be impossible?
- smt88 4y ago
  It's impossible because you need to simulate the volume and variety of Twitter usage. Especially when you consider that production Twitter is thousands of servers, you're just not going to be able to create a mirror of production. For something with many millions of users, it makes more sense to do canary releases to internal users, then to a few thousand, and then scale up from there.
  - michaelt 4y ago
    If you can accept, index and store a given volume of tweets, why should it be impossibly difficult to generate that same volume? I would imagine it would be a lot easier, in fact. And if can't afford a 1:1 scale replica, build a 1/10th scale replica and have your senior engineers use their expertise to ensure it's representative enough for test purposes.
    - sjsdaiuasgdia 4y ago
      It's a much harder target to generate truly representative load. Sure, you could make millions of tweets very fast, but is that actually representing what the production service is dealing with? Unstable connections. Traffic spikes that can be highly localized or nearly system wide. System maintenance, planned and reactive. Poorly behaved clients. Malicious traffic. Many more similar factors...and yes, each of these you could throw engineers at them and get solutions eventually. You might also end up with a testing engineering team that dwarfs the product engineering team. Also, Twitter doesn't make money. $5B revenue vs $5.5B opex in 2021. They can't really afford to run the actual system, much less a fully representative one. Even a 1/10th replica, which would be unable to find some classes of problems no matter how many engineers you throw at, would further break the bank. As would the engineering staff to support it.
    - smt88 4y ago
      Twitter can't afford a 1:1 replica for sure, and it would be meaningless without globally distributed production load anyway. A 1:10 replica is fine, but also astronomically expensive. What's better about it than 1:100? Or 1:10000? Once you give up on 1:1 (which is reasonable), you're either testing in production or not really testing at all. You'll have many lines of code that will only apply to the 1:1 scale.
  - TechBro8615 4y ago
    You don’t need one staging environment simulating the entire platform. You can have small staging environments that run some parts of the product, for example the latest version of the advertiser dashboard or the messaging inbox. Sure, you won’t be simulating a whole class of problems that can arise at scale, but you can still test the more localized features in a safe environment.
    - smt88 4y ago
      Yes, I understand that. That's why it's unclear what it means that Twitter has "no test environment". That could mean a lot of things, some catastrophic and some not.
      - m00x 4y ago
        It's pretty simple. They have no testing environment. They yeet shit in prod under feature flags (I assume), and hope for the best.
- bragr 4y ago
  >What makes you think that? For large scale sites you need a parallel copy of the internet which doesn't exist. For example, we once caused a production incident (for us and other people) by simulating a major business event to our test stack and saturated links between <Cloud Provider> and <CDN Provider>. For future tests that was always an edge case that we couldn't fully test. Rinse and repeat those kinds of problems with every vender, service, and traffic source.
shadowgovt 4y ago
> What does that mean though? "Every company has a test environment. Some companies are disciplined enough to have a production environment separate from the test environment."
- csours 4y ago
  Everyone tests in prod. Some also test before prod.

by smt88 4y ago

michaelt 4y ago

> It's impossible to create a production-like test environment for a product like Twitter. What makes you think that? I mean, I know it costs money and needs people who understand how the product works and how customers behave - but why on earth would it be impossible?

smt88 4y ago
It's impossible because you need to simulate the volume and variety of Twitter usage. Especially when you consider that production Twitter is thousands of servers, you're just not going to be able to create a mirror of production. For something with many millions of users, it makes more sense to do canary releases to internal users, then to a few thousand, and then scale up from there.
- michaelt 4y ago
  If you can accept, index and store a given volume of tweets, why should it be impossibly difficult to generate that same volume? I would imagine it would be a lot easier, in fact. And if can't afford a 1:1 scale replica, build a 1/10th scale replica and have your senior engineers use their expertise to ensure it's representative enough for test purposes.
  - sjsdaiuasgdia 4y ago
    It's a much harder target to generate truly representative load. Sure, you could make millions of tweets very fast, but is that actually representing what the production service is dealing with? Unstable connections. Traffic spikes that can be highly localized or nearly system wide. System maintenance, planned and reactive. Poorly behaved clients. Malicious traffic. Many more similar factors...and yes, each of these you could throw engineers at them and get solutions eventually. You might also end up with a testing engineering team that dwarfs the product engineering team. Also, Twitter doesn't make money. $5B revenue vs $5.5B opex in 2021. They can't really afford to run the actual system, much less a fully representative one. Even a 1/10th replica, which would be unable to find some classes of problems no matter how many engineers you throw at, would further break the bank. As would the engineering staff to support it.
  - smt88 4y ago
    Twitter can't afford a 1:1 replica for sure, and it would be meaningless without globally distributed production load anyway. A 1:10 replica is fine, but also astronomically expensive. What's better about it than 1:100? Or 1:10000? Once you give up on 1:1 (which is reasonable), you're either testing in production or not really testing at all. You'll have many lines of code that will only apply to the 1:1 scale.
- TechBro8615 4y ago
  You don’t need one staging environment simulating the entire platform. You can have small staging environments that run some parts of the product, for example the latest version of the advertiser dashboard or the messaging inbox. Sure, you won’t be simulating a whole class of problems that can arise at scale, but you can still test the more localized features in a safe environment.
  - smt88 4y ago
    Yes, I understand that. That's why it's unclear what it means that Twitter has "no test environment". That could mean a lot of things, some catastrophic and some not.
    - m00x 4y ago
      It's pretty simple. They have no testing environment. They yeet shit in prod under feature flags (I assume), and hope for the best.
bragr 4y ago
>What makes you think that? For large scale sites you need a parallel copy of the internet which doesn't exist. For example, we once caused a production incident (for us and other people) by simulating a major business event to our test stack and saturated links between <Cloud Provider> and <CDN Provider>. For future tests that was always an edge case that we couldn't fully test. Rinse and repeat those kinds of problems with every vender, service, and traffic source.

shadowgovt 4y ago

> What does that mean though? "Every company has a test environment. Some companies are disciplined enough to have a production environment separate from the test environment."

csours 4y ago
Everyone tests in prod. Some also test before prod.