3 ms·

It's impossible because you need to simulate the volume and variety of Twitter usage. Especially when you consider that production Twitter is thousands of serve

by smt88 4y ago

It's impossible because you need to simulate the volume and variety of Twitter usage. Especially when you consider that production Twitter is thousands of servers, you're just not going to be able to create a mirror of production. For something with many millions of users, it makes more sense to do canary releases to internal users, then to a few thousand, and then scale up from there.

michaelt 4y ago
If you can accept, index and store a given volume of tweets, why should it be impossibly difficult to generate that same volume? I would imagine it would be a lot easier, in fact. And if can't afford a 1:1 scale replica, build a 1/10th scale replica and have your senior engineers use their expertise to ensure it's representative enough for test purposes.
- sjsdaiuasgdia 4y ago
  It's a much harder target to generate truly representative load. Sure, you could make millions of tweets very fast, but is that actually representing what the production service is dealing with? Unstable connections. Traffic spikes that can be highly localized or nearly system wide. System maintenance, planned and reactive. Poorly behaved clients. Malicious traffic. Many more similar factors...and yes, each of these you could throw engineers at them and get solutions eventually. You might also end up with a testing engineering team that dwarfs the product engineering team. Also, Twitter doesn't make money. $5B revenue vs $5.5B opex in 2021. They can't really afford to run the actual system, much less a fully representative one. Even a 1/10th replica, which would be unable to find some classes of problems no matter how many engineers you throw at, would further break the bank. As would the engineering staff to support it.
- smt88 4y ago
  Twitter can't afford a 1:1 replica for sure, and it would be meaningless without globally distributed production load anyway. A 1:10 replica is fine, but also astronomically expensive. What's better about it than 1:100? Or 1:10000? Once you give up on 1:1 (which is reasonable), you're either testing in production or not really testing at all. You'll have many lines of code that will only apply to the 1:1 scale.
TechBro8615 4y ago
You don’t need one staging environment simulating the entire platform. You can have small staging environments that run some parts of the product, for example the latest version of the advertiser dashboard or the messaging inbox. Sure, you won’t be simulating a whole class of problems that can arise at scale, but you can still test the more localized features in a safe environment.
- smt88 4y ago
  Yes, I understand that. That's why it's unclear what it means that Twitter has "no test environment". That could mean a lot of things, some catastrophic and some not.
  - m00x 4y ago
    It's pretty simple. They have no testing environment. They yeet shit in prod under feature flags (I assume), and hope for the best.

by smt88 4y ago

michaelt 4y ago

If you can accept, index and store a given volume of tweets, why should it be impossibly difficult to generate that same volume? I would imagine it would be a lot easier, in fact. And if can't afford a 1:1 scale replica, build a 1/10th scale replica and have your senior engineers use their expertise to ensure it's representative enough for test purposes.

sjsdaiuasgdia 4y ago
It's a much harder target to generate truly representative load. Sure, you could make millions of tweets very fast, but is that actually representing what the production service is dealing with? Unstable connections. Traffic spikes that can be highly localized or nearly system wide. System maintenance, planned and reactive. Poorly behaved clients. Malicious traffic. Many more similar factors...and yes, each of these you could throw engineers at them and get solutions eventually. You might also end up with a testing engineering team that dwarfs the product engineering team. Also, Twitter doesn't make money. $5B revenue vs $5.5B opex in 2021. They can't really afford to run the actual system, much less a fully representative one. Even a 1/10th replica, which would be unable to find some classes of problems no matter how many engineers you throw at, would further break the bank. As would the engineering staff to support it.
smt88 4y ago
Twitter can't afford a 1:1 replica for sure, and it would be meaningless without globally distributed production load anyway. A 1:10 replica is fine, but also astronomically expensive. What's better about it than 1:100? Or 1:10000? Once you give up on 1:1 (which is reasonable), you're either testing in production or not really testing at all. You'll have many lines of code that will only apply to the 1:1 scale.

TechBro8615 4y ago

You don’t need one staging environment simulating the entire platform. You can have small staging environments that run some parts of the product, for example the latest version of the advertiser dashboard or the messaging inbox. Sure, you won’t be simulating a whole class of problems that can arise at scale, but you can still test the more localized features in a safe environment.

smt88 4y ago
Yes, I understand that. That's why it's unclear what it means that Twitter has "no test environment". That could mean a lot of things, some catastrophic and some not.
- m00x 4y ago
  It's pretty simple. They have no testing environment. They yeet shit in prod under feature flags (I assume), and hope for the best.