4 ms·

mudge just testified in response to a Senator's question that Twitter does not have a test env

by seafoam 4y ago

bombcar 4y ago
This is clearly perjury, you can find it at www.twitter.com /s
bushbaba 4y ago
To be fair. Many companies staging is unusable with testing in prod.
- nrmitchi 4y ago
  If I recall correctly, Uber literally did an entire post series about how they do all testing in production using feature flags and request routing, because it was "technically infeasible" at their scale to do anything differently.
  - m00x 4y ago
    Uber really isn't a great company to aspire to when it comes to tech. They have a similar mentality to Facebook of: "Spend more time getting it done than planning it out".
deleted 4y ago
[deleted]
smt88 4y ago
What does that mean though? Twitter must have test environments for developers and automatic tests. Do they just not have a full staging environment? That would be understandable. It's impossible to create a production-like test environment for a product like Twitter. You're better off just doing canary releases or feature flags with slow rollouts.
- michaelt 4y ago
  > It's impossible to create a production-like test environment for a product like Twitter. What makes you think that? I mean, I know it costs money and needs people who understand how the product works and how customers behave - but why on earth would it be impossible?
  - smt88 4y ago
    It's impossible because you need to simulate the volume and variety of Twitter usage. Especially when you consider that production Twitter is thousands of servers, you're just not going to be able to create a mirror of production. For something with many millions of users, it makes more sense to do canary releases to internal users, then to a few thousand, and then scale up from there.
    - michaelt 4y ago
      If you can accept, index and store a given volume of tweets, why should it be impossibly difficult to generate that same volume? I would imagine it would be a lot easier, in fact. And if can't afford a 1:1 scale replica, build a 1/10th scale replica and have your senior engineers use their expertise to ensure it's representative enough for test purposes.
      - sjsdaiuasgdia 4y ago
        It's a much harder target to generate truly representative load. Sure, you could make millions of tweets very fast, but is that actually representing what the production service is dealing with? Unstable connections. Traffic spikes that can be highly localized or nearly system wide. System maintenance, planned and reactive. Poorly behaved clients. Malicious traffic. Many more similar factors...and yes, each of these you could throw engineers at them and get solutions eventually. You might also end up with a testing engineering team that dwarfs the product engineering team. Also, Twitter doesn't make money. $5B revenue vs $5.5B opex in 2021. They can't really afford to run the actual system, much less a fully representative one. Even a 1/10th replica, which would be unable to find some classes of problems no matter how many engineers you throw at, would further break the bank. As would the engineering staff to support it.
faeriechangling 4y ago
The amount of companies that have a REAL test environment is far smaller than the ones with a nominal test environment as well.

mudge just testified in response to a Senator's question that Twitter does not have a test env

bombcar 4y ago

This is clearly perjury, you can find it at www.twitter.com /s

bushbaba 4y ago

To be fair. Many companies staging is unusable with testing in prod.

nrmitchi 4y ago
If I recall correctly, Uber literally did an entire post series about how they do all testing in production using feature flags and request routing, because it was "technically infeasible" at their scale to do anything differently.
- m00x 4y ago
  Uber really isn't a great company to aspire to when it comes to tech. They have a similar mentality to Facebook of: "Spend more time getting it done than planning it out".

deleted 4y ago

[deleted]

smt88 4y ago

What does that mean though? Twitter must have test environments for developers and automatic tests. Do they just not have a full staging environment? That would be understandable. It's impossible to create a production-like test environment for a product like Twitter. You're better off just doing canary releases or feature flags with slow rollouts.

michaelt 4y ago
> It's impossible to create a production-like test environment for a product like Twitter. What makes you think that? I mean, I know it costs money and needs people who understand how the product works and how customers behave - but why on earth would it be impossible?
- smt88 4y ago
  It's impossible because you need to simulate the volume and variety of Twitter usage. Especially when you consider that production Twitter is thousands of servers, you're just not going to be able to create a mirror of production. For something with many millions of users, it makes more sense to do canary releases to internal users, then to a few thousand, and then scale up from there.
  - michaelt 4y ago
    If you can accept, index and store a given volume of tweets, why should it be impossibly difficult to generate that same volume? I would imagine it would be a lot easier, in fact. And if can't afford a 1:1 scale replica, build a 1/10th scale replica and have your senior engineers use their expertise to ensure it's representative enough for test purposes.
    - sjsdaiuasgdia 4y ago
      It's a much harder target to generate truly representative load. Sure, you could make millions of tweets very fast, but is that actually representing what the production service is dealing with? Unstable connections. Traffic spikes that can be highly localized or nearly system wide. System maintenance, planned and reactive. Poorly behaved clients. Malicious traffic. Many more similar factors...and yes, each of these you could throw engineers at them and get solutions eventually. You might also end up with a testing engineering team that dwarfs the product engineering team. Also, Twitter doesn't make money. $5B revenue vs $5.5B opex in 2021. They can't really afford to run the actual system, much less a fully representative one. Even a 1/10th replica, which would be unable to find some classes of problems no matter how many engineers you throw at, would further break the bank. As would the engineering staff to support it.

faeriechangling 4y ago

The amount of companies that have a REAL test environment is far smaller than the ones with a nominal test environment as well.