5 ms·
> The first scenario is they have never set up a test environment in the first place. They're either too lazy to do so, or too lazy to look into how to do it. O
by sagichmal 6y ago
> The first scenario is they have never set up a test environment in the first place. They're either too lazy to do so, or too lazy to look into how to do it. Often confused with being 'too busy to do it'.
One lesson of modern architectures (i.e. anything more recent than the LiveJournal-style Web/App/DB 3-tier stack) is that it is literally impossible to create and maintain a test environment that has enough similarity to prod to be useful.
- UweSchmidt 6y agoAs QA I think testing, test environments, test ressources should be first class concepts in software. Building whatever "modern" architecture twice, and simulating traffic is not easy but possible, at least to an extent. You need to get that extra license for any piece of software for testing during procurement, and provide means to create test ressources, like typically test users as needed. Sounds trivial, but is often so complicated (banks, insurances...) that it should have been considered early during design. So, I wouldn't use the word "lazy", but we could do better.
- virtue3 6y agoIt's not the architecture, it's the representing all the states in prod in testing. Not saying it's impossible, just not always feasible to have it actively "perfect". I've found that an "ok" testing envrionment and solid unit testing / monitoring tends to do better. And then b/g deploys and canaries :/
- sagichmal 6y agoModern architectures look like this: https://github.com/donnemartin/system-design-primer https://github.com/donnemartin/system-design-primer Building this out twice is either actually impossible, or so cost prohibitive as to be practically infeasible.
- eitland 6y ago> Modern architectures look like this: A lot of modern architecture is not like that. And even the diagram there is something should be possible to replicate if the organization values it. Today it doesn't even need to be too expensive as it can be deployed with terraform and torn down an hour later when the full system tests are finished qnd go back to running the integration tests.
- sagichmal 6y agoIt's not possible to replicate the CDN, or DNS, or the message queue if it's hosted, or the database of it's big enough, or etc. etc. The differences between what you're able to create in a staging environment and what exists in prod are significant enough that testing against the staging environment doesn't build much more confidence than testing with mocks. Testing in prod means accepting this reality, avoiding the largely unproductive toil of building and maintaining a staging env, and making the production system observable, resilient, and operationally agile enough that you can deploy changes that can trigger unpredictable and emergent system behaviors while managing risk appropriately.
- Misdicorl 6y agoTo be useful? That seems to be an extraordinary low bar to hurdle for a QA environment. I imagine most clear that bar easily. I know mine does.
- baddox 6y agoCan you explain that more? Aren’t most of the complex modern tools specifically designed to automate provisioning servers and deploying code?
- isbvhodnvemrwvn 6y agoI'm not the person you're responding to, but having the same software and hardware is possible (most of the time), but production environment is a living thing. To simulate production you need the same data and interaction you have in production, and that's quite difficult to do in many systems, if only for regulatory reasons (PII, PCI and friends).
- sagichmal 6y agoModern architectures look like this: https://github.com/donnemartin/system-design-primer https://github.com/donnemartin/system-design-primer All of the pieces have complex and emergent behaviors as they interact with each other. Many (most?) are hosted, not even fully in control of the team using them. Recreating this environment in a hermetic way for testing is either actually impossible, or so cost prohibitive as to be practically infeasible.
- hampfi 6y agoNot sure where you are working but not a lot of people work at/on a huge code base. Most codebases are build and operated by 1-3 teams and if you are not able to reproduce your prod env as a test env, your architecture is broken.
- sagichmal 6y agoThe size of the codebase is orthogonal to the size/complexity of the production environment. Modern architectures look like this: https://github.com/donnemartin/system-design-primer https://github.com/donnemartin/system-design-primer It is generally not feasible to recreate all of the elements of this design in a totally separate testing environment.
- hampfi 6y agoI know this very well and i'm stlil not seeing an issue. We have 2 GCP accounts. One for production and one for test. Same Setup, not an issue due to terraform and the cost is reasonable for a test env.
- sagichmal 6y agoSame network rules and configuration? Same CDN settings? Same DNS? Same data in the databases? Same volume of traffic on the queues? (Rhetorical questions. Of course the answer is no.) All of these things have huge impact on the actual similarity of the systems, and consequently how they behave.
- hampfi 6y agoObviously but this is not a black and white thing. If it wouldn't make sense for us to have test and prod, we wouldn't do it but i can't imagine a scenario where this is NOT beneficial.