3 ms·
I'm not really sure I like the term "test" for what you do in production, because ideally, tests should have well defined starting and ending states. I feel li
by mr_tristan 5y ago
I'm not really sure I like the term "test" for what you do in production, because ideally, tests should have well defined starting and ending states.
I feel like chaos engineering [1] is a better definition for what's being discussed here - constantly run experiments in prod, but do not consider them "tests". Making sure you _can_ run _useful_ experiments is the key.
1: https://principlesofchaos.org/ https://principlesofchaos.org/
- giantDinosaur 5y agoThe most absolute simple form of production testing is simply trying to use the application to make sure all the parts seem to work based on what you'd expect from a glance. This seems quite distinct from randomly terminating a service or injecting random bytes to see what happens.
- exdsq 5y agoI think there's a gap between running tests without "defined starting and ending states" and chaos engineering. This would be closer to exploratory testing for me, however rather than having a QA team run them you're outsourcing to the users at the expense of risk. Everyone does this (it's called a release) and you can deal with this in two ways - have tracing and monitoring in place to collect the logs from your unpaid testers or ignore it. Add techniques like blue green or canary deployments and you somewhat reduce the risk cost. However this cannot be a replacement for decent prior software testing practices.
- joshuanapoli 5y agoIn a multi-tenant SaaS application, it should be possible to have a predictable begin and end state for a test case. Once you get this, then it's not to hard to make some canary tests periodically check the most important corners of the system.