3 ms·
If there's any question of how something is going to respond to production load or production data, there's a missing step somewhere in the release process. WR
by falcolas 9y ago
If there's any question of how something is going to respond to production load or production data, there's a missing step somewhere in the release process.
WRT the three points you bring up:
1) Instrumentation is a part of any good deploy, regardless of where, when, or why. Instrumentation is not a replacement for testing.
2) Staged rollouts are a good thing, but they don't prove the new object being rolled out is production ready. They can cordon off major failures, yes, but see my comment above about missing steps when it comes to major failures.
3) Not all companies can afford the Netflix model of having multiple fully redundant data centers at all times. I'd even hazard a guess that most companies can't really afford to triple (or more) their infrastructure costs in addition to the higher development and maintenance costs. Ultimately that's one thing that testing is good at: ensuring your disaster recovery plan works without honking off customers when you discover gaps in that plan.
None of these three strategies will replace pre-customer testing. At best full adherence to the policies (in the absence of end-to-end testing) will only limit the amount of damage major bugs can do. The question to me is: why are we OK with just limiting the damage of otherwise findable bugs?