3 ms·

A pretty pithy response would be that any time you're deploying new software to your production environment, you're testing against real customers. While you sh

by dastbe 9y ago

A pretty pithy response would be that any time you're deploying new software to your production environment, you're testing against real customers. While you should have comprehensive pre-production testing, there's no good substitute for making sure the thing going into Prod is working by running tests when it is in Prod. Looking at her list of production tests, her list generally falls into 3 categories: * instrumentation of code for observability * staged rollout of new software to observe issues on a subset of traffic/hosts against the rest of Prod * simulating failures in Prod to test against possible random failure scenarios Of those, the first should have zero impact on your customers, the second should be making your customer experience better by reducing the blast radius of a bad deployment, and the third should be making your service more resilient to failures over time for your customers. Would you rather find out that you can't survive an availability zone outage when you can easily return that AZ back to healthy in a minute, or when you have to wait for recovery for an hour? Honestly I think its disrespectful to your customers to gamble on what might happen in the future rather than asserting that your service can handle the wide array of known failures that can happen in this world.

falcolas 9y ago
If there's any question of how something is going to respond to production load or production data, there's a missing step somewhere in the release process. WRT the three points you bring up: 1) Instrumentation is a part of any good deploy, regardless of where, when, or why. Instrumentation is not a replacement for testing. 2) Staged rollouts are a good thing, but they don't prove the new object being rolled out is production ready. They can cordon off major failures, yes, but see my comment above about missing steps when it comes to major failures. 3) Not all companies can afford the Netflix model of having multiple fully redundant data centers at all times. I'd even hazard a guess that most companies can't really afford to triple (or more) their infrastructure costs in addition to the higher development and maintenance costs. Ultimately that's one thing that testing is good at: ensuring your disaster recovery plan works without honking off customers when you discover gaps in that plan. None of these three strategies will replace pre-customer testing. At best full adherence to the policies (in the absence of end-to-end testing) will only limit the amount of damage major bugs can do. The question to me is: why are we OK with just limiting the damage of otherwise findable bugs?

by dastbe 9y ago

falcolas 9y ago

If there's any question of how something is going to respond to production load or production data, there's a missing step somewhere in the release process. WRT the three points you bring up: 1) Instrumentation is a part of any good deploy, regardless of where, when, or why. Instrumentation is not a replacement for testing. 2) Staged rollouts are a good thing, but they don't prove the new object being rolled out is production ready. They can cordon off major failures, yes, but see my comment above about missing steps when it comes to major failures. 3) Not all companies can afford the Netflix model of having multiple fully redundant data centers at all times. I'd even hazard a guess that most companies can't really afford to triple (or more) their infrastructure costs in addition to the higher development and maintenance costs. Ultimately that's one thing that testing is good at: ensuring your disaster recovery plan works without honking off customers when you discover gaps in that plan. None of these three strategies will replace pre-customer testing. At best full adherence to the policies (in the absence of end-to-end testing) will only limit the amount of damage major bugs can do. The question to me is: why are we OK with just limiting the damage of otherwise findable bugs?