3 ms·
Perhaps it's just me, but if you can't run your stack locally for any reason but space or metadata, you're too tightly coupled to your platform. This is doubly
by falcolas 9y ago
Perhaps it's just me, but if you can't run your stack locally for any reason but space or metadata, you're too tightly coupled to your platform. This is doubly true in this age of Docker. Of course, Amazon and Google and Azure love tightly coupled applications, but it's not good for creating portable code. And portable code is something that's good for your pocketbook as well as your code quality.
I'll admit, I'm not a fan of testing in production, especially when you're doing that testing against paying customers. I've just seen it go poorly too often, with the common result of lost customers. When you're B2C, it's bad since you're now having to acquire more customers not to grow, but to remain even; a death-knell for VC-backed startups. When you're B2B, the loss of customers signals a coming winter for the business. I've been through a couple of layoffs due entirely to lost customers.
Remember that your product is most likely a fungible asset - your service easily replaceable by a competitor - and if you piss off your customer base by allowing out blatant bugs, you will lose customers. The sheen of newness has worn off tech companies, and customers are not going to be as forgiving of their time being wasted as they once were.
- dastbe 9y agoA pretty pithy response would be that any time you're deploying new software to your production environment, you're testing against real customers. While you should have comprehensive pre-production testing, there's no good substitute for making sure the thing going into Prod is working by running tests when it is in Prod. Looking at her list of production tests, her list generally falls into 3 categories: * instrumentation of code for observability * staged rollout of new software to observe issues on a subset of traffic/hosts against the rest of Prod * simulating failures in Prod to test against possible random failure scenarios Of those, the first should have zero impact on your customers, the second should be making your customer experience better by reducing the blast radius of a bad deployment, and the third should be making your service more resilient to failures over time for your customers. Would you rather find out that you can't survive an availability zone outage when you can easily return that AZ back to healthy in a minute, or when you have to wait for recovery for an hour? Honestly I think its disrespectful to your customers to gamble on what might happen in the future rather than asserting that your service can handle the wide array of known failures that can happen in this world.
- falcolas 9y agoIf there's any question of how something is going to respond to production load or production data, there's a missing step somewhere in the release process. WRT the three points you bring up: 1) Instrumentation is a part of any good deploy, regardless of where, when, or why. Instrumentation is not a replacement for testing. 2) Staged rollouts are a good thing, but they don't prove the new object being rolled out is production ready. They can cordon off major failures, yes, but see my comment above about missing steps when it comes to major failures. 3) Not all companies can afford the Netflix model of having multiple fully redundant data centers at all times. I'd even hazard a guess that most companies can't really afford to triple (or more) their infrastructure costs in addition to the higher development and maintenance costs. Ultimately that's one thing that testing is good at: ensuring your disaster recovery plan works without honking off customers when you discover gaps in that plan. None of these three strategies will replace pre-customer testing. At best full adherence to the policies (in the absence of end-to-end testing) will only limit the amount of damage major bugs can do. The question to me is: why are we OK with just limiting the damage of otherwise findable bugs?