3 ms·
> Actually, what I learned is that being afraid to deploy on Friday means you're lacking in testing, verification, auto healing and rollback processes. Our phi
by klodolph 8y ago
> Actually, what I learned is that being afraid to deploy on Friday means you're lacking in testing, verification, auto healing and rollback processes.
Our philosophy is that if nothing ever breaks in production, you are being too conservative with your controls and development. Or if you look at it another way, you can allocate resources towards stability and new features, and the (near) 100% testing/verification/auto healing/rollback coverage means that too much of your resources are allocated to stability and not enough towards new features. Running a service with uptime too close to 100% uptime also causes pathologies in downstream services, and if your never have to fix anything manually the skills you need to fix things manually will atrophy.
Or, for our service,
- There should be a pager with 24 hour coverage, because our service is critical,
- That pager should receive some pages but not too many, so operations stays sharp but not burdened,
- Automation and service improvements should eliminate the sources of most pages, and new development should create entirely new problems to solve,
- If the service uptime is too high, it should be periodically taken down manually to simulate production failures, and development controls should be reevaluated to see if they are too restrictive.
Eliminating all the production errors takes a long time and a lot of effort. Yes, we are spending that effort, but the only way that this process will actually “finish” is if the product is dead and no more development is being done. The operations and development teams can then be disbanded and reallocated to more profitable work. A healthy product lifecycle, in general (and not in every case), should see production errors until around the team is downsized to just a couple engineers doing maintenance.
Google calls this an "error budget". We have something similar where I work. https://landing.google.com/sre/book/chapters/embracing-risk.html https://landing.google.com/sre/book/chapters/embracing-risk....
You can phrase it as “afraid to deploy on Friday”, but I think “afraid to cause outages in production” indicates that the blast radius of your errors is too large or that you’re being too conservative.
- dwaltrip 8y agoInteresting philosophy and principles (e.g. don't let "manual fixing skills" atrophy). Definitely something to consider. Thanks for sharing
- ASalazarMX 8y agoSince that pager seems to prevent sleep (24 hour coverage receiving some pages but not too many), I mustn't be very popular among employees. I prefer my midnight emergencies at a minimum.
- klodolph 8y ago> Since that pager seems to prevent sleep… The product has 24/7 pager coverage, but that does not mean that one person has the pager the whole time! At any given time the pager is covered by two or three people in different time zones. The way my team is structured, I will only get paged after midnight if someone else drops the page. And I only have a rotation for one week every couple months or so. There are definitely employees who don’t enjoy having the pager, but we get compensated for holding the pager with comp time or cash (our choice). The comp time adds up to something like 3 weeks per year, and yes, there are people who take it all as vacation. No, these people are not passed over for promotions. No, this is not Europe. So the trade off is that seven weeks a year you carry your laptop with you everywhere you go, maybe do one or two extra hours of work those weeks, and don’t go to movies or plays, and then you get three extra weeks off. Yes, it's popular. People like pager duty because they get to spend extra time with their families, because they like to go camping, or because they want the extra cash. I have once been paged after midnight.
- mmt 8y ago> People like pager duty because they get to spend extra time with their families, because they like to go camping, or because they want the extra cash. Adequately compensating on-call is, of course, the right way to do it. All sorts of considerations that were, otherwise, problems, such as how to ensure a "fair" rotation, magically go away [1]. Unfortunately, it's vanishingly rare, at least among "silicon valley" startups (and maybe all tech companies). I suspect it's one of those pieces of Ops wisdom that's vanished from the startup ecosystem because Ops, in general, is viewed as obsolete, especially by CTOs who are really Chief Software Development Officers. Insofar as it's a prerequisite to all your other suggestions, it makes them non-starters in such companies. [1] Although I suppose if the compensation is too generous, there may still end up being complaining about unfairness in allocation