4 ms·

Actually, what I learned is that being afraid to deploy on Friday means you're lacking in testing, verification, auto healing and rollback processes. Also, if

Actually, what I learned is that being afraid to deploy on Friday means you're lacking in testing, verification, auto healing and rollback processes. Also, if something goes down in a way that requires a human to work on the weekend, it should result in a postmortem, and all of the components in the deployment chain related to the failure should be evaluated, with new tasks to fix their causes. If it happens multiple times, all project work should stop until it's fixed. This of course is balanced against how much failure your business can tolerate. If the service goes down and nobody loses money, do you really need your engineers working overtime to fix it?

Angostura 8y ago
> Actually, what I learned is that being afraid to deploy on Friday means you're lacking in testing, verification, auto healing and rollback processes. Or being afraid to deploy last thing on Fridays is an admission that maybe... just maybe... you're not infallible
- protomyth 8y ago
  I always assumed that if a deploy was scheduled for Friday, it meant we were all working the weekend because no one had confidence it would go well.
  - minor3rd 8y ago
    My team deploys probably 5 times in a given day, including Friday. They are all small deploys, and none can happen at the end of the day. If shit hits the fan, we rollback and maybe people figure out the root cause over the weekend but they aren't sweating bullets.
    - rjzzleep 8y ago
      I guess all the people commenting are much more competent than the people I've worked with. Even with tests, I've seen startups do double charges on accounts and whatnot on deployments which didn't show up until the next day. I've also seen ops people updating OES where the storage service would segfault a day later. How does DevOps and OES go together in one sentence, you ask? it doesn't, but it just means not all ops people are pure wisdom either. The guy caused others to waste 72 hours of compute resources, because of this. So it's not limited to Dev. And yes, the first company did learn from the double spending bug, but why learn on a saturday? So even if your DevOps practices are amazing and you have 70% test coverage on all your components, that doesn't mean you can't deploy faulty components where the deployment itself appears successful. Now what? Things aren't failing, they appear just fine. Someone has to go in and debug the problem, it may affect multiple components, it may be critical, and a simple rollback may not cut it. Friday deployments are fine for certain components, but surely not as a general rule for everything? Friday deployments are like Monday morning or Friday meetings. You can do them, and most of the times they'll be fine, but maybe out of respect for your colleagues you shouldn't anyway.
    - protomyth 8y ago
      I've worked in places that required a checklist worthy of NASA to deploy software, and also had blackout periods where no changes could be made to the system without an executive team sign-off[1]. The thought of any of them deploying 5 times a day is just beyond anything they would be allowed to do. I would expect most enterprises are closer to the event deploys then rapid, multiple deployments. In that type of "event" style deployments, week night deploys probably are safe, but anything scheduled for Friday is trouble. 1) common in agriculture during harvest season
ultrasaurus 8y ago
+1 Blocking deploys on a Friday is a symptom of a lot of room to grow your operation maturity: the on-call person should be on-call, and able to handle most issues (in this case, probably a rollback) and that's if your tests didn't block the deploy in the first place Code freezes (and that's what blocking deploys are) are a great tool, but primarily for managing your on-call more effectively.
jjeaff 8y ago
Please tell us about this mythical company whose processes are so airtight, that bad code is never deployed.
- peterwwillis 8y ago
  Oh no no, bad code would be deployed all the time =) We just didn't notice it, because auto healing brought back the working site so quickly. Occasionally something would break design parameters and that would require a manual fix, but then that manual fix was added into auto healing... It takes a good deal of design work to get a high level of resiliency, but it's completely within the realm of possibility. Most shops just don't dedicate the effort to it, because they're more worried about shipping new features, and this is understandable. Just different priorities.
klodolph 8y ago
> Actually, what I learned is that being afraid to deploy on Friday means you're lacking in testing, verification, auto healing and rollback processes. Our philosophy is that if nothing ever breaks in production, you are being too conservative with your controls and development. Or if you look at it another way, you can allocate resources towards stability and new features, and the (near) 100% testing/verification/auto healing/rollback coverage means that too much of your resources are allocated to stability and not enough towards new features. Running a service with uptime too close to 100% uptime also causes pathologies in downstream services, and if your never have to fix anything manually the skills you need to fix things manually will atrophy. Or, for our service, - There should be a pager with 24 hour coverage, because our service is critical, - That pager should receive some pages but not too many, so operations stays sharp but not burdened, - Automation and service improvements should eliminate the sources of most pages, and new development should create entirely new problems to solve, - If the service uptime is too high, it should be periodically taken down manually to simulate production failures, and development controls should be reevaluated to see if they are too restrictive. Eliminating all the production errors takes a long time and a lot of effort. Yes, we are spending that effort, but the only way that this process will actually “finish” is if the product is dead and no more development is being done. The operations and development teams can then be disbanded and reallocated to more profitable work. A healthy product lifecycle, in general (and not in every case), should see production errors until around the team is downsized to just a couple engineers doing maintenance. Google calls this an "error budget". We have something similar where I work. https://landing.google.com/sre/book/chapters/embracing-risk.html https://landing.google.com/sre/book/chapters/embracing-risk.... You can phrase it as “afraid to deploy on Friday”, but I think “afraid to cause outages in production” indicates that the blast radius of your errors is too large or that you’re being too conservative.
- dwaltrip 8y ago
  Interesting philosophy and principles (e.g. don't let "manual fixing skills" atrophy). Definitely something to consider. Thanks for sharing
- ASalazarMX 8y ago
  Since that pager seems to prevent sleep (24 hour coverage receiving some pages but not too many), I mustn't be very popular among employees. I prefer my midnight emergencies at a minimum.
maym86 8y ago
This is what I was thinking. If one person is able to break a system with a small mistake then processes are at least as much to blame as the person. Mistakes will always happen because programmers are fallible so a deployment process designed around infalliblity is destined to fail.
Johnny555 8y ago
Actually, what I learned is that being afraid to deploy on Friday means you're lacking in testing, verification, auto healing and rollback processes. I worked in a shop like that -- they had such great testing policies that they did continuous deployment, code went from commit to production as soon as the tests passed. Until the holiday weekend when two code changes had an unexpected interaction and ended up black-holing all new customer activity that weekend. (existing customers were fine, they only lost data for new customers). They could have recovered the data from a log on the front-end servers, but one of the admins noticed an unusual amount of disk space used on the front ends monday morning and just replaced them all (since their auto-healing allowed this without any interruption of service)... and since those logs were only used for debugging problems, they weren't persisted anywhere. It turns out that tests aren't perfect - they only test what you think you need to test. If the service goes down and nobody loses money, do you really need your engineers working overtime to fix it? Money is not the only way to value a service.... but if the service goes down and no one cares, why run the service at all?
- nojvek 8y ago
  Well ... there are different kinds of deploys with different kinds of risks. I usually don’t like a blanket don’t deploy on Friday rule. We can usually rollback with one command easily, have good monitoring and health checks so even though something makes it into prod, it’s super easy to go back. Unless you have changes like you mentioned, weird side effects, db scheme changes, config changes that affect machine configuration. Those are unknown unknown changes. Good practice to hold them back. As to web and assets changes to a css file or a self contained js change, that should only re-deploy the files that changed and generally low risk.

Angostura 8y ago

> Actually, what I learned is that being afraid to deploy on Friday means you're lacking in testing, verification, auto healing and rollback processes. Or being afraid to deploy last thing on Fridays is an admission that maybe... just maybe... you're not infallible

protomyth 8y ago
I always assumed that if a deploy was scheduled for Friday, it meant we were all working the weekend because no one had confidence it would go well.
- minor3rd 8y ago
  My team deploys probably 5 times in a given day, including Friday. They are all small deploys, and none can happen at the end of the day. If shit hits the fan, we rollback and maybe people figure out the root cause over the weekend but they aren't sweating bullets.
  - rjzzleep 8y ago
    I guess all the people commenting are much more competent than the people I've worked with. Even with tests, I've seen startups do double charges on accounts and whatnot on deployments which didn't show up until the next day. I've also seen ops people updating OES where the storage service would segfault a day later. How does DevOps and OES go together in one sentence, you ask? it doesn't, but it just means not all ops people are pure wisdom either. The guy caused others to waste 72 hours of compute resources, because of this. So it's not limited to Dev. And yes, the first company did learn from the double spending bug, but why learn on a saturday? So even if your DevOps practices are amazing and you have 70% test coverage on all your components, that doesn't mean you can't deploy faulty components where the deployment itself appears successful. Now what? Things aren't failing, they appear just fine. Someone has to go in and debug the problem, it may affect multiple components, it may be critical, and a simple rollback may not cut it. Friday deployments are fine for certain components, but surely not as a general rule for everything? Friday deployments are like Monday morning or Friday meetings. You can do them, and most of the times they'll be fine, but maybe out of respect for your colleagues you shouldn't anyway.
  - protomyth 8y ago
    I've worked in places that required a checklist worthy of NASA to deploy software, and also had blackout periods where no changes could be made to the system without an executive team sign-off[1]. The thought of any of them deploying 5 times a day is just beyond anything they would be allowed to do. I would expect most enterprises are closer to the event deploys then rapid, multiple deployments. In that type of "event" style deployments, week night deploys probably are safe, but anything scheduled for Friday is trouble. 1) common in agriculture during harvest season

ultrasaurus 8y ago

+1 Blocking deploys on a Friday is a symptom of a lot of room to grow your operation maturity: the on-call person should be on-call, and able to handle most issues (in this case, probably a rollback) and that's if your tests didn't block the deploy in the first place Code freezes (and that's what blocking deploys are) are a great tool, but primarily for managing your on-call more effectively.

jjeaff 8y ago

Please tell us about this mythical company whose processes are so airtight, that bad code is never deployed.

peterwwillis 8y ago
Oh no no, bad code would be deployed all the time =) We just didn't notice it, because auto healing brought back the working site so quickly. Occasionally something would break design parameters and that would require a manual fix, but then that manual fix was added into auto healing... It takes a good deal of design work to get a high level of resiliency, but it's completely within the realm of possibility. Most shops just don't dedicate the effort to it, because they're more worried about shipping new features, and this is understandable. Just different priorities.

klodolph 8y ago

> Actually, what I learned is that being afraid to deploy on Friday means you're lacking in testing, verification, auto healing and rollback processes. Our philosophy is that if nothing ever breaks in production, you are being too conservative with your controls and development. Or if you look at it another way, you can allocate resources towards stability and new features, and the (near) 100% testing/verification/auto healing/rollback coverage means that too much of your resources are allocated to stability and not enough towards new features. Running a service with uptime too close to 100% uptime also causes pathologies in downstream services, and if your never have to fix anything manually the skills you need to fix things manually will atrophy. Or, for our service, - There should be a pager with 24 hour coverage, because our service is critical, - That pager should receive some pages but not too many, so operations stays sharp but not burdened, - Automation and service improvements should eliminate the sources of most pages, and new development should create entirely new problems to solve, - If the service uptime is too high, it should be periodically taken down manually to simulate production failures, and development controls should be reevaluated to see if they are too restrictive. Eliminating all the production errors takes a long time and a lot of effort. Yes, we are spending that effort, but the only way that this process will actually “finish” is if the product is dead and no more development is being done. The operations and development teams can then be disbanded and reallocated to more profitable work. A healthy product lifecycle, in general (and not in every case), should see production errors until around the team is downsized to just a couple engineers doing maintenance. Google calls this an "error budget". We have something similar where I work. https://landing.google.com/sre/book/chapters/embracing-risk.html https://landing.google.com/sre/book/chapters/embracing-risk.... You can phrase it as “afraid to deploy on Friday”, but I think “afraid to cause outages in production” indicates that the blast radius of your errors is too large or that you’re being too conservative.

dwaltrip 8y ago
Interesting philosophy and principles (e.g. don't let "manual fixing skills" atrophy). Definitely something to consider. Thanks for sharing
ASalazarMX 8y ago
Since that pager seems to prevent sleep (24 hour coverage receiving some pages but not too many), I mustn't be very popular among employees. I prefer my midnight emergencies at a minimum.

maym86 8y ago

This is what I was thinking. If one person is able to break a system with a small mistake then processes are at least as much to blame as the person. Mistakes will always happen because programmers are fallible so a deployment process designed around infalliblity is destined to fail.

Johnny555 8y ago

Actually, what I learned is that being afraid to deploy on Friday means you're lacking in testing, verification, auto healing and rollback processes. I worked in a shop like that -- they had such great testing policies that they did continuous deployment, code went from commit to production as soon as the tests passed. Until the holiday weekend when two code changes had an unexpected interaction and ended up black-holing all new customer activity that weekend. (existing customers were fine, they only lost data for new customers). They could have recovered the data from a log on the front-end servers, but one of the admins noticed an unusual amount of disk space used on the front ends monday morning and just replaced them all (since their auto-healing allowed this without any interruption of service)... and since those logs were only used for debugging problems, they weren't persisted anywhere. It turns out that tests aren't perfect - they only test what you think you need to test. If the service goes down and nobody loses money, do you really need your engineers working overtime to fix it? Money is not the only way to value a service.... but if the service goes down and no one cares, why run the service at all?

nojvek 8y ago
Well ... there are different kinds of deploys with different kinds of risks. I usually don’t like a blanket don’t deploy on Friday rule. We can usually rollback with one command easily, have good monitoring and health checks so even though something makes it into prod, it’s super easy to go back. Unless you have changes like you mentioned, weird side effects, db scheme changes, config changes that affect machine configuration. Those are unknown unknown changes. Good practice to hold them back. As to web and assets changes to a css file or a self contained js change, that should only re-deploy the files that changed and generally low risk.