7 ms·

No deployments on Fridays: A good practice for software development teams

28 points by noahhh 3y ago

deleted 3y ago
[deleted]
anthonyskipper 3y ago
I get the point here but for many organizations and types of companies this is a terrible and unworkable idea. A lot of companies have to wait till weekend for maintenance (e.g. trading/market making apps), and so your only options are deploy on weekend which means weekend work or on friday afternoon/evening. Once week starts there are no good times to do a release. So for a vast number of super critical systems friday is THE day to do a release. Yes, that means potential weekend work if it goes wrong, and it increases the odds of a bad monday... but both of those can be fixed with good testing/automation.
- redserk 3y ago
  I think it depends entirely on who the end user is. It's completely understandable in situations where there is no allowable downtime except the weekend. But if it's a SaaS product to categorize sticky notes... probably not worth risking ruining a Friday.
randall 3y ago
I’m 50/50 on this. Then Thursday becomes “omg ship it ASAP” day and Friday becomes firefight Friday. I think meta had it right: put a little warning on the ship button “it’s after 1pm on Friday, do you really need to ship this now?”
sponaugle 3y ago
"Reduces stress: Deploying new code to production can be a stressful experience, especially if it's done at the end of the week. By following a "no deployments on Fridays" policy, teams can reduce their stress levels and focus on other tasks." It would seem more valuable to instead put efforts to make code deploys less 'stressful'. If you are regularly deploying code and causing huge stress you are doing something else wrong. This reminds me of the old corporate 'you can wear jeans on Friday'. Somehow wearing them on Friday is ok, but the rest of the time will lead to dispair?
- zbentley 3y ago
  That's fair, but easier said than done. Reducing the stress/risk of deploys is a long, hard process at the best of times, and also is something that you have to continue doing in perpetuity as the complexity/failure impact of the systems being deployed grows. It's also an area where many organizations struggle to (re)commit resources appropriately. If you ever find yourself behind the curve in that area (as in: deploys are risky and stressful events), then a good short-term mitigation for the symptoms might be to avoid deploys on Fridays or similar, while also committing resources towards improving the root cause (deploys suck).
- blackoil 3y ago
  You can strive to reduce stress but may not be able to bring it to 0. Like you should have for extinguishers even if all precautions have been taken to prevent fire.
jph 3y ago
If deploying new code is stressful, then here are some tips to try... Improve your deploying until you feel it's easy, routine, pipelined, automatic, monitored, tested, and post-deploy tracked so it's provably working. Good practical ways to start on this path can include devops, IaC, E2E testing, etc. Then get good at tactics such as rainbow rollouts, blue-green, canaries, feature flags, dark launch, self-throttling capabilities, and self-healing systems. A simple example is to deploy to 1% of people, and when incoming metrics prove success, then increase the percentage. Success looks like fearless deployments. Some teamwork ideas to try are group ownership, blameless incident responses, causal analysis based on system theory (CAST), service level agreements with stakeholders, and specific software quality attributes for availability, recoverability, and the like.
- redserk 3y ago
  Ideally this is where everyone ought to be anyways. In a vacuum, all of these would stop bad deployments in theory. But in practice, no amount of prep can address and prevent every potential way a deployment can go sideways. Plus if you're B2B, your clients are likely mentally checked-out on Friday anyways so they won't even see the new stuff.
cheeseface 3y ago
For software orgs deploying web services, ”no deployments on Fridays” should be seen as a temporary workaround, not a best practice to follow. It is a symptom of poor automatic QA processes (linting, automatic tests, etc.) and lack of capabilities like feature flags. The only exception is orgs that need manual QA for regulatory reasons and mobile development, where you follow a weekly/bi-weekly cadence because of external Apple/Google reviews.
- lolc 3y ago
  Exactly! It really depends on the setup. Good CI? Deploy at will. Nonexistent CI? Tuesday morning. Of course it can go wrong in either case. But I will keep less slack when the CI is good.
MichaelRo 3y ago
I'm half joking here but at one of my past workplaces, the development process was something along: fix bug, get reviewed, pull request tested by QA, integrate, retest after integration, run in UAT and only then eventually deploy to production. Anything coming up at any of these phases would throw it back to development ("fix bug" phase) so we were joking that if we get a P0 critical priority bug on Monday morning and aren't done fixing it by afternoon then there's no chance in hell it will make it to the deploy on Friday :)
StayTrue 3y ago
This article doesn’t come right out and say it but repeatedly suggests working on the weekend is the core problem. I like Friday releases because I don’t care about working weekends. In my mind slower uptake from weekend users can help identify problems before major uptake Monday morning. Although that’s theoretical because it hasn’t mattered in practice.
- PretzelPirate 3y ago
  > I don’t care about working weekends. This seems like a take I don't hear often. I won't work on a weekend unless the service is on fire and I'm the only one who can save it (which means I've done a bad job). I'm fine with Friday deployments as long as we have a good release pipeline with multiple stages of validation and slow rollout, but if I see that my team is having to regularly work weekends because we deploy Friday, I'll stop deploying and respect their time.
  - semi 3y ago
    It really depends on a lot imo. If you still get days off it can be nice to have them during most peoples work week so you can get errands done easily. As long as it's not consistently every weekend so you can still do weekend things with other people. It can also be easier to get stuff done on the weekend when most other people/distractions aren't around. I actually am really happy with how my current team works, where you will be on call during the weekend but only expected to respond to pages so usually you just need to be able to respond and not actually working.. if lots of things page it's even more incentive to improve your systems to not get bugged on the weekend. But even if nothing pages you we still take two days of our choice off the next week.
- JohnFen 3y ago
  If I'm working on a weekend, it's because the world is on fire. There is no other reason I'd be willing to give up precious personal time for an employer.
Brajeshwar 3y ago
Here is a different story from another perspective of a niche group of team. I led a few teams — small groups for each project. I realized that quite a few of the team have a strange habit of deploying around 5-6PM on Fridays! What I found out was interesting. They were young, single, and don’t really seem to be those looking out to go to bars, etc. By deploying late, and staying over-time — the company’s rule of “company-sponsored-dinners” after office-time kicks in. They get free dinner, and air-conditioned room which are better than their homes or their room in their shared flats. They anyway ends up watching videos/movies, have dinner, and go back home late. It is not uncommon, in India, to be in offices which are much better than their homes.
- jononor 3y ago
  Haha, that is most excellent detective work. Always deploy on Fridays, and enjoy the perks is also a valid strategy I guess :D
barryrandall 3y ago
A more general, but less catchy version of the rule is: don't do things with a significant probability of requiring an "all hands on deck" response when doing so will be unacceptably slow/difficult.
jaysinn_420 3y ago
A company based on doing deploys doesn't seem to understand that the goal should be de-risking the deploys, not de-risking your weekends? Charity Majors explains it better than I ever could. https://charity.wtf/2019/10/28/deploys-its-not-actually-about-fridays/ https://charity.wtf/2019/10/28/deploys-its-not-actually-abou...
- JohnFen 3y ago
  You can't "derisk" deploys, you can only reduce the amount of risk they incur. Not deploying the day before a nonwork day remains solid advice.
  - bananapub 3y ago
    > You can't "derisk" deploys, you can only reduce the amount of risk they incur. to your mind, what is the definition of the word "de-risk"?
    - JohnFen 3y ago
      Reducing the risk of a deployment that catches fire.
    - hooverd 3y ago
      Remove the risk.
  - jitl 3y ago
    Your response sounds like “you can’t derisk deploys, you can only derisk deploys”
    - JohnFen 3y ago
      I was unclear, or perhaps I misunderstood the comment I was replying to. I read it as "deploying on Fridays isn't a problem if you derisk the deploys". My response was that you can't derisk deploys to the extent that Friday deployments are good practice.
jitl 3y ago
This reads like super low effort blogspam. Seeing this advice from a deploy vendor makes me think, “these people must not be serious about their product”. If your deploys break a lot, fix your testing and deploy process! Then you’ll have much lower stress every day of the week, instead of low stress on Friday and high stress every other weekday. At least figure out how to make rollback super quick and easy. Then at least if something goes wrong at 11am when you’re rolling out the Friday deploy, you can roll back and be stress free again by 1pm.
snake_plissken 3y ago
There are two types of Friday deployments: A) Deploying major changes B) Deploying small changes/bug fixes Don't do A.
mikhailfranco 3y ago
The engineering week ends at 9 a.m. Monday. So 'I'll finish it this week' means you have until Monday morning.
wryoak 3y ago
One of my last employers did this and I begged them to just let a couple people work a five day schedule that included Saturday and or Sunday (I wanted to be one of those people) so we could deploy any day, trouble shoot any day, but they were against it. I think people need two (actually at least three, but that's a different discussion) days off every week, but I also don’t think people need to work identical schedules. Put another way: you must give the castle guards a break from staving off attacks but you must not give them all the same break.
higeorge13 3y ago
I have read in some other post that it’s not about the on-call and development overhead, but the potential operational burden to the company or its customers. What about some bad algorithm which causes millions of dollars in revenue or something blocking your customers over the weekend? Who is going to fix this kind of issues during the weekend? It’s better to deploy on Sundays then.

28 points by noahhh 3y ago

deleted 3y ago

[deleted]

I get the point here but for many organizations and types of companies this is a terrible and unworkable idea. A lot of companies have to wait till weekend for maintenance (e.g. trading/market making apps), and so your only options are deploy on weekend which means weekend work or on friday afternoon/evening. Once week starts there are no good times to do a release. So for a vast number of super critical systems friday is THE day to do a release. Yes, that means potential weekend work if it goes wrong, and it increases the odds of a bad monday... but both of those can be fixed with good testing/automation.

redserk 3y ago
I think it depends entirely on who the end user is. It's completely understandable in situations where there is no allowable downtime except the weekend. But if it's a SaaS product to categorize sticky notes... probably not worth risking ruining a Friday.

randall 3y ago

I’m 50/50 on this. Then Thursday becomes “omg ship it ASAP” day and Friday becomes firefight Friday. I think meta had it right: put a little warning on the ship button “it’s after 1pm on Friday, do you really need to ship this now?”

sponaugle 3y ago

"Reduces stress: Deploying new code to production can be a stressful experience, especially if it's done at the end of the week. By following a "no deployments on Fridays" policy, teams can reduce their stress levels and focus on other tasks." It would seem more valuable to instead put efforts to make code deploys less 'stressful'. If you are regularly deploying code and causing huge stress you are doing something else wrong. This reminds me of the old corporate 'you can wear jeans on Friday'. Somehow wearing them on Friday is ok, but the rest of the time will lead to dispair?

zbentley 3y ago
That's fair, but easier said than done. Reducing the stress/risk of deploys is a long, hard process at the best of times, and also is something that you have to continue doing in perpetuity as the complexity/failure impact of the systems being deployed grows. It's also an area where many organizations struggle to (re)commit resources appropriately. If you ever find yourself behind the curve in that area (as in: deploys are risky and stressful events), then a good short-term mitigation for the symptoms might be to avoid deploys on Fridays or similar, while also committing resources towards improving the root cause (deploys suck).
blackoil 3y ago
You can strive to reduce stress but may not be able to bring it to 0. Like you should have for extinguishers even if all precautions have been taken to prevent fire.

jph 3y ago

If deploying new code is stressful, then here are some tips to try... Improve your deploying until you feel it's easy, routine, pipelined, automatic, monitored, tested, and post-deploy tracked so it's provably working. Good practical ways to start on this path can include devops, IaC, E2E testing, etc. Then get good at tactics such as rainbow rollouts, blue-green, canaries, feature flags, dark launch, self-throttling capabilities, and self-healing systems. A simple example is to deploy to 1% of people, and when incoming metrics prove success, then increase the percentage. Success looks like fearless deployments. Some teamwork ideas to try are group ownership, blameless incident responses, causal analysis based on system theory (CAST), service level agreements with stakeholders, and specific software quality attributes for availability, recoverability, and the like.

redserk 3y ago
Ideally this is where everyone ought to be anyways. In a vacuum, all of these would stop bad deployments in theory. But in practice, no amount of prep can address and prevent every potential way a deployment can go sideways. Plus if you're B2B, your clients are likely mentally checked-out on Friday anyways so they won't even see the new stuff.

cheeseface 3y ago

For software orgs deploying web services, ”no deployments on Fridays” should be seen as a temporary workaround, not a best practice to follow. It is a symptom of poor automatic QA processes (linting, automatic tests, etc.) and lack of capabilities like feature flags. The only exception is orgs that need manual QA for regulatory reasons and mobile development, where you follow a weekly/bi-weekly cadence because of external Apple/Google reviews.

lolc 3y ago
Exactly! It really depends on the setup. Good CI? Deploy at will. Nonexistent CI? Tuesday morning. Of course it can go wrong in either case. But I will keep less slack when the CI is good.

MichaelRo 3y ago

I'm half joking here but at one of my past workplaces, the development process was something along: fix bug, get reviewed, pull request tested by QA, integrate, retest after integration, run in UAT and only then eventually deploy to production. Anything coming up at any of these phases would throw it back to development ("fix bug" phase) so we were joking that if we get a P0 critical priority bug on Monday morning and aren't done fixing it by afternoon then there's no chance in hell it will make it to the deploy on Friday :)

StayTrue 3y ago

This article doesn’t come right out and say it but repeatedly suggests working on the weekend is the core problem. I like Friday releases because I don’t care about working weekends. In my mind slower uptake from weekend users can help identify problems before major uptake Monday morning. Although that’s theoretical because it hasn’t mattered in practice.

PretzelPirate 3y ago
> I don’t care about working weekends. This seems like a take I don't hear often. I won't work on a weekend unless the service is on fire and I'm the only one who can save it (which means I've done a bad job). I'm fine with Friday deployments as long as we have a good release pipeline with multiple stages of validation and slow rollout, but if I see that my team is having to regularly work weekends because we deploy Friday, I'll stop deploying and respect their time.
- semi 3y ago
  It really depends on a lot imo. If you still get days off it can be nice to have them during most peoples work week so you can get errands done easily. As long as it's not consistently every weekend so you can still do weekend things with other people. It can also be easier to get stuff done on the weekend when most other people/distractions aren't around. I actually am really happy with how my current team works, where you will be on call during the weekend but only expected to respond to pages so usually you just need to be able to respond and not actually working.. if lots of things page it's even more incentive to improve your systems to not get bugged on the weekend. But even if nothing pages you we still take two days of our choice off the next week.
JohnFen 3y ago
If I'm working on a weekend, it's because the world is on fire. There is no other reason I'd be willing to give up precious personal time for an employer.

Brajeshwar 3y ago

Here is a different story from another perspective of a niche group of team. I led a few teams — small groups for each project. I realized that quite a few of the team have a strange habit of deploying around 5-6PM on Fridays! What I found out was interesting. They were young, single, and don’t really seem to be those looking out to go to bars, etc. By deploying late, and staying over-time — the company’s rule of “company-sponsored-dinners” after office-time kicks in. They get free dinner, and air-conditioned room which are better than their homes or their room in their shared flats. They anyway ends up watching videos/movies, have dinner, and go back home late. It is not uncommon, in India, to be in offices which are much better than their homes.

jononor 3y ago
Haha, that is most excellent detective work. Always deploy on Fridays, and enjoy the perks is also a valid strategy I guess :D

barryrandall 3y ago

A more general, but less catchy version of the rule is: don't do things with a significant probability of requiring an "all hands on deck" response when doing so will be unacceptably slow/difficult.

jaysinn_420 3y ago

A company based on doing deploys doesn't seem to understand that the goal should be de-risking the deploys, not de-risking your weekends? Charity Majors explains it better than I ever could. https://charity.wtf/2019/10/28/deploys-its-not-actually-about-fridays/ https://charity.wtf/2019/10/28/deploys-its-not-actually-abou...

JohnFen 3y ago
You can't "derisk" deploys, you can only reduce the amount of risk they incur. Not deploying the day before a nonwork day remains solid advice.
- bananapub 3y ago
  > You can't "derisk" deploys, you can only reduce the amount of risk they incur. to your mind, what is the definition of the word "de-risk"?
  - JohnFen 3y ago
    Reducing the risk of a deployment that catches fire.
  - hooverd 3y ago
    Remove the risk.
- jitl 3y ago
  Your response sounds like “you can’t derisk deploys, you can only derisk deploys”
  - JohnFen 3y ago
    I was unclear, or perhaps I misunderstood the comment I was replying to. I read it as "deploying on Fridays isn't a problem if you derisk the deploys". My response was that you can't derisk deploys to the extent that Friday deployments are good practice.

jitl 3y ago

This reads like super low effort blogspam. Seeing this advice from a deploy vendor makes me think, “these people must not be serious about their product”. If your deploys break a lot, fix your testing and deploy process! Then you’ll have much lower stress every day of the week, instead of low stress on Friday and high stress every other weekday. At least figure out how to make rollback super quick and easy. Then at least if something goes wrong at 11am when you’re rolling out the Friday deploy, you can roll back and be stress free again by 1pm.

snake_plissken 3y ago

There are two types of Friday deployments: A) Deploying major changes B) Deploying small changes/bug fixes Don't do A.

mikhailfranco 3y ago

The engineering week ends at 9 a.m. Monday. So 'I'll finish it this week' means you have until Monday morning.

wryoak 3y ago

One of my last employers did this and I begged them to just let a couple people work a five day schedule that included Saturday and or Sunday (I wanted to be one of those people) so we could deploy any day, trouble shoot any day, but they were against it. I think people need two (actually at least three, but that's a different discussion) days off every week, but I also don’t think people need to work identical schedules. Put another way: you must give the castle guards a break from staving off attacks but you must not give them all the same break.

higeorge13 3y ago

I have read in some other post that it’s not about the on-call and development overhead, but the potential operational burden to the company or its customers. What about some bad algorithm which causes millions of dollars in revenue or something blocking your customers over the weekend? Who is going to fix this kind of issues during the weekend? It’s better to deploy on Sundays then.