5 ms·

> Do not make production changes on Fridays I ~hate~ dislike this advice. If you can't deploy on a Friday, you need to fix your deployment strategy. By removin

> Do not make production changes on Fridays I ~hate~ dislike this advice. If you can't deploy on a Friday, you need to fix your deployment strategy. By removing Friday from when you can deploy, you're wasting 1/5 of your available days. Note: deploy != Release[1]. Use flags, canaries etc. [1]: https://andydote.co.uk/2022/11/02/deploy-doesnt-mean-release/ https://andydote.co.uk/2022/11/02/deploy-doesnt-mean-release... Edit: hate is far too stronger word for this

lopatin 4y ago
Interestingly my company only deploys on Friday because it has to wait for (most) markets to close for the weekend.
- pondidum 4y ago
  Oh now that is an interesting take! Would that still be required if a deployment and a release are decoupled entirely, or is it unavoidable? Genuinely interested!
  - lopatin 4y ago
    Not totally sure of the distinction. We have versioned weekly releases and they roll out to prod on Friday.
dopylitty 4y ago
This one made me laugh. I've been places that only allow deployments on Fridays because it gives the whole weekend to fix things if they break. It's a good interview question as a candidate. If you ask the interviewer when they deploy and they say only Friday (or worse only once a month) then perhaps look elsewhere for your own sanity because it's a sign of serious malfunction either organizationally, technically, or both.
- fragmede 4y ago
  Depending on your role, that is. If your desired position is straight dev with minimal to no ops work as possible, then yeah, red flag. However, if you're an SRE/DevOps-type person, setting up a continuous deployment system so they can deploy more often than that is a perfect landing task to dig your teeth into. Different strokes for different folks.
- tbrownaw 4y ago
  If there's a very strong "only during standard work hours" usage pattern with SLA penalties for downtime, adapting deployment patterns to that reality can maybe be sensible.
- rexarex 4y ago
  I love this idea.
- spc476 4y ago
  > or worse only once a month Don't discount a job because of one deployment per month---it really depends upon the service. I joke that a busy year for me involved four deployments to production, but "production" for me wasn't a website, or even a web-based service, but a service involved in the call path for phone calls. Our customer is [1] the Oligarchic Cell Phone company and the SLAs are pretty severe. I do have to ask---where do you people work where you have multiple deployments per week (or even per day)? To me, that sounds insane! [1] Still is, even though I left a few months ago, and not because of the lack of deployments, but the shoving of Enterprise Agile [2] on the company by new management. [2] Which is anything but Agile.
  - fragmede 4y ago
    Any SaaS company that that has trusted automated testing on the backend, big enough to have a platform team so that there's a continuous deployment (CD) system should be able to do that. It's less about the actual cadence and more about the fact that, given a bug or request for a new feature, developers are empowered to go and do a release.
- bityard 4y ago
  I am on a tools team, so the "customers" for our team are the company's developers. For changes that might cause an extended outage if things go sideways, we generally prefer to do those after-hours or on weekends so that we don't have all the dev teams sitting idle during work hours if something goes wrong on our end. The upshot is that this is fairly rare and we do not have an on-call rotation. If most anything breaks over the weekend, nobody is going to notice or care until Monday morning rolls around.
kevan 4y ago
I'm a huge advocate for CI/CD pipelines and my team owns a lot of them. We're confident enough to deploy anytime but we choose to limit deploys to our team's business hours and not on Fridays. Why? Because we think the return going from deploying 4 days/week to 5 days/week is outweighed by the stress and morale hit of ruined weekend plans if something weird happens. There's probably situations where that extra speed makes a difference but for us deploying to all regions safely can take a full day anyways so it's pretty normal to have multiple changes flowing at the same time.
- pondidum 4y ago
  I understand that, but would counter with that it sounds like deploy == release, and that if that weren't the case, you could deploy more often. However, I will admit it is a trade-off; some engineering time does have to be spent to get there, and perhaps that engineering time is better used elsewhere right now.
  - _vertigo 4y ago
    What is the difference between deploying and releasing? Everywhere I’ve worked, those are the same thing. We deploy probably 10 to 30 times a workday. As for Friday deployments, I tend to avoid deploying big changes on Friday but medium and small are probably fine Friday unless it’s the very end of the day.
    - pondidum 4y ago
      From the link further up the replies: > Deploy is the act of moving software into an environment and running it. > Release is the process of making a feature visible to a user or subset of users. In our case, we deploy often (10s of times per day), and make use of feature flags for releasing. Keeping an eye on data in Honeycomb before and after a deployment and before and after a flag is changed is part of the engineers job. If something doesn't look like it's working as intended, ideally it is disabled with a flag, rather than needing to figure out how a rollback can be done (often it can't), or rushing a fix and roll forward.
elric 4y ago
Hating it seems a little strong. I'm sure that any team far along enough on the quality spectrum can just read this and say "we've moved beyond this worry". The post is titled "general guidance", not "absolute truths". Adjust expectations accordingly.
- pondidum 4y ago
  You're right, hate is far too stronger term for this, I've updated the wording. Thanks.
Sevii 4y ago
The point of not deploying on friday is to reduce the risk of getting paged over the weekend. It's a quality of life move for the oncall team. No deployment strategy will change the fact that deployments are the leading cause of outages. If you can't afford to give up 1/5 of your available deployment days you have a problem somewhere in your CI/CD system.
- nijave 4y ago
  Sure but ideally you have high enough confidence in your software that those types of issues are highly unlikely.
  - maxfurman 4y ago
    Over a long enough time, with enough deployments, unlikely things are guaranteed to happen! In the long run, setting a given feature loose on Monday instead of Friday is unlikely to move the needle in terms of business results, and eliminates a category of risk. I'm not superstitious about much, but No Friday Deployments is one of my rules
    - rexarex 4y ago
      I agree. We have some advocates that really want to say we deploy on Friday, but even they don’t seem super pushy about a Friday deployment when they have weekend plans, and the org as a whole isn’t keen either. It’s high risk and low reward. I’ve also burned myself on this more than once.
grogenaut 4y ago
CI/CD, flags, canaries don't catch everything, and can still cause outages to others. We try and do pretty heavy CI/CD where I work, but not everyone does (we, like everyone, has old systems). It's actually quite easy for us to have the well behaved systems honor release hours or not depending how their release history has gone, or coverage,etc... but they're well behaved, so they usually have great tests, and they're not usually panicked about rolling out after hours, they have their sh*t together. The reason we have core hours release only without director approval (aka director approval required outside core hours) is so you don't piss off another team by paging them after hours, and so you aren't trying to shove out a thing on a system that doesn't have good coverage or by turning off the safeties. In a large company I've noticed many engineers assume urgency even where there isn't. As an approver myself, most of the time someone wants to rush is because they've not even had the convo with their manager on if it's worth the risk, they are assuming urgency because that's when the sprint ends or what some TPM added to a jira ticket 4 months ago. I admit that sounds risky itself (the engineers not having the right risk training) but this is why we have a policy and tooling... most of the times I've dug in they're just very new and worried about perception as a new employee, so my job is to shepherd them through having that convo with their managers which inevitably has the managers saying "yes it can totally wait till monday", and the change is inevitibly a bit more hot than it should be due to accidental deadline pressure.
rexarex 4y ago
I get that people really want to flex that they can deploy on Friday afternoon and NOTHING CAN GO WRONG, but it’s still foolish and flaunts Murphy’s Law. It can wait.
- lockedinspace 4y ago
  Yep, let's not forget that Murphy's Law, whenever you least expect it, boom.
- nikau 4y ago
  Plus they are likely running simple Mickey mouse systems that aren't intertwined with a bunch of other systems maintained by other groups.
charcircuit 4y ago
There isn't a bottleneck in the amount of commits you can ship per day. You just get more changes to roll out on Monday. I do disagree with the absolute statement of not doing it, but I definitely do a risk analysis whenever I ship a change on Friday and avoid anything risky and just push it off until Monday morning. Not all changes can be put behind flags and canaries don't really fix the issue (unless you are okay in blocking important fixes from being rolled out due to your bad change killing the canary)
lockedinspace 4y ago
Seems reasonable, but if you work for a large company, you can't guarantee that a major release (which is a production change) won't cause any unexpected harm. I have worked with quite bit organizations, and deployed on Friday and wasted my entire Friday-night and Saturday morning, rolling back the +130 components that an app had. If you are a small company, or you do not do extra weekend shifts, I understand your point. Elsewhere, you just want to live an adventure every Friday.
doctor_eval 4y ago
You should have both the confidence that you could deploy on Fridays, and the wisdom to know that you shouldn’t.

lopatin 4y ago

Interestingly my company only deploys on Friday because it has to wait for (most) markets to close for the weekend.

pondidum 4y ago
Oh now that is an interesting take! Would that still be required if a deployment and a release are decoupled entirely, or is it unavoidable? Genuinely interested!
- lopatin 4y ago
  Not totally sure of the distinction. We have versioned weekly releases and they roll out to prod on Friday.

dopylitty 4y ago

This one made me laugh. I've been places that only allow deployments on Fridays because it gives the whole weekend to fix things if they break. It's a good interview question as a candidate. If you ask the interviewer when they deploy and they say only Friday (or worse only once a month) then perhaps look elsewhere for your own sanity because it's a sign of serious malfunction either organizationally, technically, or both.

fragmede 4y ago
Depending on your role, that is. If your desired position is straight dev with minimal to no ops work as possible, then yeah, red flag. However, if you're an SRE/DevOps-type person, setting up a continuous deployment system so they can deploy more often than that is a perfect landing task to dig your teeth into. Different strokes for different folks.
tbrownaw 4y ago
If there's a very strong "only during standard work hours" usage pattern with SLA penalties for downtime, adapting deployment patterns to that reality can maybe be sensible.
rexarex 4y ago
I love this idea.
spc476 4y ago
> or worse only once a month Don't discount a job because of one deployment per month---it really depends upon the service. I joke that a busy year for me involved four deployments to production, but "production" for me wasn't a website, or even a web-based service, but a service involved in the call path for phone calls. Our customer is [1] the Oligarchic Cell Phone company and the SLAs are pretty severe. I do have to ask---where do you people work where you have multiple deployments per week (or even per day)? To me, that sounds insane! [1] Still is, even though I left a few months ago, and not because of the lack of deployments, but the shoving of Enterprise Agile [2] on the company by new management. [2] Which is anything but Agile.
- fragmede 4y ago
  Any SaaS company that that has trusted automated testing on the backend, big enough to have a platform team so that there's a continuous deployment (CD) system should be able to do that. It's less about the actual cadence and more about the fact that, given a bug or request for a new feature, developers are empowered to go and do a release.
bityard 4y ago
I am on a tools team, so the "customers" for our team are the company's developers. For changes that might cause an extended outage if things go sideways, we generally prefer to do those after-hours or on weekends so that we don't have all the dev teams sitting idle during work hours if something goes wrong on our end. The upshot is that this is fairly rare and we do not have an on-call rotation. If most anything breaks over the weekend, nobody is going to notice or care until Monday morning rolls around.

kevan 4y ago

I'm a huge advocate for CI/CD pipelines and my team owns a lot of them. We're confident enough to deploy anytime but we choose to limit deploys to our team's business hours and not on Fridays. Why? Because we think the return going from deploying 4 days/week to 5 days/week is outweighed by the stress and morale hit of ruined weekend plans if something weird happens. There's probably situations where that extra speed makes a difference but for us deploying to all regions safely can take a full day anyways so it's pretty normal to have multiple changes flowing at the same time.

pondidum 4y ago
I understand that, but would counter with that it sounds like deploy == release, and that if that weren't the case, you could deploy more often. However, I will admit it is a trade-off; some engineering time does have to be spent to get there, and perhaps that engineering time is better used elsewhere right now.
- _vertigo 4y ago
  What is the difference between deploying and releasing? Everywhere I’ve worked, those are the same thing. We deploy probably 10 to 30 times a workday. As for Friday deployments, I tend to avoid deploying big changes on Friday but medium and small are probably fine Friday unless it’s the very end of the day.
  - pondidum 4y ago
    From the link further up the replies: > Deploy is the act of moving software into an environment and running it. > Release is the process of making a feature visible to a user or subset of users. In our case, we deploy often (10s of times per day), and make use of feature flags for releasing. Keeping an eye on data in Honeycomb before and after a deployment and before and after a flag is changed is part of the engineers job. If something doesn't look like it's working as intended, ideally it is disabled with a flag, rather than needing to figure out how a rollback can be done (often it can't), or rushing a fix and roll forward.

elric 4y ago

Hating it seems a little strong. I'm sure that any team far along enough on the quality spectrum can just read this and say "we've moved beyond this worry". The post is titled "general guidance", not "absolute truths". Adjust expectations accordingly.

pondidum 4y ago
You're right, hate is far too stronger term for this, I've updated the wording. Thanks.

Sevii 4y ago

The point of not deploying on friday is to reduce the risk of getting paged over the weekend. It's a quality of life move for the oncall team. No deployment strategy will change the fact that deployments are the leading cause of outages. If you can't afford to give up 1/5 of your available deployment days you have a problem somewhere in your CI/CD system.

nijave 4y ago
Sure but ideally you have high enough confidence in your software that those types of issues are highly unlikely.
- maxfurman 4y ago
  Over a long enough time, with enough deployments, unlikely things are guaranteed to happen! In the long run, setting a given feature loose on Monday instead of Friday is unlikely to move the needle in terms of business results, and eliminates a category of risk. I'm not superstitious about much, but No Friday Deployments is one of my rules
  - rexarex 4y ago
    I agree. We have some advocates that really want to say we deploy on Friday, but even they don’t seem super pushy about a Friday deployment when they have weekend plans, and the org as a whole isn’t keen either. It’s high risk and low reward. I’ve also burned myself on this more than once.

grogenaut 4y ago

CI/CD, flags, canaries don't catch everything, and can still cause outages to others. We try and do pretty heavy CI/CD where I work, but not everyone does (we, like everyone, has old systems). It's actually quite easy for us to have the well behaved systems honor release hours or not depending how their release history has gone, or coverage,etc... but they're well behaved, so they usually have great tests, and they're not usually panicked about rolling out after hours, they have their sh*t together. The reason we have core hours release only without director approval (aka director approval required outside core hours) is so you don't piss off another team by paging them after hours, and so you aren't trying to shove out a thing on a system that doesn't have good coverage or by turning off the safeties. In a large company I've noticed many engineers assume urgency even where there isn't. As an approver myself, most of the time someone wants to rush is because they've not even had the convo with their manager on if it's worth the risk, they are assuming urgency because that's when the sprint ends or what some TPM added to a jira ticket 4 months ago. I admit that sounds risky itself (the engineers not having the right risk training) but this is why we have a policy and tooling... most of the times I've dug in they're just very new and worried about perception as a new employee, so my job is to shepherd them through having that convo with their managers which inevitably has the managers saying "yes it can totally wait till monday", and the change is inevitibly a bit more hot than it should be due to accidental deadline pressure.

rexarex 4y ago

I get that people really want to flex that they can deploy on Friday afternoon and NOTHING CAN GO WRONG, but it’s still foolish and flaunts Murphy’s Law. It can wait.

lockedinspace 4y ago
Yep, let's not forget that Murphy's Law, whenever you least expect it, boom.
nikau 4y ago
Plus they are likely running simple Mickey mouse systems that aren't intertwined with a bunch of other systems maintained by other groups.

charcircuit 4y ago

There isn't a bottleneck in the amount of commits you can ship per day. You just get more changes to roll out on Monday. I do disagree with the absolute statement of not doing it, but I definitely do a risk analysis whenever I ship a change on Friday and avoid anything risky and just push it off until Monday morning. Not all changes can be put behind flags and canaries don't really fix the issue (unless you are okay in blocking important fixes from being rolled out due to your bad change killing the canary)

lockedinspace 4y ago

Seems reasonable, but if you work for a large company, you can't guarantee that a major release (which is a production change) won't cause any unexpected harm. I have worked with quite bit organizations, and deployed on Friday and wasted my entire Friday-night and Saturday morning, rolling back the +130 components that an app had. If you are a small company, or you do not do extra weekend shifts, I understand your point. Elsewhere, you just want to live an adventure every Friday.

doctor_eval 4y ago

You should have both the confidence that you could deploy on Fridays, and the wisdom to know that you shouldn’t.