10 ms·

Why we don't schedule deployments during off-hours

85 points by joshuacc 16y ago

gcb 16y ago
launching at off-hours or friday night is a way to the boss to abuse young kids
thirdstation 16y ago
I do the same with minor exception. If the busiest time is during working hours you maybe want to avoid it if you are taking in money during that time. Another good reason for doing maintenance during working hours is that's when your network admins are also working - or anyone else outside your group whose help you might need.
protomyth 16y ago
I would prefer this, but I currently have an edu gig and doing anything while students are in class is a serious recipe for disaster.
twymer 16y ago
I think it really depends on the market too. If you're deploying something that businesses use every day, taking it down during business hours is a bit of a problem. If it's something I pay for and use personally, I would probably prefer it be down while I'm at work and not 5 minutes after I get home.
- nopal 16y ago
  Are more businesses moving towards deployments that have zero downtime? The company I work for does not, but it seems like a lot of others are using various strategies to bring up a new version of an application and then transition users over to that, rather than bringing down the old one and then bringing up and new one.
briandoll 16y ago
It's been many years since I worked in a shop that didn't deploy during the day (large financial institution aside), and for all the reasons the post mentions I completely agree. What enables a mid-day deploy, however, is a well thought-out deploy process that aims for a zero downtime deployment. This can be especially complicated when you have relational database changes baked into a deploy. It's interesting to see some folks suggesting that they may not be able to do this during business hours. Here's the thing: when bad things happen, you need to deploy ASAP. Developing a great deploy process that allows your system to remain standing during that deployment has numerous advantages, including the ability to deploy at will, and have no fear in doing so.
- sokoloff 16y ago
  Relational DB changes don't have to be particular show-stoppers, provided you're willing to accept a few constraints: 1. No select * in SQL; name your columns in selects and inserts (prevents the select result set from changing or the insert from breaking when you add a column later) 2. All tables need table aliases in join queries. (Otherwise, adding a column can cause a naming collision and introduce ambiguity into a presently non-ambiguous select.) 3. Transactional columns can be added, provided they have defaults or are nullable 4. Columns can only be dropped in two releases, one to remove all references to them and the second to actually remove. (You may want to rename the column first during the second release and drop it a few minutes/hours later once you're sure no one is relying on it.) 5. Modifying a column requires some care (to insure that you don't lock the table too long during the modification). Here again, you can probably add a column of a different name, gradually populate it, then rename into place later, add a new table and left outer join to that, or wait for a future downtime event (rarely required in practice). It's not necessarily "trivial" to accomplish, but it's also not deep, black magic.
  - nl 16y ago
    This is a good guide, but note that he said zero downtime deployment That imposes an additional constraint - the new code needs to be backwards compatible (while any data migration scripts are running), and - in a clustered environment - it needs to be able to deal with the case where old and new code bases and databases are available and running simultaneously.
brown9-2 16y ago
The best approach is to design and build a system so that you can make updates to the live system with minimal impact: no need to stop and start servers, risk unaffected components being down, etc. Of course a lot of this depends on your framework, tools, etc., but a little forethought can go a long way.
barclay 16y ago
As others have mentioned, this really only works if you have a simple (and solid) deploy and rollback practice. Most people don't. Given that, all it takes is one terrible deploy: shit goes way south and you're down for most of the day. Now you have the CEO standing around your desk nervously looking at you trying to re-federate the databases when you to realize a 7PM deploy is not all that bad. >Deploying a major update when I’m not in work-mode is awkward as well. I'm sorry, no. I love my family and time, too... But logging in from home for an hour or two in the evening to do a deploy and validation isn't that much to ask, especially considering most of our salaries. If you're that sensitive about it, come in late the next morning.
- joedev68 16y ago
  barclay: You obviously missed the point that "with customers around the world. There isn’t a perfect time for a deployment". Come on everyone, we've got to get passed this US-centric world-view, especially when it comes to web apps. If you're down, you're down. It's bad no matter what time of day. Maybe it feels worse and is more stressful when it's during the day because the CEO is in the office standing over your shoulder, but that doesn't make deploying at night a substitute for bad deployment environments or planning.
  - smackfu 16y ago
    Not many companies have a customer-base that is spread evenly through the world. Not having a "perfect time" doesn't mean there isn't a "better time" than 9 AM EST.
- catshirt 16y ago
  sounds like the issue here actually lies in the suggestion that most people don't have a sufficient deployment mechanism.
  - fleitz 16y ago
    Yup that is usually the problem, when deploys are the exception rather than the rule you are looking for problems.
Murkin 16y ago
Our deployment process was always at night (10,000+ user org). Even if your process is great and there is no downtime during upgrades, any problem might cause the whole organization to loose thousands of work hours (instead of just your team working during the night). If your deploying a "nice-to-have" app, its all fine. Otherwise, get to work late, order a pizza and prepare for a night of adventure.
- danilocampos 16y ago
  > prepare to [have] a night of adventure. This really put a smile on my face. That's the best approach to take with anything, but it's especially applicable to technical stuff that's complex and hard to predict. I've wasted a fair amount of time being pissy and figured out this: with computers, as in life, the only thing over which you have complete control is your attitude. Plan to have fun and learn things and even the sub-optimal can be a good time.
  - itsderek23 16y ago
    > prepare to [have] a night of adventure. This is exactly why we don't deploy at night. In what other line of work do you schedule complex work when you're going to be tired?
    - Murkin 16y ago
      I really hope they don't start fixing roads during the day.. you know.. so the workers are not tired. Or major electrical upgrades.. or phones.. or local internet backbone.. or water.. or what ever other utility that is more important to my life than github (which was down today) How hard is it to work at night one time ??
      - InclinedPlane 16y ago
        For roads it's almost impossible to avoid impacting use during maintenance, the same isn't, or shouldn't, be true with software.
      - 16y ago
plnewman 16y ago
This article is a little disappointing. The writer brings up the obvious advantages to it, but doesn't say how he manages the technical risk of the activity (unplanned downtime).
ojbyrne 16y ago
I think launching on weekends (if your usage is lower then) is a viable alternative. You can be fresh compared to doing stuff in the middle of the night and you can give everyone a day off during the week to make up for it.
keltex 16y ago
My addendum to this rule is I never deploy new features a week before I go on vacation.
- sophacles 16y ago
  I like to have a web app launch the upgrade script. That way I can log in from my fone and start the upgrade right before I get on the plane. Whats the point in having minions if they can't finish the details for you?
code_duck 16y ago
I schedule deployments during off hours. The key is to wake up at either 4 pm or 3 am.
- code_duck 16y ago
  Wait, did someone think I wasn't serious?
davidu 16y ago
What are off-hours? This is the Internet. We used to have off-peak times, but even those days are mostly gone now. Anyways, I agree with the spirit of this article. Best to do pushes when everyone is around or paying attention and you are refreshed, in case you just unleashed a disaster. :-)
- bmj 16y ago
  While our web applications get around-the-clock use during the business week, there is plenty of downtime on the weekends. Really disruptive stuff (generally hardware or OS-level changes) is done on the weekends, but normal deployments and updates are usually scheduled in the morning, for the same reasons outlined in the original post.
SebMortelmans 16y ago
I'm a big advocate of gradual deployment, both in terms of features (limiting your changelog) as towards userbase (not deploying it for everybody at once if it's a sensitive update). Not really too concerned anymore about what time of the day, I think OP has a few valid points there.
semipermeable 16y ago
A good approach I've used before is: - no Friday deployments - deployments run primarily by a single person should be done early in their work day - avoid Monday deployments ... people are usually fuzzier on Monday than they are on Tuesday, after having had a day to go through the weekend's email and get their heads back into work.
mikedanko 16y ago
My body and mind have taken a beating I don't think I can do much longer from ten years of useless night work swinging shifts sometimes three times a week, that guy is too happy. That's a great list of personal reasons, but business can come first for the same reasons. My last big deployment was to half a million set top boxes. In the end, I had to do it at night, but I had oh so many reasons for doing it during the day: * What if a percentage of the boxes bricked halfway through the deployment? A statistical insignificance in the lab doesn't help me help 15k people with bricks on their TV's. When they start playing with the boxes, I'm going to have to roll trucks and that's expensive. Instead man up the call center during a time where we don't have to pay shift diff. * From an end-user standpoint, there's a different set of eyes on the products at different times of day. Think about it this way, if you sell porn, how much are you selling at 2AM vs. 9AM do you think? They're not going to call and complain about their porno not working, but it doesn't mean there's not a huge level of dissatisfaction. There's a lot of general economics and trade offs involved. * Top notch help has the luxury of sleeping at this hour because they've probably earned the tenure. I'm not going to get the vendor of a vendor to help me with this stuff at 3am, and they're definitely not going to be fresh and chipper, and getting anyone beyond support on the phone takes hours -- by that time, they're awake anyway. Change the above as you see fit to adapt to your type of work, it's all the same.
rdl 16y ago
I believe in doing deployments for enterprise apps on Saturday or Sunday AM; everyone is still fresh vs. sleep deprived, and if something goes wrong, you have a long time to fix it. Plus, minimal impact on the customers. Another option is using the more-bogus holidays (Presidents' day?), or religious holidays you might not celebrate (Easter isn't that secular, so non-Christians can give it up). The trade for staff is that you get comp time to use later, and usually at some multiple. I love working on Christmas for 4 comp days later. Working a weekend and getting 2-3 days off mid-week is awesome; not only do you get more done while working uninterrupted on a weekend, taking a mini-vacation during the off days is very cost effective. $49 suites in Vegas, woo!
dclaysmith 16y ago
I work in Ireland for a company whose primary market is the US which, as far as builds go, is the best you could ask for! I do builds around 9-10am GMT after a good night's sleep when I'm nice and alert (and have had a cup or two of coffee). All the while, (most of) our target market is fast asleep in the US.

85 points by joshuacc 16y ago

gcb 16y ago

launching at off-hours or friday night is a way to the boss to abuse young kids

I do the same with minor exception. If the busiest time is during working hours you maybe want to avoid it if you are taking in money during that time. Another good reason for doing maintenance during working hours is that's when your network admins are also working - or anyone else outside your group whose help you might need.

protomyth 16y ago

I would prefer this, but I currently have an edu gig and doing anything while students are in class is a serious recipe for disaster.

twymer 16y ago

I think it really depends on the market too. If you're deploying something that businesses use every day, taking it down during business hours is a bit of a problem. If it's something I pay for and use personally, I would probably prefer it be down while I'm at work and not 5 minutes after I get home.

nopal 16y ago
Are more businesses moving towards deployments that have zero downtime? The company I work for does not, but it seems like a lot of others are using various strategies to bring up a new version of an application and then transition users over to that, rather than bringing down the old one and then bringing up and new one.

briandoll 16y ago

It's been many years since I worked in a shop that didn't deploy during the day (large financial institution aside), and for all the reasons the post mentions I completely agree. What enables a mid-day deploy, however, is a well thought-out deploy process that aims for a zero downtime deployment. This can be especially complicated when you have relational database changes baked into a deploy. It's interesting to see some folks suggesting that they may not be able to do this during business hours. Here's the thing: when bad things happen, you need to deploy ASAP. Developing a great deploy process that allows your system to remain standing during that deployment has numerous advantages, including the ability to deploy at will, and have no fear in doing so.

sokoloff 16y ago
Relational DB changes don't have to be particular show-stoppers, provided you're willing to accept a few constraints: 1. No select * in SQL; name your columns in selects and inserts (prevents the select result set from changing or the insert from breaking when you add a column later) 2. All tables need table aliases in join queries. (Otherwise, adding a column can cause a naming collision and introduce ambiguity into a presently non-ambiguous select.) 3. Transactional columns can be added, provided they have defaults or are nullable 4. Columns can only be dropped in two releases, one to remove all references to them and the second to actually remove. (You may want to rename the column first during the second release and drop it a few minutes/hours later once you're sure no one is relying on it.) 5. Modifying a column requires some care (to insure that you don't lock the table too long during the modification). Here again, you can probably add a column of a different name, gradually populate it, then rename into place later, add a new table and left outer join to that, or wait for a future downtime event (rarely required in practice). It's not necessarily "trivial" to accomplish, but it's also not deep, black magic.
- nl 16y ago
  This is a good guide, but note that he said zero downtime deployment That imposes an additional constraint - the new code needs to be backwards compatible (while any data migration scripts are running), and - in a clustered environment - it needs to be able to deal with the case where old and new code bases and databases are available and running simultaneously.

brown9-2 16y ago

The best approach is to design and build a system so that you can make updates to the live system with minimal impact: no need to stop and start servers, risk unaffected components being down, etc. Of course a lot of this depends on your framework, tools, etc., but a little forethought can go a long way.

barclay 16y ago

As others have mentioned, this really only works if you have a simple (and solid) deploy and rollback practice. Most people don't. Given that, all it takes is one terrible deploy: shit goes way south and you're down for most of the day. Now you have the CEO standing around your desk nervously looking at you trying to re-federate the databases when you to realize a 7PM deploy is not all that bad. >Deploying a major update when I’m not in work-mode is awkward as well. I'm sorry, no. I love my family and time, too... But logging in from home for an hour or two in the evening to do a deploy and validation isn't that much to ask, especially considering most of our salaries. If you're that sensitive about it, come in late the next morning.

joedev68 16y ago
barclay: You obviously missed the point that "with customers around the world. There isn’t a perfect time for a deployment". Come on everyone, we've got to get passed this US-centric world-view, especially when it comes to web apps. If you're down, you're down. It's bad no matter what time of day. Maybe it feels worse and is more stressful when it's during the day because the CEO is in the office standing over your shoulder, but that doesn't make deploying at night a substitute for bad deployment environments or planning.
- smackfu 16y ago
  Not many companies have a customer-base that is spread evenly through the world. Not having a "perfect time" doesn't mean there isn't a "better time" than 9 AM EST.
catshirt 16y ago
sounds like the issue here actually lies in the suggestion that most people don't have a sufficient deployment mechanism.
- fleitz 16y ago
  Yup that is usually the problem, when deploys are the exception rather than the rule you are looking for problems.

Murkin 16y ago

Our deployment process was always at night (10,000+ user org). Even if your process is great and there is no downtime during upgrades, any problem might cause the whole organization to loose thousands of work hours (instead of just your team working during the night). If your deploying a "nice-to-have" app, its all fine. Otherwise, get to work late, order a pizza and prepare for a night of adventure.

danilocampos 16y ago
> prepare to [have] a night of adventure. This really put a smile on my face. That's the best approach to take with anything, but it's especially applicable to technical stuff that's complex and hard to predict. I've wasted a fair amount of time being pissy and figured out this: with computers, as in life, the only thing over which you have complete control is your attitude. Plan to have fun and learn things and even the sub-optimal can be a good time.
- itsderek23 16y ago
  > prepare to [have] a night of adventure. This is exactly why we don't deploy at night. In what other line of work do you schedule complex work when you're going to be tired?
  - Murkin 16y ago
    I really hope they don't start fixing roads during the day.. you know.. so the workers are not tired. Or major electrical upgrades.. or phones.. or local internet backbone.. or water.. or what ever other utility that is more important to my life than github (which was down today) How hard is it to work at night one time ??
    - InclinedPlane 16y ago
      For roads it's almost impossible to avoid impacting use during maintenance, the same isn't, or shouldn't, be true with software.
    - 16y ago

plnewman 16y ago

This article is a little disappointing. The writer brings up the obvious advantages to it, but doesn't say how he manages the technical risk of the activity (unplanned downtime).

ojbyrne 16y ago

I think launching on weekends (if your usage is lower then) is a viable alternative. You can be fresh compared to doing stuff in the middle of the night and you can give everyone a day off during the week to make up for it.

keltex 16y ago

My addendum to this rule is I never deploy new features a week before I go on vacation.

sophacles 16y ago
I like to have a web app launch the upgrade script. That way I can log in from my fone and start the upgrade right before I get on the plane. Whats the point in having minions if they can't finish the details for you?

code_duck 16y ago

I schedule deployments during off hours. The key is to wake up at either 4 pm or 3 am.

code_duck 16y ago
Wait, did someone think I wasn't serious?

davidu 16y ago

What are off-hours? This is the Internet. We used to have off-peak times, but even those days are mostly gone now. Anyways, I agree with the spirit of this article. Best to do pushes when everyone is around or paying attention and you are refreshed, in case you just unleashed a disaster. :-)

bmj 16y ago
While our web applications get around-the-clock use during the business week, there is plenty of downtime on the weekends. Really disruptive stuff (generally hardware or OS-level changes) is done on the weekends, but normal deployments and updates are usually scheduled in the morning, for the same reasons outlined in the original post.

SebMortelmans 16y ago

I'm a big advocate of gradual deployment, both in terms of features (limiting your changelog) as towards userbase (not deploying it for everybody at once if it's a sensitive update). Not really too concerned anymore about what time of the day, I think OP has a few valid points there.

semipermeable 16y ago

A good approach I've used before is: - no Friday deployments - deployments run primarily by a single person should be done early in their work day - avoid Monday deployments ... people are usually fuzzier on Monday than they are on Tuesday, after having had a day to go through the weekend's email and get their heads back into work.

mikedanko 16y ago

My body and mind have taken a beating I don't think I can do much longer from ten years of useless night work swinging shifts sometimes three times a week, that guy is too happy. That's a great list of personal reasons, but business can come first for the same reasons. My last big deployment was to half a million set top boxes. In the end, I had to do it at night, but I had oh so many reasons for doing it during the day: * What if a percentage of the boxes bricked halfway through the deployment? A statistical insignificance in the lab doesn't help me help 15k people with bricks on their TV's. When they start playing with the boxes, I'm going to have to roll trucks and that's expensive. Instead man up the call center during a time where we don't have to pay shift diff. * From an end-user standpoint, there's a different set of eyes on the products at different times of day. Think about it this way, if you sell porn, how much are you selling at 2AM vs. 9AM do you think? They're not going to call and complain about their porno not working, but it doesn't mean there's not a huge level of dissatisfaction. There's a lot of general economics and trade offs involved. * Top notch help has the luxury of sleeping at this hour because they've probably earned the tenure. I'm not going to get the vendor of a vendor to help me with this stuff at 3am, and they're definitely not going to be fresh and chipper, and getting anyone beyond support on the phone takes hours -- by that time, they're awake anyway. Change the above as you see fit to adapt to your type of work, it's all the same.

rdl 16y ago

I believe in doing deployments for enterprise apps on Saturday or Sunday AM; everyone is still fresh vs. sleep deprived, and if something goes wrong, you have a long time to fix it. Plus, minimal impact on the customers. Another option is using the more-bogus holidays (Presidents' day?), or religious holidays you might not celebrate (Easter isn't that secular, so non-Christians can give it up). The trade for staff is that you get comp time to use later, and usually at some multiple. I love working on Christmas for 4 comp days later. Working a weekend and getting 2-3 days off mid-week is awesome; not only do you get more done while working uninterrupted on a weekend, taking a mini-vacation during the off days is very cost effective. $49 suites in Vegas, woo!

dclaysmith 16y ago

I work in Ireland for a company whose primary market is the US which, as far as builds go, is the best you could ask for! I do builds around 9-10am GMT after a good night's sleep when I'm nice and alert (and have had a cup or two of coffee). All the while, (most of) our target market is fast asleep in the US.