6 ms·

Our deployment process was always at night (10,000+ user org). Even if your process is great and there is no downtime during upgrades, any problem might cause

by Murkin 16y ago

Our deployment process was always at night (10,000+ user org). Even if your process is great and there is no downtime during upgrades, any problem might cause the whole organization to loose thousands of work hours (instead of just your team working during the night). If your deploying a "nice-to-have" app, its all fine. Otherwise, get to work late, order a pizza and prepare for a night of adventure.

danilocampos 16y ago
> prepare to [have] a night of adventure. This really put a smile on my face. That's the best approach to take with anything, but it's especially applicable to technical stuff that's complex and hard to predict. I've wasted a fair amount of time being pissy and figured out this: with computers, as in life, the only thing over which you have complete control is your attitude. Plan to have fun and learn things and even the sub-optimal can be a good time.
- itsderek23 16y ago
  > prepare to [have] a night of adventure. This is exactly why we don't deploy at night. In what other line of work do you schedule complex work when you're going to be tired?
  - Murkin 16y ago
    I really hope they don't start fixing roads during the day.. you know.. so the workers are not tired. Or major electrical upgrades.. or phones.. or local internet backbone.. or water.. or what ever other utility that is more important to my life than github (which was down today) How hard is it to work at night one time ??
    - InclinedPlane 16y ago
      For roads it's almost impossible to avoid impacting use during maintenance, the same isn't, or shouldn't, be true with software.
    - LaGrange 16y ago
      The big point seems to be "Many web applications, including Scout, have customers around the world. There isn’t a perfect time for a deployment." It is like a road stuck in a permanent rush hour -- if the time of repair has little effect on the users annoyance, then it's a good idea to pick the time to maximize your performance. Working at night isn't hard in the meaning "I don't know if I can pull that off," but in the "I don't know how well will I pull that off" meaning. Anyone can pull an all-nighter or two, but most people will make more errors in the night than during the day.
  - carbocation 16y ago
    Medicine? (Although your hand is usually forced into doing night procedures--it's not so much a choice, I agree.)
    - mikedanko 16y ago
      I missed some context there. I thought "yes, I need medicine after night work". Why did I completely miss the mark? Swing shifts all week.
      - carbocation 16y ago
        Heh, I meant the field of medicine. But I can understand your perspective :)
  - maw 16y ago
    1) It takes a tough man to make a tender chicken. 2) You should know what you're going to do ahead of time, and you should have done tests beforehand. The after-hours work isn't that complex if you're doing it right.
    - lsc 16y ago
      1) It takes a tough man to make a tender chicken. 2) You should know what you're going to do ahead of time, and you should have done tests beforehand. The after-hours work isn't that complex if you're doing it right. 1. you have no idea how many systems have been utterly destroyed by "tough men" working beyond their limits. I have seen this many places I've worked as a SysAdmin... it's like a mind disease, and it's not any good for anyone. 2. You need to prepare ahead of time, and you need an easy way to roll back. but you /always/ need to be prepared just in case something goes wrong. This is the other classic mistake: Make the new guy stay up all night handling the upgrade. I mean, the senior guys have tested it, right? It ends in tears for all involved. the new guy is the least able to deal with the stresses of doing the upgrade (and the rollback, and even if you test the upgrade, nobody tests the rollback) Do your upgrades when your senior people are around, preferably towards the beginning of their shift. If everything goes as planned, it won't matter when you do it, because downtime will be minimal either way. If, however, it throws up on your shoes, I don't know about you, but I am significantly more effective at dealing with problems when I'm fresh. And yeah, that means I'm a wimp. But when you get down to it, most of us are.
masklinn 16y ago
> Our deployment process was always at night (10,000+ user org). You didn't even get to the second paragraph of the article? Really? > Many web applications, including Scout, have customers around the world. There isn’t a perfect time for a deployment.
- sparky 16y ago
  FWIW, many 10k+ user organizations are "around the world" as well. Web apps for server cluster admins have customer bases evenly distributed across every time zone on the planet? Really? There may not be a perfect time, but there are almost certainly better and worse times.
  - LaGrange 16y ago
    Maybe the differences are not big enough to justify increasing the probability of mistakes.
    - sparky 16y ago
      Granted. This is a much more reasonable response to the top of this thread than sanctimonious quoting. See also scottyallen's comment here (http://news.ycombinator.com/item?id=1817378 http://news.ycombinator.com/item?id=1817378). If Google web search has (or recently had) such a thing as "peak hours," I'll hazard a guess that <your web app> does too.
opiuygtfrtghyju 16y ago
The trouble with that is the deployment is expected to be simple and under your control. The real fun bit comes when users get at it. By doing the deployment at night you find out if it worked at 9:00am the next morning when 1000s of employees connect try and do some work. If there are any issues you are then in deep xxxx By doing deployment in the afternoon, people are offline for the hour or so it takes to deploy but then you have all night to fix any issues.
InclinedPlane 16y ago
The better approach is to figure out where the risk in pushing updates is coming from and develop tools to mitigate that. More testing, pseudo-production deployment infrastructure, better change management, more frequent deployments, better understanding of the changes going out with big deployments, etc. The biggest companies in the world (amazon, google, etc.) who have literally millions of dollars an hour/minute/second at risk if they screw up deployments put in place all of these tools so they can push out deployments at peak times and be confident that they aren't going to destroy their business on accident.
- ora600 16y ago
  This is a cost-value calculation and neither approach is "best" at all cases. Does the cost of developing the tools and procedures exceed the cost of planned downtimes during the next X years? Often you build some tools that mitigate some of the risk and plan deployments at a less active hour to mitigate the rest of it.
- scottyallen 16y ago
  When I was at Google working on web search, we didn't push search infrastructure during peak hours. I think this was mostly due wanting to make sure we had spare serving capacity to failover to, in case the push was broken/had performance problems/crashed/etc. Note that "peak hours" isn't the same as "working hours". Just a few hours when we saw our highest traffic. However, we also had a strict policy of no pushing in the middle of the night, Fridays, or weekends, for all the reasons the article listed. Most problems on a relatively mature system occur when you're changing stuff. When stuff breaks, you want to have the people involved be fresh and thinking clearly. You also want everyone who might be needed to fix something available, in the office and in front of their computers if something goes wrong. Lastly, it's not sustainable to have your engineers/operations folks regularly giving up their nights and weekends for routine operations. You should save that karma for when everything goes to hell. This is all predicated on having pushes not negatively affect production traffic when everything goes according to plan, and minimize the impact as much as possible when something does break. One of the best ways I know to do that is to increase the amount of traffic proportional to increases in your confidence that the change is good. The idea is that any push starts off by only affecting a small part of your infrastructure/traffic, and the longer you run it without problems, the more traffic/infrastructure you push it to. At Shopkick, we do this by pushing first to a single machine, watching it for any errors for a few minutes, then gradually pushing it to the rest of our machines. If anything goes wrong, only a portion of traffic was affected, and the portion is well correlated to the probability of having a problem.
scottyallen 16y ago
Deploying new code should never be an "adventure". It should be a simple, replicable process that allows you to safely stop or rollback at any given time. It should have enough checks in place that the impact from any given mistake is minimized. There should never be any "oh shit" moments.
- cullenking 16y ago
  There will always be "oh shit" moments, no matter how much you think there won't. Yeah, you can mitigate alot of them, but things get through. Even using deploys which can be rolled back, frozen dependencies or bundler for us rails folk, shit can and will happen.
  - scottyallen 16y ago
    You're right - I should be more clear what I mean. "Oh shit" moments caused by a change you made due to pushing a new piece of code should never last more than a minute or two, between the time you realize something went wrong and the time you run the command that reverts things back to the known good state. At Shopkick, this usually means running a command which flips a symlink that points at the current code/config to use and restarts/reloads the server in question.
rmc00 16y ago
I work for a company that provides a mission critical service to a lot of people. Maybe we're just spoiled, but we have multiple failover environments. That means that we can deploy in the afternoon by failing the service over to a hot backup during the deployment. When the deploy is finished, we fail the backup system to the production environment and we deploy to the backups while the primary is running. Since there's 0 downtime, we like to work through the process in the morning/afternoons. No all-nighters here, and its definitely not a nice-to-have service. It's all about architecture and design.

by Murkin 16y ago

danilocampos 16y ago

> prepare to [have] a night of adventure. This really put a smile on my face. That's the best approach to take with anything, but it's especially applicable to technical stuff that's complex and hard to predict. I've wasted a fair amount of time being pissy and figured out this: with computers, as in life, the only thing over which you have complete control is your attitude. Plan to have fun and learn things and even the sub-optimal can be a good time.

itsderek23 16y ago
> prepare to [have] a night of adventure. This is exactly why we don't deploy at night. In what other line of work do you schedule complex work when you're going to be tired?
- Murkin 16y ago
  I really hope they don't start fixing roads during the day.. you know.. so the workers are not tired. Or major electrical upgrades.. or phones.. or local internet backbone.. or water.. or what ever other utility that is more important to my life than github (which was down today) How hard is it to work at night one time ??
  - InclinedPlane 16y ago
    For roads it's almost impossible to avoid impacting use during maintenance, the same isn't, or shouldn't, be true with software.
  - LaGrange 16y ago
    The big point seems to be "Many web applications, including Scout, have customers around the world. There isn’t a perfect time for a deployment." It is like a road stuck in a permanent rush hour -- if the time of repair has little effect on the users annoyance, then it's a good idea to pick the time to maximize your performance. Working at night isn't hard in the meaning "I don't know if I can pull that off," but in the "I don't know how well will I pull that off" meaning. Anyone can pull an all-nighter or two, but most people will make more errors in the night than during the day.
- carbocation 16y ago
  Medicine? (Although your hand is usually forced into doing night procedures--it's not so much a choice, I agree.)
  - mikedanko 16y ago
    I missed some context there. I thought "yes, I need medicine after night work". Why did I completely miss the mark? Swing shifts all week.
    - carbocation 16y ago
      Heh, I meant the field of medicine. But I can understand your perspective :)
- maw 16y ago
  1) It takes a tough man to make a tender chicken. 2) You should know what you're going to do ahead of time, and you should have done tests beforehand. The after-hours work isn't that complex if you're doing it right.
  - lsc 16y ago
    1) It takes a tough man to make a tender chicken. 2) You should know what you're going to do ahead of time, and you should have done tests beforehand. The after-hours work isn't that complex if you're doing it right. 1. you have no idea how many systems have been utterly destroyed by "tough men" working beyond their limits. I have seen this many places I've worked as a SysAdmin... it's like a mind disease, and it's not any good for anyone. 2. You need to prepare ahead of time, and you need an easy way to roll back. but you /always/ need to be prepared just in case something goes wrong. This is the other classic mistake: Make the new guy stay up all night handling the upgrade. I mean, the senior guys have tested it, right? It ends in tears for all involved. the new guy is the least able to deal with the stresses of doing the upgrade (and the rollback, and even if you test the upgrade, nobody tests the rollback) Do your upgrades when your senior people are around, preferably towards the beginning of their shift. If everything goes as planned, it won't matter when you do it, because downtime will be minimal either way. If, however, it throws up on your shoes, I don't know about you, but I am significantly more effective at dealing with problems when I'm fresh. And yeah, that means I'm a wimp. But when you get down to it, most of us are.

masklinn 16y ago

> Our deployment process was always at night (10,000+ user org). You didn't even get to the second paragraph of the article? Really? > Many web applications, including Scout, have customers around the world. There isn’t a perfect time for a deployment.

sparky 16y ago
FWIW, many 10k+ user organizations are "around the world" as well. Web apps for server cluster admins have customer bases evenly distributed across every time zone on the planet? Really? There may not be a perfect time, but there are almost certainly better and worse times.
- LaGrange 16y ago
  Maybe the differences are not big enough to justify increasing the probability of mistakes.
  - sparky 16y ago
    Granted. This is a much more reasonable response to the top of this thread than sanctimonious quoting. See also scottyallen's comment here (http://news.ycombinator.com/item?id=1817378 http://news.ycombinator.com/item?id=1817378). If Google web search has (or recently had) such a thing as "peak hours," I'll hazard a guess that <your web app> does too.

opiuygtfrtghyju 16y ago

The trouble with that is the deployment is expected to be simple and under your control. The real fun bit comes when users get at it. By doing the deployment at night you find out if it worked at 9:00am the next morning when 1000s of employees connect try and do some work. If there are any issues you are then in deep xxxx By doing deployment in the afternoon, people are offline for the hour or so it takes to deploy but then you have all night to fix any issues.

InclinedPlane 16y ago

The better approach is to figure out where the risk in pushing updates is coming from and develop tools to mitigate that. More testing, pseudo-production deployment infrastructure, better change management, more frequent deployments, better understanding of the changes going out with big deployments, etc. The biggest companies in the world (amazon, google, etc.) who have literally millions of dollars an hour/minute/second at risk if they screw up deployments put in place all of these tools so they can push out deployments at peak times and be confident that they aren't going to destroy their business on accident.

ora600 16y ago
This is a cost-value calculation and neither approach is "best" at all cases. Does the cost of developing the tools and procedures exceed the cost of planned downtimes during the next X years? Often you build some tools that mitigate some of the risk and plan deployments at a less active hour to mitigate the rest of it.
scottyallen 16y ago
When I was at Google working on web search, we didn't push search infrastructure during peak hours. I think this was mostly due wanting to make sure we had spare serving capacity to failover to, in case the push was broken/had performance problems/crashed/etc. Note that "peak hours" isn't the same as "working hours". Just a few hours when we saw our highest traffic. However, we also had a strict policy of no pushing in the middle of the night, Fridays, or weekends, for all the reasons the article listed. Most problems on a relatively mature system occur when you're changing stuff. When stuff breaks, you want to have the people involved be fresh and thinking clearly. You also want everyone who might be needed to fix something available, in the office and in front of their computers if something goes wrong. Lastly, it's not sustainable to have your engineers/operations folks regularly giving up their nights and weekends for routine operations. You should save that karma for when everything goes to hell. This is all predicated on having pushes not negatively affect production traffic when everything goes according to plan, and minimize the impact as much as possible when something does break. One of the best ways I know to do that is to increase the amount of traffic proportional to increases in your confidence that the change is good. The idea is that any push starts off by only affecting a small part of your infrastructure/traffic, and the longer you run it without problems, the more traffic/infrastructure you push it to. At Shopkick, we do this by pushing first to a single machine, watching it for any errors for a few minutes, then gradually pushing it to the rest of our machines. If anything goes wrong, only a portion of traffic was affected, and the portion is well correlated to the probability of having a problem.

scottyallen 16y ago

Deploying new code should never be an "adventure". It should be a simple, replicable process that allows you to safely stop or rollback at any given time. It should have enough checks in place that the impact from any given mistake is minimized. There should never be any "oh shit" moments.

cullenking 16y ago
There will always be "oh shit" moments, no matter how much you think there won't. Yeah, you can mitigate alot of them, but things get through. Even using deploys which can be rolled back, frozen dependencies or bundler for us rails folk, shit can and will happen.
- scottyallen 16y ago
  You're right - I should be more clear what I mean. "Oh shit" moments caused by a change you made due to pushing a new piece of code should never last more than a minute or two, between the time you realize something went wrong and the time you run the command that reverts things back to the known good state. At Shopkick, this usually means running a command which flips a symlink that points at the current code/config to use and restarts/reloads the server in question.

rmc00 16y ago

I work for a company that provides a mission critical service to a lot of people. Maybe we're just spoiled, but we have multiple failover environments. That means that we can deploy in the afternoon by failing the service over to a hot backup during the deployment. When the deploy is finished, we fail the backup system to the production environment and we deploy to the backups while the primary is running. Since there's 0 downtime, we like to work through the process in the morning/afternoons. No all-nighters here, and its definitely not a nice-to-have service. It's all about architecture and design.