9 ms·

Why are you still deploying overnight?

113 points by gnubardt 15y ago

coreycollins 15y ago
The article discusses deploying to a percentage of your user base at a time. I have an idea on how to do this, but it is probably the wrong way. Anybody know of any articles explaining this process for a rails app?
- amackera 15y ago
  I am also interested in this concept. What if a change requires a migration of my database?
  - fleitz 15y ago
    Sometimes it absolutely can't be avoided, but usually the last version and the next version can be coded to work off of either schema. As for deploying to a segment of users put a "version" field in the user and then gate features based on the version. If you name your versions or use a unique numbering system it should be fairly easy to remove the old gating. Yes, none of these things are foolproof but then again your existing system probably isn't either. The more you deploy the better you get at it and the more automated it becomes. Force the issue by forcing a deploy everyday to begin with, then ramp up to twice a day, etc. After a month or two you'll learn solutions to almost all your deployment problems and there will be a well-known solution in the organization for solving the problem whether it be gating features or schema migrations.
    - coreycollins 15y ago
      I cringe at the thought of putting versioning gates all over my code, but maybe that's the easiest solution. What about having two code bases on your production server and having your web server (nginx) route the traffic accordingly? 85% goes to the current code base and 15% goes to the new one? If that's possible it would seem pretty easy.
      - fleitz 15y ago
        I think that makes it less testable but would probably work, it's probably a good stop gap as long as you don't want to test more than one "feature" at a time. Gating would probably work better in the long term as you get more devs and users, and need to do things like A/B/C testing.
  - GlennS 15y ago
    A near-instantaneous switchover is possible if you're migrating a database to a new schema, but it's pretty hard work. For this to work, every action that will result in a database write needs to go in log, and that action log needs to be replayable be the updated version. In the context of a web application, you have: 1. Take your database backup, start recording action log. 2. Perform your database migration against your database backup. 3. Install your updated web application (not available to users yet). 4. Test your updated web application. 5. Disable your existing web application. 6. Replay the contents of your action log using the updated web application. 7. Making your updated web application available to your users. It's definitely not a trivial matter to sort out: it means lots of time working on your deployment process, for example you have to be set up to have both versions deployed simultaneously, which likely means a lot of mucking about with URL rewriting. The application also has to be built to support it. Maybe there is a better way?
    - lusis 15y ago
      There is a better way. It's to stop deleting and start deprecating. Write your code in a backwards compatible way. If your storage engine, whatever it is, can't handle it then switch storage engines. The author makes some amazing points that I'm surprised intelligent people are missing. Let me give an example. We are currently migrating from storing blobs of data in Voldemort (a key/value store) to storing them in S3. They should have never been in there in the first place but whatever. We're going to do it with "zero" migration time. In fact we're already doing it. - Set up a job that copies existing data in Voldemort to S3. - Deploy a minor release of our code that multiplexes the current writes to both Voldemort and S3. - Continue migrating existing data - When existing data is finished migrating, deploy a new release that forces all traffic to S3 instead of Voldemort. - Profit People need to learn to do things like dark launching and feature flags. Dark launching let's you exercise new code paths with no impact to the user. Feature flags give you the ability to enable to features to some or all of your users. Feature flags are awesome ways to A/B test as well. People need to stop doing stupid shit like redefining what some property means mid-release and instead define NEW properties and deprecate old ones in subsequent releases. Same goes for schema changes. If some part of your code base cannot tolerate an additional column that it doesn't need, that's a bug. You can also adopt some configuration management that allows you to provision duplicates of the components you're deploying so you can swap back and forth between them in the case where you might have a breaking release. That's what we do and it's one of the upsides of using EC2. All of this requires discipline and dedication but the benefit is so worth it. You have to stay on top of dependencies. You can't let bitrot take hold by going 4 major revisions without upgrading some package. We did and it bit us in the ass. This is why we adopted the swinging strategy of duplicating our entire stack (takes about 30 minutes depending on Amazon allocation latency) on major upgrades.
untog 15y ago
While there is some truth in there, I feel like some of the advice is pretty reckless: "Problem 1: You presume there will be problems that impact availability. You have no confidence in your code quality; or (or maybe, and), you have no confidence in your infrastructure and deployment process." Or you're playing it safe. You absolutely cannot guarantee that every update you are deploying will have zero problems. If your business absolutely relies on users making payments online or anything of that ilk, you could lose a lot of money. "Imagine, for a moment, that your team is rolling out an update to a service that monitors life-support systems in hospitals." What? Why? Throwing out hypothetical "what if" scenarios that would affect 0.1% of your readership isn't a very useful thing to do.
- furyg3 15y ago
  Throwing out hypothetical "what if" scenarios that would affect 0.1% of your readership isn't a very useful thing to do. Indeed. The people who use my production systems are people who are logging transactions which they have made (in the real world) into my system. It's not a hospital. If the system is down, you just come back later. We do a lot to make sure our deployments will go smoothly, downtime is minimized, and they affect as few users as possible. But the effort required for my team to deliver "five nines" would be insane. It's much easier for one guy to take the application server down for 10 minutes (at midnight) once a month.
  - MartinCron 15y ago
    ...the effort required for my team to deliver "five nines" would be insane. It's much easier for one guy to take the application server down for 10 minutes (at midnight) once a month. For the projects I've worked on lately, the ideal of "zero downtime deployments, fully automated, during the daytime, as non-events" isn't at all about getting a particular number of nines, it's about deploying more often than once every month. When the deployment you've been working on for a whole month goes wrong, which of the many hundreds of changes are problematic?
- absconditus 15y ago
  I work on medical software. Surprisingly, most hospital systems require a fair bit of downtime. Hospitals have downtime procedures that they use during these periods (basically switching to manual, paper-based systems).
RyanMcGreal 15y ago
The best reason not to do overnight deployments is that if something goes wrong, you want your developers to be alert and sharp, not dazed and sleep-deprived.
FollowSteph3 15y ago
Its not just cause ur developers are tired, which is a big issue, its also because you may not have access to the people you need to make decisions in the middle of the night if something needs to be escalated. Good luck trying to get an executive or VP up at 4am to make a decision...
schleyfox 15y ago
There are deploys and then there are deploys. 99% of your deploys should be fine to go live whenever the code hits the repo. I definitely still prefer scheduled overnight deploys whenever significant infrastructure changes need to go live. Downtime should still be minimized, but sometimes a maintenance window is necessary.
xbryanx 15y ago
I'm befuddled by these sorts of posts that are big on theory and light on details. I run tests, via a continuous integration server, and have high confidence in our code when we deploy However, we do updates at night because we often need to take our sites off line for a while, to merge our changes into the live codebase. If we didn't, changes to the database structure would bork the db as visitors clicked around. Nighttime is our lowest volume of visits, so that's the best time to deploy. I'm sure this is a noob reaction, but can anyone point me to some good technical walkthroughs about how to deploy to live without taking your site down or interrupting your database connection? Is there a tool/term/practice I'm missing?
- stingraycharles 15y ago
  I don't have a good technical walkthrough, but we're providing a high availability service, and zero-downtime upgrades are a requirement. Our setup basically consists of load balancers and application servers. When we need to upgrade, we require application version x always needs to be backwards compatible with application version x - 1, both on a protocol and data model level. Basically, our upgrade procedure is as follows: 1. distribute new package to all servers; 2. run an additional application service on all servers and run some quick tests on each of them to verify proper working; 3. add new application servers to load balancers; 4. upgrade data model to new version (we use postgresql, and this happends in a single db transaction, and remember that our new version x is compatbible with both the current and the previous data model); 5. remove old application services from load balancers; 6. upgrade successful. If anything goes wrong, we can roll back each of these steps. Note that this whole process, perhaps needless to say, is fully automated. By running multiple versions inside the load balancer at the same time, and having the requirement that version x + 1 is always compatible with version x, this procedure allows us to seamlessly upgrade to a new version without any downtime.
- mechanical_fish 15y ago
  There won't be general-purpose walkthroughs, because the general answer is "don't change the database structure of a database that's serving live traffic: You'll never enumerate all the things that could possibly go wrong." In most specific cases, there are alternatives to stopping the show during an upgrade. First, you check your backups! Then, there are various strategies. You can make many simple schema changes while the system is running. Or put the DB in read-only mode and still mostly manage to serve pages -- perhaps with an occasional HTTP 503, so hopefully that's okay -- during your upgrade. Or phase in changes over several releases, carefully architecting for backward compatibility. Or bring up a parallel system and gradually migrate active users to it. Depending on what you're doing, you may find yourself having to write a special upgrade script that migrates old formats to new formats, or even using database triggers to keep "old" and "new"-format tables in sync during the transition. A well-constrained database may help keep things sensible during the transition -- or, you may have to drop half of your supposedly-sensible constraints just to make the transition work. Moral: Even if you know exactly what you're doing, live updates are more work: More planning, more code, more infrastructure, and/or more stress. In many real-world cases, you should just take the system down for a minute. Focus your engineering effort on making sure that "minute" is as short as possible, and in making sure that you can detect problems and roll back as quickly as possible.
hopeless 15y ago
It's also interesting how U.S.-centric this thinking is: your sneaky 3am deploy in the valley could mean a mid-morning outage in London. Actually, an 8am deployment in Europe has the advantages of a rested brain and a few hours before America hammers the Internet ;-)
- virmundi 15y ago
  Many US based software companies either A) have a mostly American user base or B) have a regional distribution setup for their app. So if you are a non-American in camp A, that is a problem for you. However, remember the goal is to be the least disruptive. If you are a non-American in a region that the server is deployed in, there is probably a night-based take down of the system in your region. If the software company is either A nor B, then the software company need to notify the non-user users that they may experience a disruption.
- mseebach 15y ago
  > Actually, an 8am deployment in Europe has the advantages of a rested brain and a few hours before America hammers the Internet ;-) And the disadvantage of happening in the middle of mainland Europe morning rush. I think the point is that you take the downtime when it's least disruptive to your users, who/wherever they might be.
Vitaly 15y ago
Or may be a client simply insists that customers absolutely can't tolerate a several seconds delays during the deployment ;(
deleted 15y ago
[deleted]
chris_dcosta 15y ago
Couldn't agree more. I worked somewhere that only deployed during business hours because it was all hands on deck. We just kept the users primed with information and reminders prior to go-live, and when it happened everyone knew about it. But there's a more fundamental point. We're dealing with IT, and for internet businesses and global enterprises the issue is the same: business is 24 hours a day 7 days a week. You cannot avoid deploying somewhere in the world that's awake. All you have to do is plan a rollback and just get on with it. My experience of companies who are timid about rolling out, changes is that they're state of advancement is completely unstable. They are too scared to change meaning that some technologies become out-of-sync with the current state of play. You only have to introduce a new piece of wiz-bang software and issues come pouring out, and it's a frantic race to upgrade random components in the hope you might plug the leak. This is far far worse in every way than doing regular "risky" upgrades. Small incremental changes can be easily rolled back, because you know what you just changed.
rilindo 15y ago
"Why are you still deploying overnight?" Because you don't hire a system admin to do that job for you. :}
amalcon 15y ago
The only good point is #2: you want your people to be awake during the deployment so they can deal with any problems. The others basically amount to "You should have other processes in-place to avoid the same problems you're trying to minimize." Three words: Defense in depth. You don't always need every advantage you can get, and this is a pretty costly one. Still, it's simply wrong to assume that a particular precaution is always needless. Interestingly, you hit a bit of a sweet-spot if your primary customer base is located halfway around the world. You can roll out updates when your people are most wide awake, but when you'll only break things for a few users if something goes wrong.
Roboprog 15y ago
Why in the middle of the night? The economics of things, mostly. At my previous job, we were still in process to get session migration to work so servers within a cluster could update while others were running. At my current job, we don't have a cluster (we have a single over-provisioned server), and that method simply isn't in the budget for the foreseeable future. However, 3:00 AM PDT is 3:30 PM IST (India), which is where most of my team is, as well as the support staff they need (most staff there are on swing shift, as well). Thus, an early morning release is actually a very convenient, as well as lower cost, time to do the release. As the site is mostly US customers in both a B2C and B2B use case, this timing (around 6:00 AM EDT) works very well, and I don't see it changing. I realize this doesn't scale for a global app, but it explains the lack of pressure to spend much consideration on higher tech, more expensive, deployment methodologies.
unreal37 15y ago
I was hoping the author would have some deeply insightful answer to the argument, "because the deployment requires the site to go down for a few minutes and we want to avoid inconveniencing customers" but he avoids that topic entirely. That's the number 1 reason for an overnight deployment, by far! If you can deploy code without bringing the site down, then of course, do it during the day!
- maratd 15y ago
  Bingo. Deployment with us is automated. I don't actually stay up to 3 AM. However, during the deployment the machine is automatically updated with new packages and the new version of the software, caches are cleared, it's restarted, etc. All of this probably takes 5 minutes at most, but still ... why do that in the middle of the day? Also, we do a daily backup around midnight. If the deployment is botched, I can come in early in the morning, notice it and simply do a roll-back using the backup. Very easy. Now, if you're actually staying up until 3 AM and doing things manually ... you need to automate things.
  - jonstjohn 15y ago
    Depends on the type of site/application. If your application is managing customer data that can be updated 24 hours a day, a few minutes of down-time might be okay, whereas rolling back 6 hours of data (and losing it) is definitely not okay.
- miker64 15y ago
  He did, you do a rolling update, take a few systems out of active duty, upgrade them, and then begin directing some percent of traffic to them. That, of course, only works if you are of a scale to have that many production systems, and loose enough interdependencies to allow for new and old code to run at the same time. I think some of his point was that if you aren't able to do that sort of update, it was worth your time and effort to get there. I think he's probably right, but the work to get from where you are (it requires site outage to upgrade) to where you want to be (rolling upgrades, ability to run multiple versions in parallel, etc) is going to be site specific in most cases that it's hard to discuss that move in any depth.
kstenerud 15y ago
This is incredibly reckless and naive advice. You roll out when it least impacts customers because: 1. Impacted customers affects your bottom line. Minimize that and you minimize losses if something screws up. 2. Murphy's law. I don't care how much testing and QA you do; stuff will ALWAYS creep through, and sometimes it'll be nasty. QA works to minimize this, but it can't eliminate it entirely due to diminishing returns. Show me your guaranteed bug free deployed code and then I'll consider changing my view. 3. If you've designed and tested your rollback procedure prior to actually doing it, the chances of not being able to roll back in the real deployment is orders of magnitude lower than the chances of a failed deployment requiring a rollback, which in turn, if you've done your testing and QA, is orders of magnitude less likely than a successful rollout (but not 0, thus the midnight rollout). If you're worth your salt, you have a tested rollback procedure, laid out in simple to follow instructions (or better yet, an automated rollback mechanism with a simple-to-follow manual process when the automated method inevitably fucks up). You rollout, and if it fails, you roll back. And if that fails, you use the manual procedure. You should have the entire process time boxed to the worst case scenario (assuming successful manual rollback) so that you know beforehand what the impact is, and won't need to go around waking people up asking what to do.
- MartinCron 15y ago
  A part of this is looking at the risk assessment and the notion of "guaranteed bug free deployed code" in a different way. For a stable system that's in production, you don't need "guaranteed bug free deployed code" you need code that is not any worse than what's currently running out there. Doing frequent (daytime) deployments makes it easier to make a change, test (both with humans and robots) that change, and get it out there. You don't have to manually test everything in order to change anything when you're changing just one thing at a time. I've come to believe that the far riskier approach is to make a bunch of changes at once, introduce a bunch of bugs, test fix bugs until you feel confident, and then release this huge change all at once in the middle of the night.
- lusis 15y ago
  Rollbacks are a myth. You can never rollback. Always be rolling forward. Enabling a culture and environment that allows for small frequent changes solves that problem. The way to not impact a customer is to make deploys trivial, automated and tolerant to failure because everything fails.
krobertson 15y ago
I think a point also missed in the article is the startup notion of burning the midnight oil and pushing stuff live. "Release early, release often" is great, but that same call to action can be what motivates people to push code live at 3am when they're tired, just finished it, and perhaps skipping QA. Otherwise, I think the article as put is empty and seems more straw man. Assumes you have no confidence in your code, assumes it is complicated, assumes you can't roll back. Simply put, deploy small stuff often, make plans for big stuff, and don't be stupid.
whazzmaster 15y ago
I attended a git workshop by Scott Chacon and was surprised to learn that github's production site can be deployed by many folks on the development team, and at any time during the day. In fact, he mentioned that github.com can be updated as many as a dozen or more times per day as small features or fixes are pushed live. There is a great description of their web server stack[1] that allows them to seamlessly push new code live without downtime (see the second Slow Deploys section.) It's a great idea about how to continue serving during a deployment, and I'm looking into using Unicorn like this for some projects I'm working on. [1] https://github.com/blog/517-unicorn https://github.com/blog/517-unicorn
Hominem 15y ago
I rarely have issues but is always a chance that something is going to go wrong, so everyone is sitting there waiting for the all clear. I was able to make the case that having developers, DBAs, and sysadmins sitting around in the middle of the night was counterproductive. It was fraught with tension during the middle of the night after a long day at work. Now we do it at 1pm on saturday, it is amlost leisurely now. Just a quick call to check in. Some people even call the voice bridge from the beach!
ekidd 15y ago
Sometimes, it's cheaper to live with 10 minutes of scheduled maintenance at a low-traffic hour than to re-engineer your application to have 99.999% uptime. Even if you have unit tests, continuous integration, a staging environment, and fast rollbacks, deploying to production can still be tricky: 1) Do you want to run MySQL, PostgreSQL, Oracle or SQL Server? Watch out for ALTER TABLE; it may sometimes hold table locks or invalidate open cursors. Of course, you could always rewrite your application to use a schema-free NoSQL database and migrate your records at read time. But that has a lot of consequences. Maybe 10 minutes of scheduled maintenance is an acceptable tradeoff? 2) Does your staging environment exactly match your production environment? Can you generate a full-sized, real-world load on your staging servers? Alternatively, have you built all the tools required for a phased deployment of your back end? Now, I'm all in favor of deploying to production 5 times a day. That's pretty easy for any group with good unit test coverage and a fast rollback mechanism. But it's expensive to eliminate the occasional 10 minute maintenance window. And if you're going to take the site down for maintenance, it makes sense to do it at an off-peak hour.
- InclinedPlane 15y ago
  I don't think the author's point is that everyone should be doing middle-of-the day seamless deployments from day one. I think the point is that if you continue to do deployments at 3am indefinitely you are hurting yourself. There are lots of things that make sense at the appropriate scale. Not having any tests. Completely manual processes for builds and deployments. Having only one server. Making backups every day or maybe even just every week. Etc. However, as your business grows you need to recognize that it's important to change these things and move to better processes. Because if you don't then those things could become a very serious drag on development velocity and even business capability. You could find yourself spending all your time drowning in process when you should be spending your efforts more efficiently. You stay up until 3am to perform a risky manual deployment, sometimes you spend some time fixing problems afterward. As a consequence you either spend a day in zombie mode with limited productivity or you start working much later. In either case you've taken away a fairly decent chunk of time that could have been used for productive work. Similarly, perhaps you spend too much of every day fighting your build system, or recovering from your build always being on the floor, or excessive operational support because your platform doesn't have sufficient redundancy at every tier. Etc, etc, etc.
jakejake 15y ago
I do think 3AM is a terrible time to do a release. I'm personally a fan of doing releases in the early morning instead of late night. I'd rather get in the office at 6AM & push the release. Plan for a light day and let everybody leave early if things went smoothly. If things do go wrong it gives you about 2-3 hours before the rest of the office start showing up to roll back or fix the situation. Choosing the time should be a balance of off-peak usage time and a reasonable time for the developers. If you're a global business then you may not have a perfect time, but you can usually still choose a day and time that is less likely to cause inconvenience.
nimblegorilla 15y ago
Let's be realistic. Many applications have 24 hour availability, but almost every application I've worked on was for people in the US and core usage was sometime between 7AM and 9PM. If it only takes 5 minutes to deploy your app why not do it at 2AM when few users are online? Deployments should be simple, easy, and NOT require any problem solving by the deployer. Midday deployments with all hands on deck to handle issues seems like a bigger problem to me.
127001brewer 15y ago
"...I sincerely hope you’re not living in the Stone Age..." Whether or not you agree with the entire article, for me, the statement above was the most important one. There is no excuse why any software development team does not have an automated build and deploy process. There may be valid business reasons to deploy in the middle of the night, but there's no business reason to deploy by hand. Also, I am assuming that the posting only reflects application source code and not database changes, which is a different beast in terms of deployments.
jroseattle 15y ago
Because our environment is non-trivial. It has nothing to do with confidence in application code, and everything to do with service-level agreements, impact to existing customers, and our own psyche. Ask SalesForce if they do their maintenance updates on weekends because of a lack of confidence in their codebase. We're not doing 2am deployments, but we're certainly doing them "off-hours".
- lusis 15y ago
  Triviality has nothing to do with it. Nothing in your SLA says you have to have servers 1, 2 and 3 available. It says you have to have service X available. You can do that with out taking a downtime hit.
  - jroseattle 15y ago
    Wow, thanks for having better insight into our production environment than myself.
    - lusis 15y ago
      Feel free to correct me then. I have YET to work anywhere in any industry (financial primarily) that stated we had to have specific hardware up and running as part of an SLA. It's an "SERVICE LEVEL AGREEMENT" not a "SERVER LEVEL AGREEMENT".
      - jroseattle 15y ago
        > I have YET to work anywhere in any industry Sounds like your experience is the only one that matters, then.
dpcan 15y ago
I deploy and update certain things at night because they REQUIRE down-time. I can't have people making certain updates while something is being moved, for example, so things have to be shut down for a few minutes. That whole line about trusting your infrastructure, etc, is hog-wash. I say, don't trust your infrastructure. Don't trust your code. Be safe, be smart, do things at a time when the least number of people could be affected by a problem. Humans are responsible for everything that is happening in a deployment, and humans make mistakes. So, no, I don't trust my servers or my code at a time of deployment. It would be irresponsible of me to do so even if I had the most talented developers, and the most solid and secure platform in the world.
- jonstjohn 15y ago
  Others would argue that you can architect your infrastructure/application so that you don't require down-time for changes, ever.
casenelson 15y ago
I started writing a reply but it turned into a blog post, here: (http://casestatement.tumblr.com/post/11598425762/why-we-are-still-deploying-overnight http://casestatement.tumblr.com/post/11598425762/why-we-are-...)

113 points by gnubardt 15y ago

The article discusses deploying to a percentage of your user base at a time. I have an idea on how to do this, but it is probably the wrong way. Anybody know of any articles explaining this process for a rails app?

amackera 15y ago
I am also interested in this concept. What if a change requires a migration of my database?
- fleitz 15y ago
  Sometimes it absolutely can't be avoided, but usually the last version and the next version can be coded to work off of either schema. As for deploying to a segment of users put a "version" field in the user and then gate features based on the version. If you name your versions or use a unique numbering system it should be fairly easy to remove the old gating. Yes, none of these things are foolproof but then again your existing system probably isn't either. The more you deploy the better you get at it and the more automated it becomes. Force the issue by forcing a deploy everyday to begin with, then ramp up to twice a day, etc. After a month or two you'll learn solutions to almost all your deployment problems and there will be a well-known solution in the organization for solving the problem whether it be gating features or schema migrations.
  - coreycollins 15y ago
    I cringe at the thought of putting versioning gates all over my code, but maybe that's the easiest solution. What about having two code bases on your production server and having your web server (nginx) route the traffic accordingly? 85% goes to the current code base and 15% goes to the new one? If that's possible it would seem pretty easy.
    - fleitz 15y ago
      I think that makes it less testable but would probably work, it's probably a good stop gap as long as you don't want to test more than one "feature" at a time. Gating would probably work better in the long term as you get more devs and users, and need to do things like A/B/C testing.
- GlennS 15y ago
  A near-instantaneous switchover is possible if you're migrating a database to a new schema, but it's pretty hard work. For this to work, every action that will result in a database write needs to go in log, and that action log needs to be replayable be the updated version. In the context of a web application, you have: 1. Take your database backup, start recording action log. 2. Perform your database migration against your database backup. 3. Install your updated web application (not available to users yet). 4. Test your updated web application. 5. Disable your existing web application. 6. Replay the contents of your action log using the updated web application. 7. Making your updated web application available to your users. It's definitely not a trivial matter to sort out: it means lots of time working on your deployment process, for example you have to be set up to have both versions deployed simultaneously, which likely means a lot of mucking about with URL rewriting. The application also has to be built to support it. Maybe there is a better way?
  - lusis 15y ago
    There is a better way. It's to stop deleting and start deprecating. Write your code in a backwards compatible way. If your storage engine, whatever it is, can't handle it then switch storage engines. The author makes some amazing points that I'm surprised intelligent people are missing. Let me give an example. We are currently migrating from storing blobs of data in Voldemort (a key/value store) to storing them in S3. They should have never been in there in the first place but whatever. We're going to do it with "zero" migration time. In fact we're already doing it. - Set up a job that copies existing data in Voldemort to S3. - Deploy a minor release of our code that multiplexes the current writes to both Voldemort and S3. - Continue migrating existing data - When existing data is finished migrating, deploy a new release that forces all traffic to S3 instead of Voldemort. - Profit People need to learn to do things like dark launching and feature flags. Dark launching let's you exercise new code paths with no impact to the user. Feature flags give you the ability to enable to features to some or all of your users. Feature flags are awesome ways to A/B test as well. People need to stop doing stupid shit like redefining what some property means mid-release and instead define NEW properties and deprecate old ones in subsequent releases. Same goes for schema changes. If some part of your code base cannot tolerate an additional column that it doesn't need, that's a bug. You can also adopt some configuration management that allows you to provision duplicates of the components you're deploying so you can swap back and forth between them in the case where you might have a breaking release. That's what we do and it's one of the upsides of using EC2. All of this requires discipline and dedication but the benefit is so worth it. You have to stay on top of dependencies. You can't let bitrot take hold by going 4 major revisions without upgrading some package. We did and it bit us in the ass. This is why we adopted the swinging strategy of duplicating our entire stack (takes about 30 minutes depending on Amazon allocation latency) on major upgrades.

untog 15y ago

While there is some truth in there, I feel like some of the advice is pretty reckless: "Problem 1: You presume there will be problems that impact availability. You have no confidence in your code quality; or (or maybe, and), you have no confidence in your infrastructure and deployment process." Or you're playing it safe. You absolutely cannot guarantee that every update you are deploying will have zero problems. If your business absolutely relies on users making payments online or anything of that ilk, you could lose a lot of money. "Imagine, for a moment, that your team is rolling out an update to a service that monitors life-support systems in hospitals." What? Why? Throwing out hypothetical "what if" scenarios that would affect 0.1% of your readership isn't a very useful thing to do.

furyg3 15y ago
Throwing out hypothetical "what if" scenarios that would affect 0.1% of your readership isn't a very useful thing to do. Indeed. The people who use my production systems are people who are logging transactions which they have made (in the real world) into my system. It's not a hospital. If the system is down, you just come back later. We do a lot to make sure our deployments will go smoothly, downtime is minimized, and they affect as few users as possible. But the effort required for my team to deliver "five nines" would be insane. It's much easier for one guy to take the application server down for 10 minutes (at midnight) once a month.
- MartinCron 15y ago
  ...the effort required for my team to deliver "five nines" would be insane. It's much easier for one guy to take the application server down for 10 minutes (at midnight) once a month. For the projects I've worked on lately, the ideal of "zero downtime deployments, fully automated, during the daytime, as non-events" isn't at all about getting a particular number of nines, it's about deploying more often than once every month. When the deployment you've been working on for a whole month goes wrong, which of the many hundreds of changes are problematic?
absconditus 15y ago
I work on medical software. Surprisingly, most hospital systems require a fair bit of downtime. Hospitals have downtime procedures that they use during these periods (basically switching to manual, paper-based systems).

RyanMcGreal 15y ago

The best reason not to do overnight deployments is that if something goes wrong, you want your developers to be alert and sharp, not dazed and sleep-deprived.

FollowSteph3 15y ago

Its not just cause ur developers are tired, which is a big issue, its also because you may not have access to the people you need to make decisions in the middle of the night if something needs to be escalated. Good luck trying to get an executive or VP up at 4am to make a decision...

schleyfox 15y ago

There are deploys and then there are deploys. 99% of your deploys should be fine to go live whenever the code hits the repo. I definitely still prefer scheduled overnight deploys whenever significant infrastructure changes need to go live. Downtime should still be minimized, but sometimes a maintenance window is necessary.

xbryanx 15y ago

I'm befuddled by these sorts of posts that are big on theory and light on details. I run tests, via a continuous integration server, and have high confidence in our code when we deploy However, we do updates at night because we often need to take our sites off line for a while, to merge our changes into the live codebase. If we didn't, changes to the database structure would bork the db as visitors clicked around. Nighttime is our lowest volume of visits, so that's the best time to deploy. I'm sure this is a noob reaction, but can anyone point me to some good technical walkthroughs about how to deploy to live without taking your site down or interrupting your database connection? Is there a tool/term/practice I'm missing?

stingraycharles 15y ago
I don't have a good technical walkthrough, but we're providing a high availability service, and zero-downtime upgrades are a requirement. Our setup basically consists of load balancers and application servers. When we need to upgrade, we require application version x always needs to be backwards compatible with application version x - 1, both on a protocol and data model level. Basically, our upgrade procedure is as follows: 1. distribute new package to all servers; 2. run an additional application service on all servers and run some quick tests on each of them to verify proper working; 3. add new application servers to load balancers; 4. upgrade data model to new version (we use postgresql, and this happends in a single db transaction, and remember that our new version x is compatbible with both the current and the previous data model); 5. remove old application services from load balancers; 6. upgrade successful. If anything goes wrong, we can roll back each of these steps. Note that this whole process, perhaps needless to say, is fully automated. By running multiple versions inside the load balancer at the same time, and having the requirement that version x + 1 is always compatible with version x, this procedure allows us to seamlessly upgrade to a new version without any downtime.
mechanical_fish 15y ago
There won't be general-purpose walkthroughs, because the general answer is "don't change the database structure of a database that's serving live traffic: You'll never enumerate all the things that could possibly go wrong." In most specific cases, there are alternatives to stopping the show during an upgrade. First, you check your backups! Then, there are various strategies. You can make many simple schema changes while the system is running. Or put the DB in read-only mode and still mostly manage to serve pages -- perhaps with an occasional HTTP 503, so hopefully that's okay -- during your upgrade. Or phase in changes over several releases, carefully architecting for backward compatibility. Or bring up a parallel system and gradually migrate active users to it. Depending on what you're doing, you may find yourself having to write a special upgrade script that migrates old formats to new formats, or even using database triggers to keep "old" and "new"-format tables in sync during the transition. A well-constrained database may help keep things sensible during the transition -- or, you may have to drop half of your supposedly-sensible constraints just to make the transition work. Moral: Even if you know exactly what you're doing, live updates are more work: More planning, more code, more infrastructure, and/or more stress. In many real-world cases, you should just take the system down for a minute. Focus your engineering effort on making sure that "minute" is as short as possible, and in making sure that you can detect problems and roll back as quickly as possible.

hopeless 15y ago

It's also interesting how U.S.-centric this thinking is: your sneaky 3am deploy in the valley could mean a mid-morning outage in London. Actually, an 8am deployment in Europe has the advantages of a rested brain and a few hours before America hammers the Internet ;-)

virmundi 15y ago
Many US based software companies either A) have a mostly American user base or B) have a regional distribution setup for their app. So if you are a non-American in camp A, that is a problem for you. However, remember the goal is to be the least disruptive. If you are a non-American in a region that the server is deployed in, there is probably a night-based take down of the system in your region. If the software company is either A nor B, then the software company need to notify the non-user users that they may experience a disruption.
mseebach 15y ago
> Actually, an 8am deployment in Europe has the advantages of a rested brain and a few hours before America hammers the Internet ;-) And the disadvantage of happening in the middle of mainland Europe morning rush. I think the point is that you take the downtime when it's least disruptive to your users, who/wherever they might be.

Vitaly 15y ago

Or may be a client simply insists that customers absolutely can't tolerate a several seconds delays during the deployment ;(

deleted 15y ago

[deleted]

chris_dcosta 15y ago

Couldn't agree more. I worked somewhere that only deployed during business hours because it was all hands on deck. We just kept the users primed with information and reminders prior to go-live, and when it happened everyone knew about it. But there's a more fundamental point. We're dealing with IT, and for internet businesses and global enterprises the issue is the same: business is 24 hours a day 7 days a week. You cannot avoid deploying somewhere in the world that's awake. All you have to do is plan a rollback and just get on with it. My experience of companies who are timid about rolling out, changes is that they're state of advancement is completely unstable. They are too scared to change meaning that some technologies become out-of-sync with the current state of play. You only have to introduce a new piece of wiz-bang software and issues come pouring out, and it's a frantic race to upgrade random components in the hope you might plug the leak. This is far far worse in every way than doing regular "risky" upgrades. Small incremental changes can be easily rolled back, because you know what you just changed.

rilindo 15y ago

"Why are you still deploying overnight?" Because you don't hire a system admin to do that job for you. :}

amalcon 15y ago

The only good point is #2: you want your people to be awake during the deployment so they can deal with any problems. The others basically amount to "You should have other processes in-place to avoid the same problems you're trying to minimize." Three words: Defense in depth. You don't always need every advantage you can get, and this is a pretty costly one. Still, it's simply wrong to assume that a particular precaution is always needless. Interestingly, you hit a bit of a sweet-spot if your primary customer base is located halfway around the world. You can roll out updates when your people are most wide awake, but when you'll only break things for a few users if something goes wrong.

Roboprog 15y ago

Why in the middle of the night? The economics of things, mostly. At my previous job, we were still in process to get session migration to work so servers within a cluster could update while others were running. At my current job, we don't have a cluster (we have a single over-provisioned server), and that method simply isn't in the budget for the foreseeable future. However, 3:00 AM PDT is 3:30 PM IST (India), which is where most of my team is, as well as the support staff they need (most staff there are on swing shift, as well). Thus, an early morning release is actually a very convenient, as well as lower cost, time to do the release. As the site is mostly US customers in both a B2C and B2B use case, this timing (around 6:00 AM EDT) works very well, and I don't see it changing. I realize this doesn't scale for a global app, but it explains the lack of pressure to spend much consideration on higher tech, more expensive, deployment methodologies.

unreal37 15y ago

I was hoping the author would have some deeply insightful answer to the argument, "because the deployment requires the site to go down for a few minutes and we want to avoid inconveniencing customers" but he avoids that topic entirely. That's the number 1 reason for an overnight deployment, by far! If you can deploy code without bringing the site down, then of course, do it during the day!

maratd 15y ago
Bingo. Deployment with us is automated. I don't actually stay up to 3 AM. However, during the deployment the machine is automatically updated with new packages and the new version of the software, caches are cleared, it's restarted, etc. All of this probably takes 5 minutes at most, but still ... why do that in the middle of the day? Also, we do a daily backup around midnight. If the deployment is botched, I can come in early in the morning, notice it and simply do a roll-back using the backup. Very easy. Now, if you're actually staying up until 3 AM and doing things manually ... you need to automate things.
- jonstjohn 15y ago
  Depends on the type of site/application. If your application is managing customer data that can be updated 24 hours a day, a few minutes of down-time might be okay, whereas rolling back 6 hours of data (and losing it) is definitely not okay.
miker64 15y ago
He did, you do a rolling update, take a few systems out of active duty, upgrade them, and then begin directing some percent of traffic to them. That, of course, only works if you are of a scale to have that many production systems, and loose enough interdependencies to allow for new and old code to run at the same time. I think some of his point was that if you aren't able to do that sort of update, it was worth your time and effort to get there. I think he's probably right, but the work to get from where you are (it requires site outage to upgrade) to where you want to be (rolling upgrades, ability to run multiple versions in parallel, etc) is going to be site specific in most cases that it's hard to discuss that move in any depth.

kstenerud 15y ago

This is incredibly reckless and naive advice. You roll out when it least impacts customers because: 1. Impacted customers affects your bottom line. Minimize that and you minimize losses if something screws up. 2. Murphy's law. I don't care how much testing and QA you do; stuff will ALWAYS creep through, and sometimes it'll be nasty. QA works to minimize this, but it can't eliminate it entirely due to diminishing returns. Show me your guaranteed bug free deployed code and then I'll consider changing my view. 3. If you've designed and tested your rollback procedure prior to actually doing it, the chances of not being able to roll back in the real deployment is orders of magnitude lower than the chances of a failed deployment requiring a rollback, which in turn, if you've done your testing and QA, is orders of magnitude less likely than a successful rollout (but not 0, thus the midnight rollout). If you're worth your salt, you have a tested rollback procedure, laid out in simple to follow instructions (or better yet, an automated rollback mechanism with a simple-to-follow manual process when the automated method inevitably fucks up). You rollout, and if it fails, you roll back. And if that fails, you use the manual procedure. You should have the entire process time boxed to the worst case scenario (assuming successful manual rollback) so that you know beforehand what the impact is, and won't need to go around waking people up asking what to do.

MartinCron 15y ago
A part of this is looking at the risk assessment and the notion of "guaranteed bug free deployed code" in a different way. For a stable system that's in production, you don't need "guaranteed bug free deployed code" you need code that is not any worse than what's currently running out there. Doing frequent (daytime) deployments makes it easier to make a change, test (both with humans and robots) that change, and get it out there. You don't have to manually test everything in order to change anything when you're changing just one thing at a time. I've come to believe that the far riskier approach is to make a bunch of changes at once, introduce a bunch of bugs, test fix bugs until you feel confident, and then release this huge change all at once in the middle of the night.
lusis 15y ago
Rollbacks are a myth. You can never rollback. Always be rolling forward. Enabling a culture and environment that allows for small frequent changes solves that problem. The way to not impact a customer is to make deploys trivial, automated and tolerant to failure because everything fails.

krobertson 15y ago

I think a point also missed in the article is the startup notion of burning the midnight oil and pushing stuff live. "Release early, release often" is great, but that same call to action can be what motivates people to push code live at 3am when they're tired, just finished it, and perhaps skipping QA. Otherwise, I think the article as put is empty and seems more straw man. Assumes you have no confidence in your code, assumes it is complicated, assumes you can't roll back. Simply put, deploy small stuff often, make plans for big stuff, and don't be stupid.

whazzmaster 15y ago

I attended a git workshop by Scott Chacon and was surprised to learn that github's production site can be deployed by many folks on the development team, and at any time during the day. In fact, he mentioned that github.com can be updated as many as a dozen or more times per day as small features or fixes are pushed live. There is a great description of their web server stack[1] that allows them to seamlessly push new code live without downtime (see the second Slow Deploys section.) It's a great idea about how to continue serving during a deployment, and I'm looking into using Unicorn like this for some projects I'm working on. [1] https://github.com/blog/517-unicorn https://github.com/blog/517-unicorn

Hominem 15y ago

I rarely have issues but is always a chance that something is going to go wrong, so everyone is sitting there waiting for the all clear. I was able to make the case that having developers, DBAs, and sysadmins sitting around in the middle of the night was counterproductive. It was fraught with tension during the middle of the night after a long day at work. Now we do it at 1pm on saturday, it is amlost leisurely now. Just a quick call to check in. Some people even call the voice bridge from the beach!

ekidd 15y ago

Sometimes, it's cheaper to live with 10 minutes of scheduled maintenance at a low-traffic hour than to re-engineer your application to have 99.999% uptime. Even if you have unit tests, continuous integration, a staging environment, and fast rollbacks, deploying to production can still be tricky: 1) Do you want to run MySQL, PostgreSQL, Oracle or SQL Server? Watch out for ALTER TABLE; it may sometimes hold table locks or invalidate open cursors. Of course, you could always rewrite your application to use a schema-free NoSQL database and migrate your records at read time. But that has a lot of consequences. Maybe 10 minutes of scheduled maintenance is an acceptable tradeoff? 2) Does your staging environment exactly match your production environment? Can you generate a full-sized, real-world load on your staging servers? Alternatively, have you built all the tools required for a phased deployment of your back end? Now, I'm all in favor of deploying to production 5 times a day. That's pretty easy for any group with good unit test coverage and a fast rollback mechanism. But it's expensive to eliminate the occasional 10 minute maintenance window. And if you're going to take the site down for maintenance, it makes sense to do it at an off-peak hour.

InclinedPlane 15y ago
I don't think the author's point is that everyone should be doing middle-of-the day seamless deployments from day one. I think the point is that if you continue to do deployments at 3am indefinitely you are hurting yourself. There are lots of things that make sense at the appropriate scale. Not having any tests. Completely manual processes for builds and deployments. Having only one server. Making backups every day or maybe even just every week. Etc. However, as your business grows you need to recognize that it's important to change these things and move to better processes. Because if you don't then those things could become a very serious drag on development velocity and even business capability. You could find yourself spending all your time drowning in process when you should be spending your efforts more efficiently. You stay up until 3am to perform a risky manual deployment, sometimes you spend some time fixing problems afterward. As a consequence you either spend a day in zombie mode with limited productivity or you start working much later. In either case you've taken away a fairly decent chunk of time that could have been used for productive work. Similarly, perhaps you spend too much of every day fighting your build system, or recovering from your build always being on the floor, or excessive operational support because your platform doesn't have sufficient redundancy at every tier. Etc, etc, etc.

jakejake 15y ago

I do think 3AM is a terrible time to do a release. I'm personally a fan of doing releases in the early morning instead of late night. I'd rather get in the office at 6AM & push the release. Plan for a light day and let everybody leave early if things went smoothly. If things do go wrong it gives you about 2-3 hours before the rest of the office start showing up to roll back or fix the situation. Choosing the time should be a balance of off-peak usage time and a reasonable time for the developers. If you're a global business then you may not have a perfect time, but you can usually still choose a day and time that is less likely to cause inconvenience.

nimblegorilla 15y ago

Let's be realistic. Many applications have 24 hour availability, but almost every application I've worked on was for people in the US and core usage was sometime between 7AM and 9PM. If it only takes 5 minutes to deploy your app why not do it at 2AM when few users are online? Deployments should be simple, easy, and NOT require any problem solving by the deployer. Midday deployments with all hands on deck to handle issues seems like a bigger problem to me.

127001brewer 15y ago

"...I sincerely hope you’re not living in the Stone Age..." Whether or not you agree with the entire article, for me, the statement above was the most important one. There is no excuse why any software development team does not have an automated build and deploy process. There may be valid business reasons to deploy in the middle of the night, but there's no business reason to deploy by hand. Also, I am assuming that the posting only reflects application source code and not database changes, which is a different beast in terms of deployments.

jroseattle 15y ago

Because our environment is non-trivial. It has nothing to do with confidence in application code, and everything to do with service-level agreements, impact to existing customers, and our own psyche. Ask SalesForce if they do their maintenance updates on weekends because of a lack of confidence in their codebase. We're not doing 2am deployments, but we're certainly doing them "off-hours".

lusis 15y ago
Triviality has nothing to do with it. Nothing in your SLA says you have to have servers 1, 2 and 3 available. It says you have to have service X available. You can do that with out taking a downtime hit.
- jroseattle 15y ago
  Wow, thanks for having better insight into our production environment than myself.
  - lusis 15y ago
    Feel free to correct me then. I have YET to work anywhere in any industry (financial primarily) that stated we had to have specific hardware up and running as part of an SLA. It's an "SERVICE LEVEL AGREEMENT" not a "SERVER LEVEL AGREEMENT".
    - jroseattle 15y ago
      > I have YET to work anywhere in any industry Sounds like your experience is the only one that matters, then.

dpcan 15y ago

I deploy and update certain things at night because they REQUIRE down-time. I can't have people making certain updates while something is being moved, for example, so things have to be shut down for a few minutes. That whole line about trusting your infrastructure, etc, is hog-wash. I say, don't trust your infrastructure. Don't trust your code. Be safe, be smart, do things at a time when the least number of people could be affected by a problem. Humans are responsible for everything that is happening in a deployment, and humans make mistakes. So, no, I don't trust my servers or my code at a time of deployment. It would be irresponsible of me to do so even if I had the most talented developers, and the most solid and secure platform in the world.

jonstjohn 15y ago
Others would argue that you can architect your infrastructure/application so that you don't require down-time for changes, ever.

casenelson 15y ago

I started writing a reply but it turned into a blog post, here: (http://casestatement.tumblr.com/post/11598425762/why-we-are-still-deploying-overnight http://casestatement.tumblr.com/post/11598425762/why-we-are-...)