7 ms·

Sometimes, it's cheaper to live with 10 minutes of scheduled maintenance at a low-traffic hour than to re-engineer your application to have 99.999% uptime. Eve

by ekidd 15y ago

Sometimes, it's cheaper to live with 10 minutes of scheduled maintenance at a low-traffic hour than to re-engineer your application to have 99.999% uptime. Even if you have unit tests, continuous integration, a staging environment, and fast rollbacks, deploying to production can still be tricky: 1) Do you want to run MySQL, PostgreSQL, Oracle or SQL Server? Watch out for ALTER TABLE; it may sometimes hold table locks or invalidate open cursors. Of course, you could always rewrite your application to use a schema-free NoSQL database and migrate your records at read time. But that has a lot of consequences. Maybe 10 minutes of scheduled maintenance is an acceptable tradeoff? 2) Does your staging environment exactly match your production environment? Can you generate a full-sized, real-world load on your staging servers? Alternatively, have you built all the tools required for a phased deployment of your back end? Now, I'm all in favor of deploying to production 5 times a day. That's pretty easy for any group with good unit test coverage and a fast rollback mechanism. But it's expensive to eliminate the occasional 10 minute maintenance window. And if you're going to take the site down for maintenance, it makes sense to do it at an off-peak hour.

InclinedPlane 15y ago
I don't think the author's point is that everyone should be doing middle-of-the day seamless deployments from day one. I think the point is that if you continue to do deployments at 3am indefinitely you are hurting yourself. There are lots of things that make sense at the appropriate scale. Not having any tests. Completely manual processes for builds and deployments. Having only one server. Making backups every day or maybe even just every week. Etc. However, as your business grows you need to recognize that it's important to change these things and move to better processes. Because if you don't then those things could become a very serious drag on development velocity and even business capability. You could find yourself spending all your time drowning in process when you should be spending your efforts more efficiently. You stay up until 3am to perform a risky manual deployment, sometimes you spend some time fixing problems afterward. As a consequence you either spend a day in zombie mode with limited productivity or you start working much later. In either case you've taken away a fairly decent chunk of time that could have been used for productive work. Similarly, perhaps you spend too much of every day fighting your build system, or recovering from your build always being on the floor, or excessive operational support because your platform doesn't have sufficient redundancy at every tier. Etc, etc, etc.
- MartinCron 15y ago
  middle-of-the day seamless deployments from day one From my experience, it's easier to start there from day one than to try to get there after years of dysfunctional deployment processes.
  - InclinedPlane 15y ago
    Some things are generally better and easier to do a bit before you really "need" them. Deployment, build, branching, and backup procedures almost certainly fall into that category. Performance as well, as long as it's done smartly. Interestingly, scaling is probably the least valuable thing to "overbuild" early and yet that's by far the most common thing teams tend to do.
    - MartinCron 15y ago
      I would just add "test automation" and "restore from backups" to that list.
      - InclinedPlane 15y ago
        Indeed! Also, pretty much everything from the "Joel test" goes without saying.
  - wisty 15y ago
    Is there any way to do fast database schema migrations? This is not a nitpick, I've posted a question on SO about this (http://stackoverflow.com/questions/6740856/how-do-big-companies-like-say-facebook-do-migrations-without-having-downtime http://stackoverflow.com/questions/6740856/how-do-big-compan...). As is suggest above, you could use NOSQL. But NOSQL still has schemas, in a sense - if you decide to restructure anything you may still need to knock the db out for a while (or write really hairy code to work around two different versions of your data, but be unable to restructure stuff because you're too scared to change anything on the db).
    - 15y ago

by ekidd 15y ago

InclinedPlane 15y ago

I don't think the author's point is that everyone should be doing middle-of-the day seamless deployments from day one. I think the point is that if you continue to do deployments at 3am indefinitely you are hurting yourself. There are lots of things that make sense at the appropriate scale. Not having any tests. Completely manual processes for builds and deployments. Having only one server. Making backups every day or maybe even just every week. Etc. However, as your business grows you need to recognize that it's important to change these things and move to better processes. Because if you don't then those things could become a very serious drag on development velocity and even business capability. You could find yourself spending all your time drowning in process when you should be spending your efforts more efficiently. You stay up until 3am to perform a risky manual deployment, sometimes you spend some time fixing problems afterward. As a consequence you either spend a day in zombie mode with limited productivity or you start working much later. In either case you've taken away a fairly decent chunk of time that could have been used for productive work. Similarly, perhaps you spend too much of every day fighting your build system, or recovering from your build always being on the floor, or excessive operational support because your platform doesn't have sufficient redundancy at every tier. Etc, etc, etc.

MartinCron 15y ago
middle-of-the day seamless deployments from day one From my experience, it's easier to start there from day one than to try to get there after years of dysfunctional deployment processes.
- InclinedPlane 15y ago
  Some things are generally better and easier to do a bit before you really "need" them. Deployment, build, branching, and backup procedures almost certainly fall into that category. Performance as well, as long as it's done smartly. Interestingly, scaling is probably the least valuable thing to "overbuild" early and yet that's by far the most common thing teams tend to do.
  - MartinCron 15y ago
    I would just add "test automation" and "restore from backups" to that list.
    - InclinedPlane 15y ago
      Indeed! Also, pretty much everything from the "Joel test" goes without saying.
- wisty 15y ago
  Is there any way to do fast database schema migrations? This is not a nitpick, I've posted a question on SO about this (http://stackoverflow.com/questions/6740856/how-do-big-companies-like-say-facebook-do-migrations-without-having-downtime http://stackoverflow.com/questions/6740856/how-do-big-compan...). As is suggest above, you could use NOSQL. But NOSQL still has schemas, in a sense - if you decide to restructure anything you may still need to knock the db out for a while (or write really hairy code to work around two different versions of your data, but be unable to restructure stuff because you're too scared to change anything on the db).
  - 15y ago