4 ms·
I don't think the author's point is that everyone should be doing middle-of-the day seamless deployments from day one. I think the point is that if you continue
by InclinedPlane 15y ago
I don't think the author's point is that everyone should be doing middle-of-the day seamless deployments from day one. I think the point is that if you continue to do deployments at 3am indefinitely you are hurting yourself.
There are lots of things that make sense at the appropriate scale. Not having any tests. Completely manual processes for builds and deployments. Having only one server. Making backups every day or maybe even just every week. Etc. However, as your business grows you need to recognize that it's important to change these things and move to better processes. Because if you don't then those things could become a very serious drag on development velocity and even business capability. You could find yourself spending all your time drowning in process when you should be spending your efforts more efficiently.
You stay up until 3am to perform a risky manual deployment, sometimes you spend some time fixing problems afterward. As a consequence you either spend a day in zombie mode with limited productivity or you start working much later. In either case you've taken away a fairly decent chunk of time that could have been used for productive work. Similarly, perhaps you spend too much of every day fighting your build system, or recovering from your build always being on the floor, or excessive operational support because your platform doesn't have sufficient redundancy at every tier. Etc, etc, etc.
- MartinCron 15y agomiddle-of-the day seamless deployments from day one From my experience, it's easier to start there from day one than to try to get there after years of dysfunctional deployment processes.
- InclinedPlane 15y agoSome things are generally better and easier to do a bit before you really "need" them. Deployment, build, branching, and backup procedures almost certainly fall into that category. Performance as well, as long as it's done smartly. Interestingly, scaling is probably the least valuable thing to "overbuild" early and yet that's by far the most common thing teams tend to do.
- MartinCron 15y agoI would just add "test automation" and "restore from backups" to that list.
- InclinedPlane 15y agoIndeed! Also, pretty much everything from the "Joel test" goes without saying.
- wisty 15y agoIs there any way to do fast database schema migrations? This is not a nitpick, I've posted a question on SO about this (http://stackoverflow.com/questions/6740856/how-do-big-companies-like-say-facebook-do-migrations-without-having-downtime http://stackoverflow.com/questions/6740856/how-do-big-compan...). As is suggest above, you could use NOSQL. But NOSQL still has schemas, in a sense - if you decide to restructure anything you may still need to knock the db out for a while (or write really hairy code to work around two different versions of your data, but be unable to restructure stuff because you're too scared to change anything on the db).
- arohner 15y agoI'm being serious, but also snarky: Don't do schema migrations. I saw a talk by Brett Durrett, VP Dev & Ops at IMVU last week. They don't make backwards incompatible schema changes. Ever. Also, on big tables, they don't use alter statements ever, because MySQL sometimes takes an hour to ALTER. Instead, they create a new table that is the "updated" version of the old table, and then their code does copy-on-read. i.e. they look for the row in the new table, if it's not there, copy it out of the old table and insert into the new table. Later, a background job will come through and migrate all old records into the new table. Eventually, the old table will be deleted and the copy-on-read code will be removed. It's a lot of extra work, but they think it's worth the effort. I need to finish my blog post on the rest of the talk...
- jonstjohn 15y agoWould love to see that blog post since a lot of people struggle with schema changes, particularly 'alter table'.
- InclinedPlane 15y agoThis x infinity. Any time you have a thought to do something that will lock an entire table that is active in production you should think about another way to do it. The read-through "cache" with rolling back-compatability method is a great way to make such breaking changes without causing significant downtime. No matter what you end up doing it'll be extra work. It's best to find a way of working that provides enough flexibility to continue developing at a good pace without being an operational nightmare. Generally that means you'll have to take things a bit slower, but that'd be true almost regardless of the technology you're using, schema changes rarely have a trivial impact.
- barrkel 15y agoThere's an easy way to do 3am US deployments: do them from Europe. I used to work at a consultancy in Ireland that did this, and it worked quite well. Deploy first thing in the morning, and you have up until some time in the afternoon to fix things if it goes pear-shaped.
- saurik 15y ago"You stay up until 3am... spend a day in zombie mode with limited productivity or you start working much later." Another option is to just wake up much later: my office is dead until 2pm, and the "key players" often don't wake up until 6pm.
- robryan 15y agoI guess it makes some sense if you are trying to deploy in the late hours but it doesn't sound sustainable keeping a whole team so far out of sync with everyone else. I'd imagine those with families or do things like sports of a night would eventually quit.