3 ms·

middle-of-the day seamless deployments from day one From my experience, it's easier to start there from day one than to try to get there after years of dysfunc

middle-of-the day seamless deployments from day one From my experience, it's easier to start there from day one than to try to get there after years of dysfunctional deployment processes.

InclinedPlane 15y ago
Some things are generally better and easier to do a bit before you really "need" them. Deployment, build, branching, and backup procedures almost certainly fall into that category. Performance as well, as long as it's done smartly. Interestingly, scaling is probably the least valuable thing to "overbuild" early and yet that's by far the most common thing teams tend to do.
- MartinCron 15y ago
  I would just add "test automation" and "restore from backups" to that list.
  - InclinedPlane 15y ago
    Indeed! Also, pretty much everything from the "Joel test" goes without saying.
wisty 15y ago
Is there any way to do fast database schema migrations? This is not a nitpick, I've posted a question on SO about this (http://stackoverflow.com/questions/6740856/how-do-big-companies-like-say-facebook-do-migrations-without-having-downtime http://stackoverflow.com/questions/6740856/how-do-big-compan...). As is suggest above, you could use NOSQL. But NOSQL still has schemas, in a sense - if you decide to restructure anything you may still need to knock the db out for a while (or write really hairy code to work around two different versions of your data, but be unable to restructure stuff because you're too scared to change anything on the db).
- arohner 15y ago
  I'm being serious, but also snarky: Don't do schema migrations. I saw a talk by Brett Durrett, VP Dev & Ops at IMVU last week. They don't make backwards incompatible schema changes. Ever. Also, on big tables, they don't use alter statements ever, because MySQL sometimes takes an hour to ALTER. Instead, they create a new table that is the "updated" version of the old table, and then their code does copy-on-read. i.e. they look for the row in the new table, if it's not there, copy it out of the old table and insert into the new table. Later, a background job will come through and migrate all old records into the new table. Eventually, the old table will be deleted and the copy-on-read code will be removed. It's a lot of extra work, but they think it's worth the effort. I need to finish my blog post on the rest of the talk...
  - jonstjohn 15y ago
    Would love to see that blog post since a lot of people struggle with schema changes, particularly 'alter table'.
  - InclinedPlane 15y ago
    This x infinity. Any time you have a thought to do something that will lock an entire table that is active in production you should think about another way to do it. The read-through "cache" with rolling back-compatability method is a great way to make such breaking changes without causing significant downtime. No matter what you end up doing it'll be extra work. It's best to find a way of working that provides enough flexibility to continue developing at a good pace without being an operational nightmare. Generally that means you'll have to take things a bit slower, but that'd be true almost regardless of the technology you're using, schema changes rarely have a trivial impact.
- jules 15y ago
  Here is a paper on that topic: http://pmg.csail.mit.edu/~ajmani/papers/ecoop06-upgrades.pdf http://pmg.csail.mit.edu/~ajmani/papers/ecoop06-upgrades.pdf Their examples show that it is not possible to do all application upgrades without downtime, and they show how to keep downtime to a minimum (e.g. by only making parts of your application inaccessible).

middle-of-the day seamless deployments from day one From my experience, it's easier to start there from day one than to try to get there after years of dysfunctional deployment processes.

InclinedPlane 15y ago

Some things are generally better and easier to do a bit before you really "need" them. Deployment, build, branching, and backup procedures almost certainly fall into that category. Performance as well, as long as it's done smartly. Interestingly, scaling is probably the least valuable thing to "overbuild" early and yet that's by far the most common thing teams tend to do.

MartinCron 15y ago
I would just add "test automation" and "restore from backups" to that list.
- InclinedPlane 15y ago
  Indeed! Also, pretty much everything from the "Joel test" goes without saying.

wisty 15y ago

Is there any way to do fast database schema migrations? This is not a nitpick, I've posted a question on SO about this (http://stackoverflow.com/questions/6740856/how-do-big-companies-like-say-facebook-do-migrations-without-having-downtime http://stackoverflow.com/questions/6740856/how-do-big-compan...). As is suggest above, you could use NOSQL. But NOSQL still has schemas, in a sense - if you decide to restructure anything you may still need to knock the db out for a while (or write really hairy code to work around two different versions of your data, but be unable to restructure stuff because you're too scared to change anything on the db).

arohner 15y ago
I'm being serious, but also snarky: Don't do schema migrations. I saw a talk by Brett Durrett, VP Dev & Ops at IMVU last week. They don't make backwards incompatible schema changes. Ever. Also, on big tables, they don't use alter statements ever, because MySQL sometimes takes an hour to ALTER. Instead, they create a new table that is the "updated" version of the old table, and then their code does copy-on-read. i.e. they look for the row in the new table, if it's not there, copy it out of the old table and insert into the new table. Later, a background job will come through and migrate all old records into the new table. Eventually, the old table will be deleted and the copy-on-read code will be removed. It's a lot of extra work, but they think it's worth the effort. I need to finish my blog post on the rest of the talk...
- jonstjohn 15y ago
  Would love to see that blog post since a lot of people struggle with schema changes, particularly 'alter table'.
- InclinedPlane 15y ago
  This x infinity. Any time you have a thought to do something that will lock an entire table that is active in production you should think about another way to do it. The read-through "cache" with rolling back-compatability method is a great way to make such breaking changes without causing significant downtime. No matter what you end up doing it'll be extra work. It's best to find a way of working that provides enough flexibility to continue developing at a good pace without being an operational nightmare. Generally that means you'll have to take things a bit slower, but that'd be true almost regardless of the technology you're using, schema changes rarely have a trivial impact.
jules 15y ago
Here is a paper on that topic: http://pmg.csail.mit.edu/~ajmani/papers/ecoop06-upgrades.pdf http://pmg.csail.mit.edu/~ajmani/papers/ecoop06-upgrades.pdf Their examples show that it is not possible to do all application upgrades without downtime, and they show how to keep downtime to a minimum (e.g. by only making parts of your application inaccessible).