4 ms·

I really enjoyed Thomas Limoncelli's Time Management for System Administrators. I've also got The Practice of System and Network Administration on my shelf, and

I really enjoyed Thomas Limoncelli's Time Management for System Administrators. I've also got The Practice of System and Network Administration on my shelf, and hopefully one of these years I'll read it---maybe once my time management is better. :-) I like that Tom was practicing and advocating system-administration-by-writing-code way before DevOps, and to me he represents a tradition of automation that far predates the buzzwords. I've lived in the world of Rails and Django so long, I sort of take schema migrations for granted. It's interesting to see such a high-level person covering this ground; it reminds me of how much these frameworks have given us, and how so many organizations are still practically emailing around zip files of source code. His Technique 2 feels harder to me: write the code to work on both old & new schemas, and deploy the schema change after the code change. I know this is what Google does though, and I think it's the only safe way to deploy rollback-able code changes. I'm curious who else is doing it, and especially how you are scheduling it. Especially with a CI/CD workflow, I'm used to getting things integrated and letting them get deployed "whenever". So how do you make sure the schema changes go out "later", and what gating is required first, and how do you get buy-in to spend time on that after the feature is already live? I'm not asking about the technical how-to, but the organizational/political how-to. For instance if you're using Pivotal Tracker, do you make a separate PT story? Leave the old one open until the schema changes are deployed? What does it concretely look like?

djrobstep 8y ago
> I've lived in the world of Rails and Django so long, I sort of take schema migrations for granted. And there's lots more room for improvement over Rails and Django migrations. Some of the problems: - Annoying version number management - Management of long "chain" of migration files - Unnecessary coupling between ORM and migration tool - Heavyweight process to generate and run each migration, which slows down local development - Bad testability Autosyncing local dev databases, and automatically generating migration scripts with a diff tool is a better approach, imho.
- evanelias 8y ago
  Totally agree -- I wrote some similar thoughts elsewhere in the thread. It might just take some time for more developers to gain experience with the declarative/diff-based approach, to become comfortable with it.
- all_factz 8y ago
  Don't disagree with the rest of these points, but I do want to point out that Django lets you squash long "chain" migration files just as you'd do with commits.
GauntletWizard 8y ago
Think of Schema changes much like a two-phase commit in a database (Which has more than two phases - I hate that term.) First, propose the new schema. You should check in a version of your schema that contains both the fields from your new and old code. You can't do certain changes in this phase - If you're deleting fields, you can't do that yet. If you're changing column types, you should be careful; Hopefully your DAO already abstracts this, but you may need to roll that code change out first. Then, make your application use the new schema. Start writing to the new columns as well as the old. Prepare a migration that will copy any data from old columns that needs to be. Your application should use the old data as the source-of-truth during this time, but when writing, it needs to write both versions of those fields. At this point, you can still easily roll back, and may want to, if you decide to proceed a different way for development. Next, flip the source of truth. Your new fields should be the primary ones, but the old ones still need to be valid. You should be prepared to roll back still - The previous release should still be usable. Once you're certain you're not going to roll back, stop reading/writing the old data. This is your hard flip, but once it's started, you can proceed leisurely. It doesn't matter - You've entered cleanup phase. Once that software change is finished, you can do a second schema change to delete old fields. (This is where the significant value of Schemaless is - You don't need to do anything in this second change, just let old data naturally clear itself out. Schemaless actually requires more attention to schema than a Schema-driven database, for this reason, but the iteration time can be quite fast if you have a good schema-control system, like, say, storing protobuf definitions in your version control and following their update rules[1]) All in all, this requires at least two periods of schema change and one period that involves multiple software rollouts. It's not simple, but nor is it that complex - And if done right, it's easy to roll other software changes at the same time, never losing velocity (aside from that which is spent reaching this target) All of this requires that your application is well formed. Your application needs to be properly modularized, use reasonable DAOs or otherwise have complete understanding of how the data is accessed. It may require coordination between two projects that move, while not in lockstep, with gates on the other. [1]https://developers.google.com/protocol-buffers/docs/proto#updating https://developers.google.com/protocol-buffers/docs/proto#up...
- desc 8y ago
  Indeed. It's doable, but requires a reasonably fast rollout tempo and a manageable level of technical debt. It is utterly impractical when you're deploying to individual customer sites, for customers who agree to an upgrade on timescales ranging from three months to five years... [edit] Part 1, the automated schema updates, I'm right alongside. I can't fathom how people can operate with manual schema updates all the time. But part 2 is IMO wishful thinking outside of a deployed environment your own team can upgrade pretty much at will.
navinsylvester 8y ago
Like the feature flag approach for all feature rollout(with or without schema change). If the flag is set the new feature kicks in so its easy to do a rollback if need be and we do this all the time. We don't close the feature ticket until the flag(old schema too) is removed and made as default in production after the smooth transition.

djrobstep 8y ago

> I've lived in the world of Rails and Django so long, I sort of take schema migrations for granted. And there's lots more room for improvement over Rails and Django migrations. Some of the problems: - Annoying version number management - Management of long "chain" of migration files - Unnecessary coupling between ORM and migration tool - Heavyweight process to generate and run each migration, which slows down local development - Bad testability Autosyncing local dev databases, and automatically generating migration scripts with a diff tool is a better approach, imho.

evanelias 8y ago
Totally agree -- I wrote some similar thoughts elsewhere in the thread. It might just take some time for more developers to gain experience with the declarative/diff-based approach, to become comfortable with it.
all_factz 8y ago
Don't disagree with the rest of these points, but I do want to point out that Django lets you squash long "chain" migration files just as you'd do with commits.

GauntletWizard 8y ago

Think of Schema changes much like a two-phase commit in a database (Which has more than two phases - I hate that term.) First, propose the new schema. You should check in a version of your schema that contains both the fields from your new and old code. You can't do certain changes in this phase - If you're deleting fields, you can't do that yet. If you're changing column types, you should be careful; Hopefully your DAO already abstracts this, but you may need to roll that code change out first. Then, make your application use the new schema. Start writing to the new columns as well as the old. Prepare a migration that will copy any data from old columns that needs to be. Your application should use the old data as the source-of-truth during this time, but when writing, it needs to write both versions of those fields. At this point, you can still easily roll back, and may want to, if you decide to proceed a different way for development. Next, flip the source of truth. Your new fields should be the primary ones, but the old ones still need to be valid. You should be prepared to roll back still - The previous release should still be usable. Once you're certain you're not going to roll back, stop reading/writing the old data. This is your hard flip, but once it's started, you can proceed leisurely. It doesn't matter - You've entered cleanup phase. Once that software change is finished, you can do a second schema change to delete old fields. (This is where the significant value of Schemaless is - You don't need to do anything in this second change, just let old data naturally clear itself out. Schemaless actually requires more attention to schema than a Schema-driven database, for this reason, but the iteration time can be quite fast if you have a good schema-control system, like, say, storing protobuf definitions in your version control and following their update rules[1]) All in all, this requires at least two periods of schema change and one period that involves multiple software rollouts. It's not simple, but nor is it that complex - And if done right, it's easy to roll other software changes at the same time, never losing velocity (aside from that which is spent reaching this target) All of this requires that your application is well formed. Your application needs to be properly modularized, use reasonable DAOs or otherwise have complete understanding of how the data is accessed. It may require coordination between two projects that move, while not in lockstep, with gates on the other. [1]https://developers.google.com/protocol-buffers/docs/proto#updating https://developers.google.com/protocol-buffers/docs/proto#up...

desc 8y ago
Indeed. It's doable, but requires a reasonably fast rollout tempo and a manageable level of technical debt. It is utterly impractical when you're deploying to individual customer sites, for customers who agree to an upgrade on timescales ranging from three months to five years... [edit] Part 1, the automated schema updates, I'm right alongside. I can't fathom how people can operate with manual schema updates all the time. But part 2 is IMO wishful thinking outside of a deployed environment your own team can upgrade pretty much at will.

navinsylvester 8y ago

Like the feature flag approach for all feature rollout(with or without schema change). If the flag is set the new feature kicks in so its easy to do a rollback if need be and we do this all the time. We don't close the feature ticket until the flag(old schema too) is removed and made as default in production after the smooth transition.