4 ms·

Instead of an error prone human checklist, has anyone made frameworks with these sequenced structural migrations in mind? I’ve worked with both Rails migrations

Instead of an error prone human checklist, has anyone made frameworks with these sequenced structural migrations in mind? I’ve worked with both Rails migrations and Alembic, but they seem a layer of abstraction too low to only allow safe migrations.

YesThatTom2 5y ago
This is what the humans need to do. It works with any framework or database. You might find some software frameworks that make this easier than others but expecting to find a technology that replaces human methodology is like reading a book on Scrum and saying “can’t we make a software system that does this for us?”
- zeckalpha 5y ago
  No. Then we shouldn’t be using programming languages. I have learned to trust humans to not get migrations right. The system can have guardrails for certain structures of migrations.
0xbadcafebee 5y ago
Not a framework, but schemaless key-value stores allow you to play fast and loose and avoid a lot of the reason for migrations, with obvious downsides. Monorepos also allow you to change and deploy all of the code at once without having to coordinate versions of both dependencies and deployments.
- dgb23 5y ago
  Also using a key value store or some other no-sql non-graph storage can sometimes be put between two structured migration endpoints. The advantage is that you have a clear way to first pull things apart before you put them together again and an intermediate step that is durable and easy to analyze. But I think the article doesn't talk about full data migrations or integrations of two very distinct models, but rather about incremental, on the fly migrations on production databases, so the above doesn't apply well I think.
- makapuf 5y ago
  Schema-less data does not make it unstructured data, it's just that it's now unknown/hidden and that you're perpetually in the middle of a migration with no obvious way to know you've finished migrating.
- simonw 5y ago
  Even a monorepo won't help if you run multiple servers with your application code - there will still be a few seconds (or even a few minutes) during your deploys when some of those servers are running an old version of the software and some are running the latest release.
rtpg 5y ago
Par of the problem is that tools don’t really have a notion of “behavior of system 2 releases ago”. If you had a Django plugin that, like, let you say “oh the previous version of the app that was deployed had this model shape, so during this migration I need to do X/Y/Z” using the git info, you could do more. But that would be a leaky abstraction and would run into a sort of coordination problem mid-deploy (what states are other machines in?). There might be some restrictions you could place to make this work though
mleonhard 5y ago
The lack of such tooling is one reason why devops is so hard and so hard to hire for.

YesThatTom2 5y ago

This is what the humans need to do. It works with any framework or database. You might find some software frameworks that make this easier than others but expecting to find a technology that replaces human methodology is like reading a book on Scrum and saying “can’t we make a software system that does this for us?”

zeckalpha 5y ago
No. Then we shouldn’t be using programming languages. I have learned to trust humans to not get migrations right. The system can have guardrails for certain structures of migrations.

0xbadcafebee 5y ago

Not a framework, but schemaless key-value stores allow you to play fast and loose and avoid a lot of the reason for migrations, with obvious downsides. Monorepos also allow you to change and deploy all of the code at once without having to coordinate versions of both dependencies and deployments.

dgb23 5y ago
Also using a key value store or some other no-sql non-graph storage can sometimes be put between two structured migration endpoints. The advantage is that you have a clear way to first pull things apart before you put them together again and an intermediate step that is durable and easy to analyze. But I think the article doesn't talk about full data migrations or integrations of two very distinct models, but rather about incremental, on the fly migrations on production databases, so the above doesn't apply well I think.
makapuf 5y ago
Schema-less data does not make it unstructured data, it's just that it's now unknown/hidden and that you're perpetually in the middle of a migration with no obvious way to know you've finished migrating.
simonw 5y ago
Even a monorepo won't help if you run multiple servers with your application code - there will still be a few seconds (or even a few minutes) during your deploys when some of those servers are running an old version of the software and some are running the latest release.

rtpg 5y ago

Par of the problem is that tools don’t really have a notion of “behavior of system 2 releases ago”. If you had a Django plugin that, like, let you say “oh the previous version of the app that was deployed had this model shape, so during this migration I need to do X/Y/Z” using the git info, you could do more. But that would be a leaky abstraction and would run into a sort of coordination problem mid-deploy (what states are other machines in?). There might be some restrictions you could place to make this work though

mleonhard 5y ago

The lack of such tooling is one reason why devops is so hard and so hard to hire for.