3 ms·

What I need from migrations is collaboration. When I make a schema migration locally and push it to the repository, then 3 other PRs with their own migrations

by linkdd 6y ago

What I need from migrations is collaboration. When I make a schema migration locally and push it to the repository, then 3 other PRs with their own migrations are being merged before mine, it is very likely that my local database will be in the wrong state. Correct me if I'm wrong, but using an ORM (be it Django, Prisma, liquibase, ...) to handle your migrations will only generate the SQL queries for you, it won't handle "migration history merges" which is what you'll most likely need. Your schema may be versioned in your repository, but not your data. Therefore, you still need to coordinate your developers when database change are needed. And you'll most likely need to rm -rf your database locally and recreate it. Unfortunately, that is not an easy problem to solve. Fortunately, this should not be a problem on production environments (where their history is more linear). I'd be curious of any work on that subject.

sorenbs 6y ago
The Prisma Migrate team is thinking about these issues. I'll let them elaborate when they wake up tomorrow :-) We are taking inspiration from this great writeup on how Github has implemented a workflow that enable their developers to collaborate on Schema Migrations at scale: https://github.blog/2020-02-14-automating-mysql-schema-migrations-with-github-actions-and-more/ https://github.blog/2020-02-14-automating-mysql-schema-migra...
csncsu 6y ago
EF 6 had an interesting take on this problem. If two migrations had the same parent (aka we both branched and added a migration), whoever merged second would have to generate a "merge" migration to resolve the conflict. It wasn't the most elegant solution, or even a solution period, but it made the problem explicit. This happened because the full schema model was serialized and encoded into the migration journal table.
- linkdd 6y ago
  This is the actual model of git. You try to push on a remote after somebody else, you have to create a merge commit. But it would be nice to have a rebase to keep linearity in your history.
tomhoule 6y ago
(prisma migrate team member here) The development flow in migrate (the `migrate dev`) command is quite pedantic: it will check things like migrations missing from your migrations folder but already applied to the dev database, or migrations that were modified since they were applied (via a checksum) and guide you towards resolving the problem. That can happen because of merges, but even more commonly when you are just switching branches locally, or editing migrations. We're also looking into ways to integrate with your CI and review process to provide more insight, for example when your branch gets stale and you need to "rebase" your migrations on the migrations from the main branch. It's something we are actively exploring, and we're more than happy to get your feedback and ideas on github/slack :) My personal view is that _some level_ of consciousness of the migration process (schema and data) will always be required from developers on large enough projects. What we can do is build tools that help people getting their migrations right, but it can't be completely automated. There's a lot more to do, now that we have a stable, production ready foundation.
- linkdd 6y ago
  Thank you for your clarification. I like to think about a single migration being a function and a migration history multiple functions being composed. Category theory's main subject (I think) is composition and a lot of good ideas can be taken from there (see this paper: "Functorial Data Migration"[1]). There is an obvious relationship between a schema and the data, and a migration operates on both. The schema might be versioned, but unless you use a Schema-less database (ie: MongoDB) and handle the data versioning application side, your data is not. Any document/graph-based database can avoid this problem by assigning a version to your documents (like apiVersion+kind in Kubernetes), then the application should know how to process each version of the data. In an SQL database, it would require different tables which complexify the data model. IMHO, in every case, you need to handle some of the logic application side or maintain an history of your data (bye-bye storage space?). [1] - http://math.mit.edu/~dspivak/informatics/CD-FDM.pdf http://math.mit.edu/~dspivak/informatics/CD-FDM.pdf
  - tomhoule 6y ago
    Ooh it's interesting you mention David Spivak's work, I got interested into it recently. Some of these ideas are being commercialized by https://conexus.com/ https://conexus.com/ (he's a co-founder). You might also be interested by Project Cambria[1] for novel ideas about data and schema migrations based on lenses. We haven't gone in the fancy direction yet, because we wanted to first have the tool our users told us they were missing - namely a migration tool similar to what other ORMs have, but playing on the strengths of the Prisma schema. But these novel ideas about migrations are definitely something we always read with a lot of interest and love to discuss — I hope we get to implement something radical like that some day. [1]: https://www.inkandswitch.com/cambria.html https://www.inkandswitch.com/cambria.html

by linkdd 6y ago

sorenbs 6y ago

The Prisma Migrate team is thinking about these issues. I'll let them elaborate when they wake up tomorrow :-) We are taking inspiration from this great writeup on how Github has implemented a workflow that enable their developers to collaborate on Schema Migrations at scale: https://github.blog/2020-02-14-automating-mysql-schema-migrations-with-github-actions-and-more/ https://github.blog/2020-02-14-automating-mysql-schema-migra...

csncsu 6y ago

EF 6 had an interesting take on this problem. If two migrations had the same parent (aka we both branched and added a migration), whoever merged second would have to generate a "merge" migration to resolve the conflict. It wasn't the most elegant solution, or even a solution period, but it made the problem explicit. This happened because the full schema model was serialized and encoded into the migration journal table.

linkdd 6y ago
This is the actual model of git. You try to push on a remote after somebody else, you have to create a merge commit. But it would be nice to have a rebase to keep linearity in your history.

tomhoule 6y ago

(prisma migrate team member here) The development flow in migrate (the `migrate dev`) command is quite pedantic: it will check things like migrations missing from your migrations folder but already applied to the dev database, or migrations that were modified since they were applied (via a checksum) and guide you towards resolving the problem. That can happen because of merges, but even more commonly when you are just switching branches locally, or editing migrations. We're also looking into ways to integrate with your CI and review process to provide more insight, for example when your branch gets stale and you need to "rebase" your migrations on the migrations from the main branch. It's something we are actively exploring, and we're more than happy to get your feedback and ideas on github/slack :) My personal view is that _some level_ of consciousness of the migration process (schema and data) will always be required from developers on large enough projects. What we can do is build tools that help people getting their migrations right, but it can't be completely automated. There's a lot more to do, now that we have a stable, production ready foundation.

linkdd 6y ago
Thank you for your clarification. I like to think about a single migration being a function and a migration history multiple functions being composed. Category theory's main subject (I think) is composition and a lot of good ideas can be taken from there (see this paper: "Functorial Data Migration"[1]). There is an obvious relationship between a schema and the data, and a migration operates on both. The schema might be versioned, but unless you use a Schema-less database (ie: MongoDB) and handle the data versioning application side, your data is not. Any document/graph-based database can avoid this problem by assigning a version to your documents (like apiVersion+kind in Kubernetes), then the application should know how to process each version of the data. In an SQL database, it would require different tables which complexify the data model. IMHO, in every case, you need to handle some of the logic application side or maintain an history of your data (bye-bye storage space?). [1] - http://math.mit.edu/~dspivak/informatics/CD-FDM.pdf http://math.mit.edu/~dspivak/informatics/CD-FDM.pdf
- tomhoule 6y ago
  Ooh it's interesting you mention David Spivak's work, I got interested into it recently. Some of these ideas are being commercialized by https://conexus.com/ https://conexus.com/ (he's a co-founder). You might also be interested by Project Cambria[1] for novel ideas about data and schema migrations based on lenses. We haven't gone in the fancy direction yet, because we wanted to first have the tool our users told us they were missing - namely a migration tool similar to what other ORMs have, but playing on the strengths of the Prisma schema. But these novel ideas about migrations are definitely something we always read with a lot of interest and love to discuss — I hope we get to implement something radical like that some day. [1]: https://www.inkandswitch.com/cambria.html https://www.inkandswitch.com/cambria.html