10 ms·
Using the expand and contract pattern for schema changes
- leetrout 11mo agoI use this example when I speak about and teach devops trainings. I call it the migration sandwich. (Nothing to do with the cube rule). A piece of bread isn't a sandwich and a single migration in a tool like alembic isn't a "sandwich" either. You have a couple layers of bread with one or several layers of toppings and it's not a sandwich until it's all done. People get a laugh out of the "idiot sandwich meme" and we always have a good conversation about what gnarly migrations people have seen or done (72+ hours of runtime, splitting to dozens or more tables and then reconstructing things, splitting things out to safely be worked on in the expanded state for weeks, etc). I had never heard it called "expand and contract" before reading this article a few years ago. What does everyone else call these?
- hobs 11mo agoTo me is just a blue-green type deployment for schemas. You have an old and a new thing, you split and merge as traffic replays to the new thing and shows that its viable and not breaking, you swap over as you can.
- mcdonje 11mo agoThis is the same pattern as versioning, but with an extremely short sunset for the old version.
- marcosdumay 11mo agoIt's actually not. Versioning is a concept where each version lives in non-intersecting time intervals. This concept is completely focusing on the fact that your structures lifetimes must absolutely have non-empty intersections. It's close to the opposite.
- chrisweekly 11mo ago> "Versioning is a concept where each version lives in non-intersecting time intervals." Is it? Node.js publishes "Current", "LTS" and "Maintenance" versions, and there's always a reasonable time interval during which consumers typically upgrade from eg Maintenance to newer LTS or even Current. From the publishing side, that's very similar to "expand and contract", in temporarily expanding what's supported to include Current, and dropping support for oldest versions leaving Maintenance. It's continuous instead of ad hoc, and there are more than 2 versions involved, but the principle is basically the same (at least if you squint). Though I guess if you're talking strictly about schema management strategies, then yeah, "versioning" might be very different from "expand contract", as you noted.
- UltraSane 11mo agoIt is often necessary to use multiple versions at the same time
- afiori 11mo agoThat is a very reductionist view and not particularly useful either. It shares elements with versioning and you could likely implement this using explicit versioning, but it is completely independent of it. The main difference I see is that of focus here the focus is to migrate a database without downtime or excessive global locks, keeping multiple versions of the schema is a detail.
- mcdonje 11mo agoSeems like you agree with my assessment.
- afiori 11mo ago:( Things can have similarities without being the same. Also not being unique is not a moral failure. I think I am failing to understand your stance.
- mcdonje 11mo agoSaying the pattern is the same isn't the same as saying two things fitting the pattern are the same in every respect. Patterns are necessarily reductionist, like any sort of comparison of things that aren't 100% similar. There is value in recognizing patterns. They're useful for comprehension and memory.
- tczMUFlmoNk 11mo agoI have usually heard it called "A–AB–B migrations". As in, you support version A, then you support both version A and version B, then you support just version B. The rest of the sequencing details follow from this idea.
- mayanraisins 11mo agoI’ve always paired this with the strangler pattern. “In programming, the strangler fig pattern or strangler pattern is an architectural pattern that involves wrapping old code, with the intent of redirecting it to newer code.” https://en.wikipedia.org/wiki/Strangler_fig_pattern https://en.wikipedia.org/wiki/Strangler_fig_pattern
- Normal_gaussian 11mo agoEvens between odds. Where even schema versions are migrations and 'divisible between two'. But green/blue and A/AB/B I've used before to discuss the same.
- fuzzy2 11mo agoI’m confused. I thought Expand and Contract was about mutating an existing schema, adding columns and tables, not creating a full replacement schema. But maybe I misunderstood? What’s in the article, I know as the Strangler Fig Pattern.
- jerriep 11mo agoWhat you describe is what they describe as well: > For column-level changes, this often means adding new columns to a table that have the characteristics you want while leaving the current columns as-is. I think what makes it confusing is that their diagrams depict a completely separate schema, but what they describe is really just altering the existing schema.
- DeathArrow 11mo ago> What’s in the article, I know as the Strangler Fig Pattern. Strangler fig pattern is mostly concerned with migrating from an old software to a new software, from example from a monolith to microservices. But I guess you can also apply it to database schemas.
- sarchertech 11mo agoThis is just the natural solution that falls out if you want to change a schema with no downtime. I always just called it “dual writing”.
- evancordell 11mo agoWe always called these "four-phase migrations". An old Stripe article used similar naming[0]. [0]: https://stripe.com/blog/online-migrations https://stripe.com/blog/online-migrations
- sarchertech 11mo agoI’ve heard that one too. I think the key insight is that you need to “stop the bleeding” (stop creating more old data that needs to be migrated) before you do any backfilling. That’s why I always called it dual writing because that’s the stop the bleeding step.
- manmal 11mo agoIn the step where writes are added, from the client to the new schema instance, how does this work when writes depend on existing data, when the existing data hasn’t even been copied? Constrains like foreign keys will prevent this from working, no?
- sarchertech 11mo agoYou wouldn’t enable foreign key constraints until you finish backfilling the old data in that case. Or you do it in phases where you do all of these steps and migrate the dependency first.
- tonetheman 11mo ago[dead]
- skywhopper 11mo agoThis is the model we used at the SaaS I worked for a decade ago. It worked great to allow for smooth, zero-downtime upgrades across a fleet of thousands of DB servers serving tens of thousands of app servers and millions of active users.
- maffyoo 11mo agoExpand Contract from Fowler's bliki https://martinfowler.com/bliki/ParallelChange.html https://martinfowler.com/bliki/ParallelChange.html
- layer8 11mo agoExpand the interface contract and then contract the interface contract? ;)
- krystofee 11mo agoIs there any easy way to implement this pattern in AWS RDS deployments where we need to deploy multiple times a day and need it to be done in few minutes?
- numbsafari 11mo agoIn my experience, this process typically spans multiple deploys. I would say the key insight that I have taken away from decades of applying this approach, is that data migrations need to be done in an __eventually consistent__ approach, rather than as an all-or-nothing, stop-the-world, global transaction or transformation. Indeed, this pattern, in particular, is extremely useful in environments where you are trying to making changes to one part of a system while multiple deploys are happening across the entire system, or where you are dealing with a change that requires a large number of clients to be updated where you don't have direct control of those clients or they operate in a loosely-connected fashion. So, regardless of AWS RDS as your underlying database technology, plan to break these steps up into individual deployment steps. I have, in fact, done this with systems deployed over AWS RDS, but also with systems deployed to on-prem SQL Server and Oracle, to nosql systems (this is especially helpful in those environments), to IoT and mobile systems, to data warehouse and analysis pipelines, and on and on.
- isuckatcoding 11mo agoOk hear me out. What if this whole process was statefully managed for you as an add on to your database? Like you essentially defined the steps in a temporal like workflow and then it does all the work of expanding, verifying and contracting.
- nightpool 11mo agoOn the Rails side, Gitlab has an extensive set of helpers for this that a lot of Rails projects have adopted—I would love to see them pulled out into a Gem or adopted into Rails core proper: https://gitlab.com/gitlab-org/gitlab-foss/blob/master/lib/gitlab/database/migration_helpers.rb https://gitlab.com/gitlab-org/gitlab-foss/blob/master/lib/gi...
- Fripplebubby 11mo agoI'm hearing you out, but how is this going to affect the part of this that is client behavior rather than database behavior? If there is some kind of sdk that actually captures the interface here (that is, that the client needs to be compatible with both versions of the schema at once for a while) and pushes that back to the client, that could be interesting, like a way to define that column "name" and columns "first name", "last name" are conceptually part of the same thing and that the client code paths must provide handling for both at once.
- marcosdumay 11mo agoIf you solve that "verifying" step, you will already revolutionize software development.
- vvern 11mo agoIt should be this way. Clients should have some protocol to communicate the schema they expect to the database probably with some versioning scheme. The database should be able to serve multiple mutually compatible views over the schema (stay robust to column renames for example). The database should manage and prevent the destruction of in use views of that schema. After an old view has been made incompatible, old clients needing that view should be locked out.
- davedx 11mo agoI use Prisma on almost all my node.js projects these days, and I wish that part of schema migrations was also automated by Prisma. But last I checked, it doesn't even rename columns properly. I feel like maybe they should invest more R&D in their migrations technology? The ORM is pretty great.
- chrisweekly 11mo agoLack of reasonable support for migrations turned me off to Prisma when I first encountered it (in a KCDodds Remix app circa 2021-ish). I'm surprised that's still unaddressed.
- pphysch 11mo agoLack of support for running migrations, or generating migrations?
- ambicapter 11mo agoWhat do you find good about Prisma?
- davedx 11mo agoThe ORM is great! I usually setup my projects so the Prisma generated types are shared with the front end and build my own types on top of those. I like the prisma schema first way of specifying my models too. It’s pretty intuitive and readable, it centralizes all my models in one place. The migration system could be more advanced but does the job. Multiple production projects I worked heavily on use it. Overall I think it’s very well designed software
- AtlasBarfed 11mo ago1) double write essentially 2) migration involves the problem of mixing a migration write with an actual live in flight mutation. Cassandra would solve this with additional per cell write time tracking or a migrated vs new mutation flag 3) and then you have deletes. So you'll need a tombstone mechanism, because if a live delete of a cell value is overwritten by a migrated value, than data that is deleted comes back to life
- deleted 11mo ago[deleted]
- 8note 11mo agoim not sure about the name, but this is a great little doc for introducing junior devs to migrations. the only thing i would add is a minor and major version changes, so its clear how the different class ent stages are labeled/how you track when you're ready to backfill
- sanjayjc 11mo agoThe following academic work (132 page pdf) elucidates this pattern in the context of a real application: https://ris.utwente.nl/ws/portalfiles/portal/275963001/PDEng_Thesis_Public_Jorryt_Jan_Dijkstra.pdf https://ris.utwente.nl/ws/portalfiles/portal/275963001/PDEng...