4 ms·
> Sorry for maybe a silly question, but how do feature flags work with migrations? If your migrations run automatically on deploy Basically they don't. Databas
by toast0 5y ago
> Sorry for maybe a silly question, but how do feature flags work with migrations? If your migrations run automatically on deploy
Basically they don't. Database migration based on frontend deploy doesn't really make sense at facebook scale, because deploy is no where close to synchronous; even feature flag changes aren't synchronous. I didn't work on FB databases while I was employed by them, but when you've got a lot of frontends and a lot of sharded databases, you don't have much choice; if your schema is changing, you've got to have a multiphased push:
a) push frontend that can deal with either schema
b) migrate schema
c) push frontend that uses new schema for new feature (with the understanding that the old frontend code will be running on some nodes) --- this part could be feature flagged
d) data cleanup if necessary
e) push code that can safely assume all frontends are new feature aware and all rows are new feature ready
IMHO, this multiphase push is really needed regardless of scale, but if you're small, you can cross your fingers and hope. Or if you're willing to take downtime, you can bring down the service, make the database changes without concurrent access, and bring the service back with code assuming the changes; most people don't like downtime though.
- kgeist 5y ago>Basically they don't. Database migration based on frontend deploy doesn't really make sense at facebook scale, because deploy is no where close to synchronous; even feature flag changes aren't synchronous. Our deployments aren't strictly "synchronous" either. We have thousands of database shards which are all migrated one by one (with some degree of parallelism), and new code is deployed only after all the shards have migrated. So there's a large window (sometimes up to an hour) when some shards see the new schema and others see the old schema (while still running old code). It's one click of a button, however, and one logical release, we don't split it into separate releases (so I view them as "automatic"). The problem still stays, though, that you can only guard code with feature flags, migrations can't be conditionally disabled. With this setup, if a poorly tested migration goes awry, it's even more difficult to rollback, because it will take another hour to roll back all the shards.
- withinboredom 5y agoWe don't have a staging environment (for the backend) at work either. However, depending on the size of the tables in-question, a migration might take days. Thus, we usually ask DBA's for a migration days/weeks before any code goes live. There's usually quite a bit of discussion, and sometimes suggestions for an entirely different table with a join and/or application-only (in code, multiple query) join.
- catlifeonmars 5y agoSorry for the silly question, perhaps, but what is the purpose of a db migration? Do schemas in production change that often? For context, the last couple of services I wrote all have fixed, but implicit schema, (built on key value stores). That is, the DB has no types. So instead, the type system is enforced by the API layer. Any field changes so far are gated via API access and APIs have backwards compatibility contracts with API callers. I’m not saying that the way I do it currently is “correct” - far from it. I strongly suspect it’s influenced by my lack of familiarity with relational databases.
- withinboredom 5y agoA fairly recent example is a couple of tables for users who are “tagged” for marketing purposes (such as we sent them an email and want to display the same messaging in the app). These tags have an expiration date at the tag level but we wanted the expiration date per-user too. This enables marketing to create static tags. This requires a migration to the data so this can be supported. Schemas don’t change that often, in my experience.
- 20after4 5y agoRelational databases can be very strict, for example if you use foreign key references then the data base enforces that a row exists in the referenced table for every foreign key in the referring table. This strict enforcement makes it difficult to change schema. The way you handle things with API level enforcement is actually a good architecture and it would probably make schema changes easier to deal with even on a relational database backend.
- deleted 5y ago[deleted]
- hinkley 5y agoFor minor changes there's a simpler path, where you add a new field to the database, default the value to some reasonable value, then add it to the workflow in stages. Depending on how your database feels about new columns and default values, there may be additional intermediate steps to keep it happy.