4 ms·
The big question that wasn't really covered, is how they are selectively updating the writers and readers on a live environment. Especially migrating on a per o
by overcast 10y ago
The big question that wasn't really covered, is how they are selectively updating the writers and readers on a live environment. Especially migrating on a per object basis to have them start dual writing, and reading.
- rockostrich 10y agoFrom my understanding, they migrate the backed-up old data to the new db after writers have already started writing any new updates to the new db. So the MapReduce job on that backed-up data should just have to look at timestamps on create/update to check whether or not to load that row into the new db.
- overcast 10y agoRight, but how is that new code rolled out to live systems with zero downtime? They said they try not to update more than a few hundred lines of code, and only on individual objects. What process are they using to roll that out to start the writing?
- jey 10y agoHere's one simple model: launch new containers into the load balancer pool (having the new code), and shutdown old containers. This can be done in an incremental way so you get a staged rollout with no downtime.
- overcast 10y agoContainers make sense, thanks.
- icebraining 10y agoOr use workers with a manager like uwsgi. Containers are overkill if you just want zero-downtime upgrades.
- clay_to_n 10y agoDoesn't even need to be containers, the key part is having a load-balancer that can shift the load to the new web applications deployed without downtime.
- luhn 10y agoFeature flags, most likely. Deploy the dual-write code behind a feature flag, enable the feature flag for a small fraction of requests, start notching up the percentage until you're at 100%, then remove the feature flag and the old single-write code. Rinse and repeat for the remaining three steps.
- overcast 10y agoThe actual deployment was my curiosity :)
- luhn 10y agoAh, my bad, I misunderstood. It's pretty normal for tech companies to do dozens of deploys a day, thanks to CI/CD. There are lots of ways to achieve this, but they all pretty boil down to the same idea: Boot up the new version before shutting down the old version.
- elgenie 10y agoWriting Code: if (migration_complete) { write_new_data_model(); } else if (migration_in_progress) { write_old_data_model(); write_new_data_model(); } else { write_old_data_model(); } Reading Code: if (migration_complete) { read_new_data_model(); } else { read_old_data_model(); } The `migration_in_progress` feature flag is flipped first, then the migration jobs run to backfill the new data model from the old data model, then the `migration_complete` flag is flipped. As many times as necessary, `migration_in_progress` can be flipped to off at any time and the new data model can be blown away and new writing code deployed. If there's no confidence in the code for reading the new data model, then a certain percentage of requests/users can be directed to read from the new data model while double-writing continues.
- overcast 10y agoI get the code bit, I was more curious as to how they in particular get that slipstreamed in there. Load balancing deployments as others mentioned below seems to be the most feasible.