4 ms·
100% of my rollbacks were like this: * Deploy new code in new VMs. * Route some prod traffic to the new nodes. * Watch the nodes misbehave somehow. * Route
by nine_k 7y ago
100% of my rollbacks were like this:
* Deploy new code in new VMs.
* Route some prod traffic to the new nodes.
* Watch the nodes misbehave somehow.
* Route 100% of the prod traffic back to old nodes (which nobody tore down).
Rollback complete.
In the case of normal deployment, 100% of prod traffic would eventually be directed to new modes. After a few hours of everything running smoothly, the old nodes would be spun down.
- mooreds 7y agoSo all of your new code was completely backward compatible with your old code (in terms of state)? That is, the database didn't care which version was interacting with it?
- nine_k 7y agoYes, it was backwards-compatible for at least one DB migration step. That is, you can roll forward / back the database schema without breaking other code, or roll forward / back the code without affecting the DB. This does take more planning and gradual deployment, but saves the day when it matters.