3 ms·

What are the best practices redarding rollbacks when the database is affected. I would think a large amount of overhead would be required.

ademarre 10y ago
They touched on this in their SRE post last week: https://cloudplatform.googleblog.com/2017/03/reliable-releases-and-rollbacks-CRE-life-lessons.html https://cloudplatform.googleblog.com/2017/03/reliable-releas...
- retreatguru 10y ago
  Thanks. Basically they do a db change inbetween rollouts with no feature yet. Clever.
- ams6110 10y ago
  From that link: At Google, our philosophy is that “rollbacks are normal.” When an error is found or reasonably suspected in a new release, the releasing team rolls back first and investigates the problem second. I like that -- reminds me of aviation, where a go-around is normal. If your approach to landing isn't stabilized, you're too high, too low, too fast, too slow, etc. don't try to save it. Go around and try again.
  - londons_explore 10y ago
    At our place, we rollback every few weeks just to test the system, even if nothing appears abnormal. Next we plan to automate the process - 1 in 10 rollouts will actually be a rollout, a rollback, and another rollout, checking system health at each step.
    - hectormalot 10y ago
      I just wanted to suggest this. Similar to 'chaos monkey' killing my processes once every few days, developers would also look differently at rollback procedures if I guaranteed that one in 10 would be rolled back randomly.
was_boring 10y ago
At my job the DB is backwards compatible by one release. This means we roll back code, not data. It takes a little longer to do something like delete a column, but it's worth it. Rainforest QA had a blog post on strategies a while ago.
- londons_explore 10y ago
  Simply testing that you maintain backwards compatibility is a royal pain though. One could imagine a test rig which created a db with test users in it, then ran v1 of the software, then v2, then v1 again, and checked v1 didn't crash on the now mutated datastructures. Most datastore mutations might only happen on certain user actions and inputs, so making sure you get full coverage would be rather tricky.

ademarre 10y ago

They touched on this in their SRE post last week: https://cloudplatform.googleblog.com/2017/03/reliable-releases-and-rollbacks-CRE-life-lessons.html https://cloudplatform.googleblog.com/2017/03/reliable-releas...

retreatguru 10y ago
Thanks. Basically they do a db change inbetween rollouts with no feature yet. Clever.
ams6110 10y ago
From that link: At Google, our philosophy is that “rollbacks are normal.” When an error is found or reasonably suspected in a new release, the releasing team rolls back first and investigates the problem second. I like that -- reminds me of aviation, where a go-around is normal. If your approach to landing isn't stabilized, you're too high, too low, too fast, too slow, etc. don't try to save it. Go around and try again.
- londons_explore 10y ago
  At our place, we rollback every few weeks just to test the system, even if nothing appears abnormal. Next we plan to automate the process - 1 in 10 rollouts will actually be a rollout, a rollback, and another rollout, checking system health at each step.
  - hectormalot 10y ago
    I just wanted to suggest this. Similar to 'chaos monkey' killing my processes once every few days, developers would also look differently at rollback procedures if I guaranteed that one in 10 would be rolled back randomly.

was_boring 10y ago

At my job the DB is backwards compatible by one release. This means we roll back code, not data. It takes a little longer to do something like delete a column, but it's worth it. Rainforest QA had a blog post on strategies a while ago.

londons_explore 10y ago
Simply testing that you maintain backwards compatibility is a royal pain though. One could imagine a test rig which created a db with test users in it, then ran v1 of the software, then v2, then v1 again, and checked v1 didn't crash on the now mutated datastructures. Most datastore mutations might only happen on certain user actions and inputs, so making sure you get full coverage would be rather tricky.