3 ms·

They touched on this in their SRE post last week: https://cloudplatform.googleblog.com/2017/03/reliable-releases-and-rollbacks-CRE-life-lessons.html https://clo

• retreatguru 10y ago
  Thanks. Basically they do a db change inbetween rollouts with no feature yet. Clever.
• ams6110 10y ago
  From that link: At Google, our philosophy is that “rollbacks are normal.” When an error is found or reasonably suspected in a new release, the releasing team rolls back first and investigates the problem second. I like that -- reminds me of aviation, where a go-around is normal. If your approach to landing isn't stabilized, you're too high, too low, too fast, too slow, etc. don't try to save it. Go around and try again.
  • londons_explore 10y ago
    At our place, we rollback every few weeks just to test the system, even if nothing appears abnormal. Next we plan to automate the process - 1 in 10 rollouts will actually be a rollout, a rollback, and another rollout, checking system health at each step.
    • hectormalot 10y ago
      I just wanted to suggest this. Similar to 'chaos monkey' killing my processes once every few days, developers would also look differently at rollback procedures if I guaranteed that one in 10 would be rolled back randomly.