3 ms·
SREs only have a supporting role to play in change rollback. They maintain the tools/services used by SWEs, DBAs, etc to follow the rollback procedure documente
by peterwwillis 10y ago
SREs only have a supporting role to play in change rollback. They maintain the tools/services used by SWEs, DBAs, etc to follow the rollback procedure documented in Change Management.
Rollback is often a suitable first response because the rollback is not a button, it is a procedure, and it can be practiced in a test environment and on a small subset of production without significant customer impact.
Like the other commenter suggested, gradual rollout is a great way to spot bugs or customer feedback. You can accomplish this by building deployment tools that allow for targeted deployment, and by maintaining services that can be targeted by release. For example, you can use your Configuration Management engine to snapshot a DB, deploy schema changes to it and then your code to the app servers using only that DB. If there's a problem you can roll back the code, the data, the schema, whatever changed at that release version, and reload all the services. For things that affect customer assets you'll need extra procedures to work around potential failures there, but that's not complicated (usually).