3 ms·
My team has also become good at that. We do "game days" where we push a simulated breakage and the team tries to fix it, but after a few "faulty" code pushes,
by jrockway 5y ago
My team has also become good at that. We do "game days" where we push a simulated breakage and the team tries to fix it, but after a few "faulty" code pushes, everyone checks the rollout history for recent code or configuration changes and suggests a rollback as the first step. Having this log makes it very hard to break things!
This just means the breakers have to be more clever. My favorites are taking a global lock in Postgres (all reads and writes instantly stop, and someone has to figure out how to kill the session that holds the lock), and installing a service worker that blocked traffic to our domain. (You can roll back, but the service worker is still there!)
- Guest42 5y agoDoes fixing constitute bringing back everything to the state things were at when the global lock was taken or does it also need to handle requests that occurred after? (I’m more of a db person with a small understanding of web apps)
- jrockway 5y agoFixing restores the ability of the application to handle future requests. Past requests were already replied to with an error.