3 ms·

If they are reading this and need an SRE to help out let me know. We are really good at reverting the shit out of an release when it breaks prod.

• jrockway 5y ago
  My team has also become good at that. We do "game days" where we push a simulated breakage and the team tries to fix it, but after a few "faulty" code pushes, everyone checks the rollout history for recent code or configuration changes and suggests a rollback as the first step. Having this log makes it very hard to break things! This just means the breakers have to be more clever. My favorites are taking a global lock in Postgres (all reads and writes instantly stop, and someone has to figure out how to kill the session that holds the lock), and installing a service worker that blocked traffic to our domain. (You can roll back, but the service worker is still there!)
  • Guest42 5y ago
    Does fixing constitute bringing back everything to the state things were at when the global lock was taken or does it also need to handle requests that occurred after? (I’m more of a db person with a small understanding of web apps)
    • jrockway 5y ago
      Fixing restores the ability of the application to handle future requests. Past requests were already replied to with an error.