4 ms·
I hate it when button mashers are in charge of incident response. Things like when traffic ramps up in the morning and the site falls over and someone in charg
by lamontcg 3y ago
I hate it when button mashers are in charge of incident response.
Things like when traffic ramps up in the morning and the site falls over and someone in charge blames the deployment the night before and screams "rollback", and then it eventually turns out to have nothing at all to do with the deployment.
- dmoy 3y agoTo be fair, if your rollbacks are cheap (1-3 button pushes), safe, and automated, it doesn't really hurt to try a rollback while you look for other causes.
- lamontcg 3y agoMost of my production experience precedes CI/CD as a concept by a number of years. You still don't know though if the rollback fixed it or if just bouncing the software alone fixed some kind of brownout, and you may not be able to replicate what happened and get stuck unsure of what the bug was and be in a state where you don't know if you can roll forward again or not. Then a developer throws a bit of spackle on the code to address what the problem "might" be -- then when you roll forward people assume that fixed it, which it might not, which is how cargo cults get formed.
- JohnMakin 3y agoCode changes should be trivial to roll back. Infra changes can be very difficult to roll back, however.
- rurp 3y agoNot to mention the times someone has said that an issue couldn't possibly have been caused by a recent change they made, only to find out that it was.