3 ms·
The biggest mistake is not ringfencing this update in a test environment before sign-off for general deployment.
by linker3000 2y ago
The biggest mistake is not ringfencing this update in a test environment before sign-off for general deployment.
- jaza 2y agoAgreed. What happened to Patch Tuesdays?!
- chippiewill 2y agoPresumably they do test their updates, they're just maybe not good enough tests. The ideal would be to do canary rollouts (1%, then 5%, 10% etc.) to minimise blast radius, but I guess that's incompatible with antiviruses protecting you from 0-day exploits.
- spike021 2y agoWhile I'm usually a proponent of update waves like that, I know some teams can get loose with the idea if they determine the update isn't worth that kind of carefulness. Not saying CS doesn't care enough but what may be a minor update to the team that did this and not necessary for a slow rollout is actually something that really should be supervised in that way.
- tetha 2y agoOur worst outage occurred when we were deploying some kernel security patches and we grew complacent and updated the main database and it's replica at the same time. We had a maintenance with downtime anyway at the same time, so whatever. The update worked on the other couple hundred systems. Except, unknown to us, our virtualization provider had a massive infrastructural issue at exactly that moment preventing VMs from booting back up... That wasn't a fun night to failover services into the secondary DC.
- rk06 2y agoWas this update meant to save from a 0 day?
- grumple 2y agoUpdate: change color of text in console