4 ms·
No lesson learned, they're still doing it wrong. 1. They shouldn't be deploying to one of two redundant systems and then deploying to the other one an hour lat
by boner666 6y ago
No lesson learned, they're still doing it wrong.
1. They shouldn't be deploying to one of two redundant systems and then deploying to the other one an hour later. One system should keep running the old code for days, if not weeks. That would have prevented the loss of AGC.
2. They should have rollback procedures in case an update causes problems. Instead they added an emergency patch on top of the new bad code. This worked out OK but could have fucked things up even worse.
- BunsanSpace 6y ago1. when doing a system upgrade, a lot of times the new version won't play nicely with the old version. Key example iFix, not backwards (or forwards!) compatible. 2. sometimes addin a quick fix with a support engineer on the line is quicker than attempting a role back. You're making a lot of assumptions here, but likely what happened was a try to fix this asap and if that fails roll back scenario.