3 ms·
While testing a live update to a safety-critical traffic monitoring system I had four sessions running. Development, deployment, shadow, and a link to the live
by RiderOfGiraffes 16y ago
While testing a live update to a safety-critical traffic monitoring system I had four sessions running. Development, deployment, shadow, and a link to the live system.
I was checking the deployment to the shadow system, repeatedly running over the check-list to ensure that the deployment went smoothly, and then running a comparison between the shadow system and the live system to check that the update took place properly.
Between each test I took down the shadow system and brought it back up in a clean duplicate of the live system. Except on this one occasion when I had a brain-fart and killed the live system.
It all came back as designed within 30 seconds, re-initialised and restarted without loss of data, but it was a scary moment.