4 ms·
We just did a large migration, transitioning our deployment pipeline and front facing web server to a new set up. I stayed at work till like 4 am. Everything we
by photonios 8y ago
We just did a large migration, transitioning our deployment pipeline and front facing web server to a new set up. I stayed at work till like 4 am. Everything went smoothly.
The next day I woke up and realized I forgot to enable password protection on the staging environment. I opened up my laptop, navigated to the dashboard, add the flag and reload the config. 5 seconds later, my phone starts beeping. Tons of alerts that the production website was down and responding with "403 access denied". Immediately tens of people jump on Slack "website down?!".
Turn out, with my sleepy head, I added the flag to the production website instead of the staging website. And thus enabled password protection on the live website. As soon as I noticed, I undid it and in the end, downtime was roughly a minute.
Suffice it to say, I got slapped for this fuck up.
- exolymph 8y agoOh noooo! Can't believe you had to stay that late at work. Poor planning leading up to the migration, or just unexpected issues?
- photonios 8y agoNo other time to pull it off. We run a large website in a different part of the world, so we're several time zones behind. We could only do this migration during low traffic, which for us meant the middle of the night. Sounds worse than it is. Got to work in the afternoon, got pizza and took a couple of extra free days exchange.