5 ms·
When i was working for one of the biggest regional websites: We had blue/green deployments using AWS Elastic Beanstalk - our actual deployment process was stil
by avh02 5y ago
When i was working for one of the biggest regional websites:
We had blue/green deployments using AWS Elastic Beanstalk - our actual deployment process was still manual and we had an informal (i.e: not written down but everybody knew) checklist to follow - one of the steps involved checking that capacities matched.
Well, everything looked good and I switched environments, as I was checking that the website was still up, i was getting 503's. I realised immediately that I forgot to actually scale up the environment that was now receiving traffic - 1 server was handling what we generally had a minimum of 30 to handle.
Immediately swapped back, scaled up, and swapped again. ~5 minutes of downtime in the morning.
- samus 5y agoNot too bad. Unless you have a contract or regulations that says otherwise, 5 mins downtime once in a while is really nothing to lose your sleep about. Did you even have to report it?
- avh02 5y agoI honestly don't remember, but i definitely remember the checklist became more formal as we made a lot of errors like this. I left soon afterwards but the stack was also a transition-to-AWS stack so was later moved to something more suited. 5 mins downtime was definitely small for that place - we regularly had larger issues. I learned a phenomenal amount in my time there though.