I worked with someone who pushed out a config change to production and went home. Soon, he got a call asking what he did because all of our credit card process

I worked with someone who pushed out a config change to production and went home. Soon, he got a call asking what he did because all of our credit card processing went down. Shortly after that we got a call from our credit card processor (one of the largest in the country) asking what we did because it was causing a cascading failure in their systems and had taken down a big chunk of online credit cards processing including Apple and Walmarts websites. We tracked it down to the commit which had accidentally nulled the a username for authenticating with our credit card processor. They had a previously unknown bug that caused a memory fault when the username was null. The processing was queue based so when one machine failed, another would pick it up and try and process it and would fail. This happened until the entire data center fell over. A few of the lessons learned: 1. Always check the length of user inputs. 2. Build in circuit breakers to prevent cascading failures. 3. Don’t push code without review to production, minutes before leaving. Everyone involved kept their jobs and we learned a lot of programming lessons that day.