When I worked at Twitter we wanted to find out if we were finally going to survive New Years. Our head of SRE wanted me to test in prod, which horrified me but

When I worked at Twitter we wanted to find out if we were finally going to survive New Years. Our head of SRE wanted me to test in prod, which horrified me but he convinced me in the end. In order to simulate the population of Japan I had to make a bunch of fake users. I spent a fair amount of time making sure they wouldn’t get caught up in any analytics, throwing off our active user numbers, and I managed to peg the ‘follows’ service getting them all to follow each other in a reasonable distribution. I also needed to bypass the rate limiter, but since I was in prod I could just reset my own counter in prod and effectively be totally limitless. Two things broke in a visible way during all of this. During testing everything was wired up to my personal account. I managed to spam all my followers with thousands of happy new year tweets in a couple seconds since I wasn’t subject to the rate limiter. I deleted all but one of those, which I left to remind myself that with great power comes great stories of things going wrong. The other thing was a bit more dramatic, albeit short-lived. The first big test had everyone ready to go. I hit enter on the job, and at the time (maybe still) I had no way to get metrics out of production at a granularity less than one minute. A very worried minute goes by, and then we realize I’ve DDOSed the authentication service. All my fake accounts needed to auth to actually tweet, and naturally they did that first. Since the whole point of the test was for load to all hit in roughly the same second, the auth load also all arrived in the same second. Oops. We decided that was an unfair test, I spent a few hours getting auth tokens for my fake users, and we tried again. That time everything worked, and we also survived New Year’s... But it was fun getting there.