5 ms·
This was in 2012. Let's just say that a router config only getting partially applied (because of a software bug) can cause a LOT of traffic to suddenly go place
by packetslave 9y ago
This was in 2012. Let's just say that a router config only getting partially applied (because of a software bug) can cause a LOT of traffic to suddenly go places it's not supposed to go.
- ronald_raygun 9y agoOh wow, I was a noogler there 4ish years ago, and they still tell that story as an example of "its okay to break things, we care about fixing them after so they dont break in the future"
- grivescorbett 9y agoWas a noogler about a year ago. Story still told.
- aalleavitch 9y agoIt appears we have a living legend here!
- ianai 9y agoThey made him a slave of the packets though.
- packetslave 9y agoThere's a couple of good stories about massive outages and good incident response. Mine is just one of them (and at some level, I was very lucky). There's also the one where all the frontend servers worldwide went into a crash loop from a bad configuration push. The SRE doing the push noticed some "weirdness" and rolled back even before the full scope of the issue was known. That one's in the SRE book.
- mynewtb 9y agoGFE? SRE?
- mikejb 9y agoSRE is Site Reliability Engineer; GFE is the "Google Front End"
- Bluestrike2 9y agoSite Reliability Engineering.[0] Google's SRE book is a pretty interesting read. 0. https://landing.google.com/sre/interview/ben-treynor.html https://landing.google.com/sre/interview/ben-treynor.html
- packetslave 9y agoI think my favorite quote about my outage was "boy, it's a good thing you didn't decide to go take a crap right about then..."
- ceph_ 9y agodb31.lpp01 lol
- packetslave 9y agoWhen they decommed it, I asked if I could have db31 to go Office Space on. They said no :-(