5 ms·
Maybe it’s just the trained operator in me but the whole time he’s describing the depressed metrics and blip I’m screaming in my head: it’s exogenous! Check the
by encoderer 4y ago
Maybe it’s just the trained operator in me but the whole time he’s describing the depressed metrics and blip I’m screaming in my head: it’s exogenous! Check the news! He finally gets there.
For us (at normal company scale) I would actually go look at recent activity, new customers, new workloads we are running, but I guess at a Facebook scale it’s a lot harder to do that.
- nerdponx 4y agoOne interesting aspect of the story is that the author was a junior at the time (almost brand new to the job, if I read it correctly), and it was one of the more experienced operators who realized it was exogenous. in addition to being a good story about the site reliability, it's also a great lesson in the value of collaboration, mentorship, and having senior people around whom you can ask for help!
- MrAwesome 4y agoAuthor here, and that's exactly right! I think it was in my second or third month on the job. And I really agree with your second point too! Without that in-person transfer of tribal knowledge, it would have taken me ages to learn the ropes well enough to take such a high-pressure high-stakes oncall.