3 ms·

My "become the joker" take is that the SRE book is completely misunderstood and misapplied by small shops that just do not have the profit margins of GOOG to st

My "become the joker" take is that the SRE book is completely misunderstood and misapplied by small shops that just do not have the profit margins of GOOG to staff internal tooling & operations at the same scale. So instead of spending scarce resources on the 80/20 monitoring of compute/memory/disk/network resources we see shops build Rube Goldberg machines to drive KPI dashboards that the CTO can stare at. Except that there's no one really spending the time to pick meaningful KPIs and levels. Hey, I get it.. "cattle not pets".. but, if some pods are constantly falling over then maybe monitor & address that rather than waiting for some 3 steps removed KPI on error rates crosses the 97.5% level. I've been at too many shops where they got observability-pilled and constantly still fell over on the types of underlying resource starvation outages most tooling had well addressed by 2005. If you don't notice you ran out of some storage resources last night until a cascade of failures cause a user facing data availability SLA to be breached.. that's silly.

nunez 3y ago
That's not "become the joker;" that's facts. The SRE book was, literally, Googlers documenting how they did SRE at Google. The book (i.e. the first two chapters) got cargo-culted everywhere, everything else became footnotes. See also: The massive set difference of places that have something resembling SLOs and places that implement interrupts and fair compensation for on-call rotations.
- steveBK123 3y ago
  Yes, exactly. I've seen small companies that are smaller than Google's SRE staff alone try to implement the concept. As it turns out if your entire tech org is 50~200 people, hiring 2 people to "do SRE" because the CTO read a book.. is not gonna do the trick. CTO gonna get pretty dashboard though!
  - nunez 3y ago
    Ah, yes, the SLO Dashboard™ that makes at least eight gagillion dollars for Datadog every year that looks super pretty for execs but is amazingly useless for SREs that actually need to fix shit at 0330. i.e. basically this: https://www.youtube.com/watch?v=VQPIdZvoV4g&pp=ygUadGhlIG1hY2hpbmUgdGhhdCBnb2VzIHBpbmc%3D https://www.youtube.com/watch?v=VQPIdZvoV4g&pp=ygUadGhlIG1hY...
    - steveBK123 3y ago
      I assume it was on a Gartner Magic Quadrant

nunez 3y ago

That's not "become the joker;" that's facts. The SRE book was, literally, Googlers documenting how they did SRE at Google. The book (i.e. the first two chapters) got cargo-culted everywhere, everything else became footnotes. See also: The massive set difference of places that have something resembling SLOs and places that implement interrupts and fair compensation for on-call rotations.

steveBK123 3y ago
Yes, exactly. I've seen small companies that are smaller than Google's SRE staff alone try to implement the concept. As it turns out if your entire tech org is 50~200 people, hiring 2 people to "do SRE" because the CTO read a book.. is not gonna do the trick. CTO gonna get pretty dashboard though!
- nunez 3y ago
  Ah, yes, the SLO Dashboard™ that makes at least eight gagillion dollars for Datadog every year that looks super pretty for execs but is amazingly useless for SREs that actually need to fix shit at 0330. i.e. basically this: https://www.youtube.com/watch?v=VQPIdZvoV4g&pp=ygUadGhlIG1hY2hpbmUgdGhhdCBnb2VzIHBpbmc%3D https://www.youtube.com/watch?v=VQPIdZvoV4g&pp=ygUadGhlIG1hY...
  - steveBK123 3y ago
    I assume it was on a Gartner Magic Quadrant