3 ms·

I always wonder how stuff like this squares with the nearly-unanimous praise for Google's SRE book series. My colleagues in IT engineering all hold those books

I always wonder how stuff like this squares with the nearly-unanimous praise for Google's SRE book series. My colleagues in IT engineering all hold those books up as the gold standard in site reliability, but then really stupid mistakes like this surface in the news. So, which is it? Is Google's SRE practice not all it's cracked up to be? Are they ignoring their own well-publicized advice regarding the avoidance of these blunders? I do not want to entertain the idea that the system was working as designed and calling it a mistake is some kind of coverup, as other people in this thread have said, because the idea that a corporation the size and power of YouTube is willing to reverse unethical decisions and throw their own engineers under the bus as scapegoats is horrific.

dannyw 6y ago
SRE is a discipline. Content and account moderation is a cost centre.
joshuamorton 6y ago
"abuse system deletes some comments" isn't a failure mode sre concerns itself with. Buggy software can, and will get released despite whatever operational principles you have. Sre isn't about buglessness, but reliability.
- stonogo 6y ago
  Isn't this utterly at odds with the concept of code ownership, and how SRE doesn't take systems into production without owning them first?
  - joshuamorton 6y ago
    SRE doesn't own the code. They own the system reliability. Devs are still responsible for writing the functional code. Certain aspects of the release process may fall under SRE's purview, but what the tool does is ultimately the responsibility of developers (and product). The SRE book covers testing in one chapter (17), and most of that chapter is about production testing. The question you need to ask is "is the system still within in SLOs?" If so, SRE is doing their job. If you think that resulted in a bad experience, change the SLO. But I'd ask what the heck kind of SLO would have prevented this issue?
    - londons_explore 6y ago
      "99.9 % of spammers get to see their spam posted within 300ms"

dannyw 6y ago

SRE is a discipline. Content and account moderation is a cost centre.

joshuamorton 6y ago

"abuse system deletes some comments" isn't a failure mode sre concerns itself with. Buggy software can, and will get released despite whatever operational principles you have. Sre isn't about buglessness, but reliability.

stonogo 6y ago
Isn't this utterly at odds with the concept of code ownership, and how SRE doesn't take systems into production without owning them first?
- joshuamorton 6y ago
  SRE doesn't own the code. They own the system reliability. Devs are still responsible for writing the functional code. Certain aspects of the release process may fall under SRE's purview, but what the tool does is ultimately the responsibility of developers (and product). The SRE book covers testing in one chapter (17), and most of that chapter is about production testing. The question you need to ask is "is the system still within in SLOs?" If so, SRE is doing their job. If you think that resulted in a bad experience, change the SLO. But I'd ask what the heck kind of SLO would have prevented this issue?
  - londons_explore 6y ago
    "99.9 % of spammers get to see their spam posted within 300ms"