5 ms·
Been a few years since I worked at Google as an SRE, but I did not find "There’s no incentive in big tech to write software with no bugs" to be particularly tru
by smcameron 2y ago
Been a few years since I worked at Google as an SRE, but I did not find "There’s no incentive in big tech to write software with no bugs" to be particularly true. Perhaps because I was an SRE for a few of the of the older, intrinsic, core products. A lot of the pages we got were for things outside our control (e.g. some fiber optic cables got broken (we found out later), so we've got to drain some cluster because nothing's getting through there, or something's getting overloaded because it's the superbowl today, so throw more machines/memory at it, or other weird external things.) I don't remember a lot of pages due to outright bugs in the code... though it's been enough years now that I might have forgotten.
- nijave 2y agoWere the systems designed to scale based on load and handle transient failures? It seems like lack of automated remediation would be a bug unless it's an "accepted business risk" i.e. cheaper to throw people at to manually fix than build a software solution.
- Xylakant 2y agoYou can’t plan for all failure modes. Weird shit happens and it needs human intervention to figure out what went wrong. Sometimes someone needs to assess what path forward is the lowest (financial) harm and weigh the options. No computer should make that call.
- VirusNewbie 2y agoIf a bug is causing multiple pages, SRE will absolutely either force the devs to fix it, or fix the bug themselves.