4 ms·
Most of my work for SRE was the opposite; I did things manually because the automated systems were guaranteed to mess up some fraction of things. At some point
by dekhn 4y ago
Most of my work for SRE was the opposite; I did things manually because the automated systems were guaranteed to mess up some fraction of things. At some point my managers wanted me to automate a hardware management process- I checked and it would take 6 months to deploy the code to prod. Instead, I identified all the broken machines and filed tickets manually- getting things fixed far more quickly without a high rate of false positives and churn (google's hardware repair system churns a lot).
Many of the automated systems at Google were developed by geniuses. Others, not so much, and it ended up making a lot of work for other people.
- jeffbee 4y agoI was also on the side of hands-on operations in SRE and it earned me no friends in that org to be sure. But I like to think that point of view is still basically correct. The "annealing" people have been working on their wacky automaton for more than a decade now and a critical reading of their publication reveals that it still fundamentally doesn't work. https://www.usenix.org/publications/loginonline/prodspec-and-annealing-intent-based-actuation-google-production https://www.usenix.org/publications/loginonline/prodspec-and...
- dekhn 4y agoThose are exactly the kinds of systems I had to deal with constantly breaking things. That said, I always made friends with SRE (was one for a while) because they own the memes of production.