4 ms·

Thanks for that perspective. Do they ever find time to do side-projects? It sounds like these people work 24/7.

• dekhn 12y ago
  Absolutely. They're busy but work on side projects. Depends on the person.
  • blutoot 12y ago
    Do they usually pursue side projects that help them get better at their troubleshooting/automation skills or do they tend to keep them complementary to avoid burn-out?
    • dekhn 12y ago
      yes
• eropple 12y ago
  I'm a devops engineer (I guess we're called SREs now too) at Localytics. We're not Google-scale, but our stuff isn't small, and a big focus of what we're doing is making sure we don't have to work 24/7. Automation, automation, automation. Handle failure states automatically and only inform humans after the system has fixed itself or if the system can't fix itself. It's pretty fun.
  • blutoot 12y ago
    How do you control the quality of that automation code? Doesn't that add an overhead to your operations?
    • dekhn 12y ago
      Good testing. I got hired as a Test Engineer in SRE at Google to write automation code tests. Good coverage from unit tests, good integration tests, and good system tests (many of them using fake systems with scripted failure modes). Then finally, a fair amount of testing on "real" machines as new code gets integrated. Then make sure you have great monitoring because it will still fail at runtime and you're the poor sap on call who has to deal with it.