4 ms·

Build a culture of fixing the alerts and pages. Having the people who write the software answer the pages builds in a closed feedback loop, incentivizing reduci

Build a culture of fixing the alerts and pages. Having the people who write the software answer the pages builds in a closed feedback loop, incentivizing reducing the operational burden. If an external Ops group takes the burden of responding to pages, there is no feedback loop nor incentive for those writing the software to reduce the burden!

m0zg 7y ago
>> Having the people who write the software answer the pages builds This is the Stockholm syndrome you often see in ex-AMZN people. Having devs burn themselves out within 2 years in insane oncall rotations is just not very smart. Nor is it the only way to do it. >> there is no feedback loop Not if "external Ops group" can refuse to support your shit if it sucks. That's how Google works: your service has to pass PRR (production readiness review) by SRE before SREs will agree to support it. If your service begins to deteriorate over time, SREs can dump it right back in your lap to fix, and require another PRR before they'll support it again. And you don't hand the service over to SREs, there are still devs oncall, but the load is much ligher, and dare I say, SREs are much better at running infrastructure (and building tools to help run it reliably) because it's their job. They're also in multiple geographic zones, so at night an SRE in e.g. Zurich can take care of simple issues (or escalate to a SWE if it's something gnarly). And I don't know how it is now, but SWEs used to get extra pay while they are oncall.
- muttled 7y ago
  When they dump it back in your lap, does that mean you're now on-call for your own service again? I was just curious how that looks, do you get an email saying you're back on the hook for a service because it's broken too many times?
- sgt101 7y ago
  I've worked in an environment that was similar, but the difference was that the support folks haven't got enough power, or technical knowledge, to effectively manage the interface. I think that ops groups are the way to go, and devops is not an enterprise or long term ready prospect.
michaelt 7y ago
Works great if you create both the feedback loop and the culture of fixing the alerts. But I've had friends work at companies that said "That monthly 3am page costs us 1 hour of overtime, fixing it will take 80 hours, that's a weak business case so the fix is a low priority" - i.e. the feedback loop without the culture of fixing the alerts. Of course, a sane company would account for employee retention costs in that business case, and wouldn't let their codebase get into a bad enough state that fixing bugs was infeasible. But when you're at the job interview stage, companies aren't likely to be upfront about such things.

m0zg 7y ago

>> Having the people who write the software answer the pages builds This is the Stockholm syndrome you often see in ex-AMZN people. Having devs burn themselves out within 2 years in insane oncall rotations is just not very smart. Nor is it the only way to do it. >> there is no feedback loop Not if "external Ops group" can refuse to support your shit if it sucks. That's how Google works: your service has to pass PRR (production readiness review) by SRE before SREs will agree to support it. If your service begins to deteriorate over time, SREs can dump it right back in your lap to fix, and require another PRR before they'll support it again. And you don't hand the service over to SREs, there are still devs oncall, but the load is much ligher, and dare I say, SREs are much better at running infrastructure (and building tools to help run it reliably) because it's their job. They're also in multiple geographic zones, so at night an SRE in e.g. Zurich can take care of simple issues (or escalate to a SWE if it's something gnarly). And I don't know how it is now, but SWEs used to get extra pay while they are oncall.

muttled 7y ago
When they dump it back in your lap, does that mean you're now on-call for your own service again? I was just curious how that looks, do you get an email saying you're back on the hook for a service because it's broken too many times?
sgt101 7y ago
I've worked in an environment that was similar, but the difference was that the support folks haven't got enough power, or technical knowledge, to effectively manage the interface. I think that ops groups are the way to go, and devops is not an enterprise or long term ready prospect.

michaelt 7y ago

Works great if you create both the feedback loop and the culture of fixing the alerts. But I've had friends work at companies that said "That monthly 3am page costs us 1 hour of overtime, fixing it will take 80 hours, that's a weak business case so the fix is a low priority" - i.e. the feedback loop without the culture of fixing the alerts. Of course, a sane company would account for employee retention costs in that business case, and wouldn't let their codebase get into a bad enough state that fixing bugs was infeasible. But when you're at the job interview stage, companies aren't likely to be upfront about such things.