6 ms·

Re SREs: we've taken a different approach. Devs own their systems. If the issue is network or hardware, there are more ops oriented folk that make that work. Bu

Re SREs: we've taken a different approach. Devs own their systems. If the issue is network or hardware, there are more ops oriented folk that make that work. But when it comes to services, teams own their stuff. Bug a 2am causes a service to go down? The team is expected to start debugging it and pull people in as needed. The result? Shifting to this model from an SRE firefighting model has lowered total alerts, incidents, and outages. When we first shifted, people dreaded going on their on call rotation for their team; you expected to be paged multiple times a night. Fast forward and now I can count on one hand the amount of pages I've had on my last multiple on call rotations. Work is prioritized for system reliability because we feel the pain.

fatnoah 7y ago
>The result? Shifting to this model from an SRE firefighting model has lowered total alerts, incidents, and outages. When we first shifted, people dreaded going on their on call rotation for their team; you expected to be paged multiple times a night. I implemented a similar change at a previous company. Prior to the change, the operations team fronted all on-call issues. After, anything related to the software/services was handled by software teams. We went from nightly alerts to less than monthly. As a bonus, time to diagnose and repair went waayyyyy down due to much better logging. Note, better logging <> more logging in this case. It meant less spurious errors/warnings, more useful INFO, and better observability of the system.
test1235 7y ago
Did everyone get paid more to take on these responsibilities? I feel we're about to go through exactly the same thing at my company.
- sethammons 7y ago
  The logic was that we are highly paid developers and are expected to build robust, reliable software with high availability. Typically a brutal on call night yields untracked time off.
- sokoloff 7y ago
  We’re many years down the road, so we’ve mostly baked it into job descriptions and expectations at this point. In some geographies, we’ve had to craft those responsibilities and pay/comp time philosophies in specific ways to comply with local laws. In others where the law is silent, we’ve taken a laxer “if you got crushed by off-hours work, take appropriate comp time”. During the old days of “you commit code and ops runs it” and during transition to “you build and run it”, we did have some bonuses for being on-call or doing stints in problem management. Now, we bake that expectation into base salary except in locales where we have to split it out to comply with local laws. Along the way, we've given enough merit increases, market adjustments, and promotions that it's impossible to say whether we had a bump specifically because of the industry's new philosophy on ops. If you wanted to make the case that we did, you could. If you wanted to make the case that we didn't, you equally could.
- POiNTx 7y ago
  At my previous company, on call was an opt-in responsibility that came with extra pay. We had enough developers so not every developer had to do on call.
afthonos 7y ago
This addresses only 50% of what SREs contribute. The other 50% is designing the production infrastructure, processes, and tools that facilitate efficient response. Depending on the size of your company, you can of course make do with off-the-shelf solutions, but that’s not scalable part a certain point; creating robust production systems (as opposed to good code) is a specialty like any other.
- halbritt 7y ago
  I've seen this service ownership model fail when implemented across a dozen service teams. Each team was a snowflake in that it's production infrastructure was unique, and not necessarily reliable. It also assumes that designing production infrastructure or even systems level thinking is a skill that most software developers have. Certainly, enough do that it's common, but it's not universal. In this particular case, the org standardized on terraform. Some developers welcomed the ability to self-service. Others wanted nothing to do with it.
devonkim 7y ago
I’ve seen this shared responsibility model only work when feature development is traded off with reliability as a business level guidance on top of empowerment of developers to own their systems fully. Oftentimes by the time things get bad with reliability I’ve rarely seen a business get out of the death spiral of bad technical debts being paid out with just interest daily and burning everyone out while feature requests flood in and customers leave unless there is massive funding (and subsequent hiring with a solid plan for enabling these engineers) growth. Only ever seen it possible to turn around with empowered engineers that have the cycles to fix crippling systemic defects and the business has the means to support it. Otherwise it’s basically a Sisyphean task to maintain things and the business should have offshored the labor by then to save on costs if the cost-feature curve isn’t working out in growth stages.
twoquestions 7y ago
This assumes a level of control and autonomy that not many at all have. Downtime is valuable currency in intra-corporate negotiations, and the ones to answer the phone at 3am are business inputs, not people.
- huherto 7y ago
  That is the whole purpose of doing microservices. Every team can operate independently. Choose their tools, processes, etc. As long as they respect their API Contracts.
  - cuddlecake 7y ago
    I think it's a misinterpretation of microservices to say that the purpose is that the teams can choose their tools and processes. Microservices merely ensure that the complexity of using different tools per microservice does not lead to an increase of maintenance burden. Nonetheless, you still have a maintenance burden if every microservice is built upon different tools and processes, since you cannot address patterns of problems easily that occur within combinations of patterns and processes. The cardinality just increases with every tool and process added into the mix. This burden of maintenance automatically takes its toll on productivity. Teams will not be able to create, maintain or iterate on code that produces business value (as opposed to managing the platform of microservices itself) I think there is value in a well defined set of processes and tools because there will eventually be platform concerns that become increasingly difficult otherwise. I do not have insight in how most companies with successful microservice architectures achieved their success, but I would bet my life that a majority of those companies do not let their engineering teams use tools and processes arbitrarily, unless it serves to REALLY produce value that would be unachievable by the tools and processes used up until that point (for example, because performance is usually sufficient, but a specific service suddenly needs to outperform everything that came so far, so you use C, Go or Rust instead of Python) I am biased into thinking this because I suspect that most of these companies did either start with monolithic architecture (i.e. supporting multiple languages was impossible or just not feasible enough) or they started with a microservice approach focusing on producing value as quickly as possible after an initial ramp up time. Supporting many different tools and processes from the get go would make the ramp up time longer than necessary. TL;DR Microservice platform maintenance suffers if tools and processes are chosen freely for each microservice, without any push towards unification. Oof I need to sleep. Don't mind me, just trying to sort myself out.
calciphus 7y ago
This answer always downplays what "on call" really means. Even if you only get paged once a year, if you're "on call" and the expectation is that within X minutes you are at a computer working, this affects the life you live. You can't go camping, out to a nice dinner, on vacation, etc without lugging a laptop, internet connection, etc with you. I've heard managers often tout that "if your code is stable you'll never get paged". But that hides the fact that you're still on the hook to be available. So either you start shirking your responsibility or you reshape your life around some percent of non-working hours being owned by the company still.
- rednerrus 7y ago
  Or you put your laptop in the trunk of your car when you're on call and have tethering on your phone. Expecting another group to maintain your code seems an awful lot like throwing it over the wall.
  - matlock 7y ago
    If thats priced into the contract and made clear from the beginning sure. But it doesn't come for free.
  - afthonos 7y ago
    This is why SRE organizations typically have stringent requirements for stability before accepting a rotation, and will give it back if reliability drops below a certain level. SREs have global responsibility to the system and have a correspondingly global view. If your system broke down because of a bug you wrote, that’s on you. If a system three times removed from yours that you didn’t know depended on you broke because you changed a non-API behavior, that’s where SREs shine: they know how to quickly isolate the problem, roll back the necessary systems, and define how to avoid similar problems in the future.
  - lazyasciiart 7y ago
    And you keep your phone on during a film, and at the nice dinner, and during your daughter's wedding, and never go camping without phone service. That is exactly what is being described by "This answer always downplays what "on call" really means."
rednerrus 7y ago
What happens when teams change? What happens when a team no longer owns a service or it's in a state where no one is actively working on it? I think you need dedicated resources to thinking about the picture as a whole.
- sethammons 7y ago
  For sure. Unlike most developers these days (or so it seems), I've been at my company for nearly a decade. Started out as engineer 15 in a company that is now a couple thousand. I've seen services transfer teams, get forgotten about, be rewritten, and just about every permutation in between. The way we mitigate this is by tracking the service ownership by team and audit repos from time to time. Every running service is required to have a runbook/playbook that should be thorough enough that a new team member could fix something broken. The biggest problem I've seen is some teams being afraid of really digging into a legacy service if it is transferred to them.
CodeMage 7y ago
I'm curious: how many people did you lose because of the shift?

fatnoah 7y ago

>The result? Shifting to this model from an SRE firefighting model has lowered total alerts, incidents, and outages. When we first shifted, people dreaded going on their on call rotation for their team; you expected to be paged multiple times a night. I implemented a similar change at a previous company. Prior to the change, the operations team fronted all on-call issues. After, anything related to the software/services was handled by software teams. We went from nightly alerts to less than monthly. As a bonus, time to diagnose and repair went waayyyyy down due to much better logging. Note, better logging <> more logging in this case. It meant less spurious errors/warnings, more useful INFO, and better observability of the system.

test1235 7y ago

Did everyone get paid more to take on these responsibilities? I feel we're about to go through exactly the same thing at my company.

sethammons 7y ago
The logic was that we are highly paid developers and are expected to build robust, reliable software with high availability. Typically a brutal on call night yields untracked time off.
sokoloff 7y ago
We’re many years down the road, so we’ve mostly baked it into job descriptions and expectations at this point. In some geographies, we’ve had to craft those responsibilities and pay/comp time philosophies in specific ways to comply with local laws. In others where the law is silent, we’ve taken a laxer “if you got crushed by off-hours work, take appropriate comp time”. During the old days of “you commit code and ops runs it” and during transition to “you build and run it”, we did have some bonuses for being on-call or doing stints in problem management. Now, we bake that expectation into base salary except in locales where we have to split it out to comply with local laws. Along the way, we've given enough merit increases, market adjustments, and promotions that it's impossible to say whether we had a bump specifically because of the industry's new philosophy on ops. If you wanted to make the case that we did, you could. If you wanted to make the case that we didn't, you equally could.
POiNTx 7y ago
At my previous company, on call was an opt-in responsibility that came with extra pay. We had enough developers so not every developer had to do on call.

afthonos 7y ago

This addresses only 50% of what SREs contribute. The other 50% is designing the production infrastructure, processes, and tools that facilitate efficient response. Depending on the size of your company, you can of course make do with off-the-shelf solutions, but that’s not scalable part a certain point; creating robust production systems (as opposed to good code) is a specialty like any other.

halbritt 7y ago
I've seen this service ownership model fail when implemented across a dozen service teams. Each team was a snowflake in that it's production infrastructure was unique, and not necessarily reliable. It also assumes that designing production infrastructure or even systems level thinking is a skill that most software developers have. Certainly, enough do that it's common, but it's not universal. In this particular case, the org standardized on terraform. Some developers welcomed the ability to self-service. Others wanted nothing to do with it.

devonkim 7y ago

I’ve seen this shared responsibility model only work when feature development is traded off with reliability as a business level guidance on top of empowerment of developers to own their systems fully. Oftentimes by the time things get bad with reliability I’ve rarely seen a business get out of the death spiral of bad technical debts being paid out with just interest daily and burning everyone out while feature requests flood in and customers leave unless there is massive funding (and subsequent hiring with a solid plan for enabling these engineers) growth. Only ever seen it possible to turn around with empowered engineers that have the cycles to fix crippling systemic defects and the business has the means to support it. Otherwise it’s basically a Sisyphean task to maintain things and the business should have offshored the labor by then to save on costs if the cost-feature curve isn’t working out in growth stages.

twoquestions 7y ago

This assumes a level of control and autonomy that not many at all have. Downtime is valuable currency in intra-corporate negotiations, and the ones to answer the phone at 3am are business inputs, not people.

huherto 7y ago
That is the whole purpose of doing microservices. Every team can operate independently. Choose their tools, processes, etc. As long as they respect their API Contracts.
- cuddlecake 7y ago
  I think it's a misinterpretation of microservices to say that the purpose is that the teams can choose their tools and processes. Microservices merely ensure that the complexity of using different tools per microservice does not lead to an increase of maintenance burden. Nonetheless, you still have a maintenance burden if every microservice is built upon different tools and processes, since you cannot address patterns of problems easily that occur within combinations of patterns and processes. The cardinality just increases with every tool and process added into the mix. This burden of maintenance automatically takes its toll on productivity. Teams will not be able to create, maintain or iterate on code that produces business value (as opposed to managing the platform of microservices itself) I think there is value in a well defined set of processes and tools because there will eventually be platform concerns that become increasingly difficult otherwise. I do not have insight in how most companies with successful microservice architectures achieved their success, but I would bet my life that a majority of those companies do not let their engineering teams use tools and processes arbitrarily, unless it serves to REALLY produce value that would be unachievable by the tools and processes used up until that point (for example, because performance is usually sufficient, but a specific service suddenly needs to outperform everything that came so far, so you use C, Go or Rust instead of Python) I am biased into thinking this because I suspect that most of these companies did either start with monolithic architecture (i.e. supporting multiple languages was impossible or just not feasible enough) or they started with a microservice approach focusing on producing value as quickly as possible after an initial ramp up time. Supporting many different tools and processes from the get go would make the ramp up time longer than necessary. TL;DR Microservice platform maintenance suffers if tools and processes are chosen freely for each microservice, without any push towards unification. Oof I need to sleep. Don't mind me, just trying to sort myself out.

calciphus 7y ago

This answer always downplays what "on call" really means. Even if you only get paged once a year, if you're "on call" and the expectation is that within X minutes you are at a computer working, this affects the life you live. You can't go camping, out to a nice dinner, on vacation, etc without lugging a laptop, internet connection, etc with you. I've heard managers often tout that "if your code is stable you'll never get paged". But that hides the fact that you're still on the hook to be available. So either you start shirking your responsibility or you reshape your life around some percent of non-working hours being owned by the company still.

rednerrus 7y ago
Or you put your laptop in the trunk of your car when you're on call and have tethering on your phone. Expecting another group to maintain your code seems an awful lot like throwing it over the wall.
- matlock 7y ago
  If thats priced into the contract and made clear from the beginning sure. But it doesn't come for free.
- afthonos 7y ago
  This is why SRE organizations typically have stringent requirements for stability before accepting a rotation, and will give it back if reliability drops below a certain level. SREs have global responsibility to the system and have a correspondingly global view. If your system broke down because of a bug you wrote, that’s on you. If a system three times removed from yours that you didn’t know depended on you broke because you changed a non-API behavior, that’s where SREs shine: they know how to quickly isolate the problem, roll back the necessary systems, and define how to avoid similar problems in the future.
- lazyasciiart 7y ago
  And you keep your phone on during a film, and at the nice dinner, and during your daughter's wedding, and never go camping without phone service. That is exactly what is being described by "This answer always downplays what "on call" really means."

rednerrus 7y ago

What happens when teams change? What happens when a team no longer owns a service or it's in a state where no one is actively working on it? I think you need dedicated resources to thinking about the picture as a whole.

sethammons 7y ago
For sure. Unlike most developers these days (or so it seems), I've been at my company for nearly a decade. Started out as engineer 15 in a company that is now a couple thousand. I've seen services transfer teams, get forgotten about, be rewritten, and just about every permutation in between. The way we mitigate this is by tracking the service ownership by team and audit repos from time to time. Every running service is required to have a runbook/playbook that should be thorough enough that a new team member could fix something broken. The biggest problem I've seen is some teams being afraid of really digging into a legacy service if it is transferred to them.

CodeMage 7y ago

I'm curious: how many people did you lose because of the shift?