7 ms·

SRE has an identity crisis. It means too many things to too many people. In some companies, it is an operational help desk. In a few companies, SRE is rumored t

SRE has an identity crisis. It means too many things to too many people. In some companies, it is an operational help desk. In a few companies, SRE is rumored to be a different aspect of software engineering. Pay is a rough signal on what a company expects from SRE. If SREs get paid as much or more than Software Engineers, it is a good sign. Yet even that signal is not accurate. I know highly paid SREs stuck in Ops Hell. You could not pay me enough money to do a job like that. It is also not sustainable because eventually the company will fire you and replace you with a lower paid employee who can do that job just as well. I strongly oppose organizations that have SRE separate from Software Engineers. I strongly believe that SREs must be fully integrated into Software Engineering teams (aka the embedded model), with the same reporting lines as their product colleagues. Keeping SRE organizationally separate has done a great disservice to SRE. SRE 2.0 needs to tear down the mini empires that SRE 1.0 built and get back to software engineering.

dilyevsky 4y ago
Imo embedded mode is no better - usually just a token ops person that everyone on the team throws over the fence to. I’d say better signal is if sre has its own separate org structure which roots at the cto/vp of eng level. Soon as you have sres reporting to product eng managers it’s beginning of an end.
- aluminussoma 4y ago
  In the several examples of separate SRE orgs that I've seen, SRE builds their own little empires with no alignment to the product. A cynic would guess that the goal of SRE management is to hire enough SREs so there can be senior managers and directors and eventually a VP with pay packages to boot. They come to Product from the mountain-top with their holy scriptures (the Google SRE book), prescribing surface solutions to problems with deep roots (pray it away with SLOs). SRE 2.0 needs to make the embedded model work.
  - joshuamorton 4y ago
    The entire point of the unembedded model is to create the management structure that isn't strictly beholden to the whims of product. An embedded model can't work unless the company leadership wants to make it work, and product leadership won't care enough (as demonstrated by: everything pre-sre). An embedded model is designed to fail, because the incentives are misaligned. Like, what your asking is "hey, random engineer or engineering manager who is an 'SRE', make the VP of product prioritize your work and staff your team correctly!". That's not something anyone can fix, no matter how good, unless the management is willing to listen, and continue to listen and prioritize appropriately.
    - arminiusreturns 4y ago
      Greybeard sysadmin turned SRE-mask ops dude here, and this is correct. I've seen inside literally hundreds of orgs in my career, and almost always the real disconnect exists at the c-suite and board level. For a long time I blamed them for this dysfunction, but it tooks some years and maturity to start thinking more about how I can speak their language to reach their lofty, double-scheduled minds. Its a failure on both ends. This is where a good CTO/CIO with actual leadership qualities will make or break an org, as they should usually be the SRE-esque visionaries, leaders, defenders. When that doesnt happen, and lets face it, it doesnt happen very often, you get cycles of "try this other ops/admin/swe/sre structure and management method" until it falls apart and new management tries something else or they start rolling heads right under the CTO/CIO until it becomes an 1000 and 1 nights. One of the biggest warning signs of this is when tech competent middle managers (worth their weight in gold) start jumping ship enmasse.
    - donavanm 4y ago
      Isnt this a bit reductive? The other side of the fence is “How would an independent SRE org be able to fund/influence/prioritize inside the product org.” And Im pretty sure the answer there is > A dedicated model can't work unless the company leadership wants to make it work. Product reliability and performance is not something anyone can fix, no matter how good, unless the management is willing to listen, and continue to listen and prioritize appropriately. So you have the same answer, incentives and top down buy in. And if you have incentives at management layers, and top down buy in, then youve solved the root people + business problem. And now how/why do you choose between options?
- pramodbiligiri 4y ago
  A recent talk I saw suggested that SREs report into the COO (Chief Operations Officer) and not the CTO or VP of Engg. I thought that’s a good idea. The COO is interested in the stability of ongoing operations whereas the other two roles are about launching new things (Unless I have gotten the definition of COO wrong).
  - nunez 4y ago
    organizationally that's a bad idea at a lot of places because the CTO will usually have more oversight and a bigger budget to match, and SREs are closer to SWEs than traditional ops
dijit 4y ago
> SRE has an identity crisis. It means too many things to too many people. In some companies, it is an operational help desk. In a few companies, SRE is rumored to be a different aspect of software engineering. SRE is Sysadmins but with better contracts between teams, and automation focused. I haven't had much trouble with that definition, does away with all the cruft of the "Production" talk by Benjamin Treynor Sloss and the majority of the SRE books (which are a prescription of a method of creating those contracts). That said: "DevOps" and Systems Administration in general has had an identity crisis for a while. People don't like the idea of "Sysadmin" for some unknown reason, but give the same role to someone and call them "DevOps" and suddenly all is well. Same is now true of SRE. Someone on HN said to me: "DevOps is how you get Engineering to respect Systems Administration".. Maybe the same is true of SRE? I definitely feel the identity crisis of DevOps is much, much worse than that of SRE- but I guess that depends on your circles.
- kodah 4y ago
  > SRE is Sysadmins but with better contracts between teams, and automation focused. That's not even close to true, which I think is OPs point. For instance, I have an extensive SRE-SE background, but I'm also a software engineer. If I was tasked with "sysadmin" stuff, or shepherding peoples Jenkins jobs every day I'd probably lose my mind.
  - Rapzid 4y ago
    Same, background in cloud/systems engineering, SRE, and staff full stack engineer. Or, FullStack++ as I sometimes like to call it haha.
    - pojzon 4y ago
      How do you call an Engineer that can do everything beside Frontend ?
  - dijit 4y ago
    I think we misunderstand each other completely. There are many types of systems administrators; when we talk about as SRE in particular there some overlap with what I was doing as a “Linux Systems Administrator” in a Platform Engineering team, in 2009. Automating migrations, distributed job control, instrumenting the message queue system, managing cache invalidation across our entire fleet of memcache servers (which was ugly hacked with MySQL, not proud), as well as “maintenance” programs to upgrade “versions” (multi-tenant SaaS) and warm caches, dark launches, canary deployments… well you get the picture. Of course I was responsible for the reliability of the systems these things ran on, but there was a really-really strong focus on automation as part of the job, and this was when puppet was quite new, we were stuck with tools like cfengine. Coddling Jenkins sounds like build or release engineering. Which is also a separate thing. People seem to forget that systems administration has a focus on software development too- I mean for goodness sake terraform, ansible and salt all come from sysadmins; I suspect it’s because of some very foolish admins proudly exclaiming they don’t code. But they were always the bottom rung and end up being “promoted” to management at some point.
throwaway894345 4y ago
One of the functions that SRE provides in our organization is building out the platform and solving cross-cutting problems. How do you do that without some reporting structure? Shared ownership and guilds can only go so far since each team has its own priorities and incentives (and will rarely choose to yield engineering time to platform things if they can devote it to product things). I’m not arguing against an embedded model, but I do think there’s tension that’s not easily resolved.
- aluminussoma 4y ago
  In my division, there is an infrastructure team and product teams. Those platform type projects are handled by the infrastructure team. In a product only team, it will always be difficult to work on such projects. Ultimately, the answer ties back into the identity question: what is SRE?
  - throwaway894345 4y ago
    In our org, we have SRE who handle platform and SRE who are embedded on product teams, but all report up through the SRE org. One of the goals is to spread more infrastructure knowledge to the engineering teams to reduce their dependence on SRE so that SRE is mostly doing platform work and special infra-intense projects. I’m not saying this is the one true way or even a particularly good way, and I agree about the identity crisis (same as with DevOps half a decade ago).
daenz 4y ago
Former Senior SRE here. Was doing everything: working with engineering teams to build a microservice authoring framework. Putting out fires and responding to outages. Adding monitoring/metrics/logging to all the things. Building productivity tools for engineers. Spinning up new software in sandboxed environments for evaluation. Onboarding and offboarding new engineers. Coordinating downtime for software upgrades. Refactoring our infra and making it more IaC. Security audits. Our team was basically a dumping ground for anything that required experience in many different types systems and/or privileged access. On top of that, we had projects were shoe-horned into roadmaps. So while 90% of our duties were reactive to what the org needed, we also had the pressure of trying to meet deadlines for normal software engineering work. It was unhealthy and I wanted very much to not be defined as an "SRE"
- samstave 4y ago
  Director of Ops title for a decade here: Add on top of this managing multiples of you and having to do the CFOs work for the CEO and litterally manage to the penny the instances we spun up and dealing with all the expectations of ops WRT to developing tooling on top of tooling.
hiddenwaffle 4y ago
My company expects SRE to be synonymous with cloud administrator, acting as the gateway to doing anything AWS-related. They've also asked us to troubleshoot for database backup processes, write Crystal Reports on user activity, and be Slack admins. Some of the developers believe that we do manual production deployments of their builds (we don't). I get requests all the time from users asking if I can make accounts on various systems for them. I was given a programming test for this job :(
dub 4y ago
Agreed about having SRE fully integrated into feature teams in terms of day-to-day work, seating charts, team building events, etc. From a management perspective I think it still makes sense for SRE to have a different reporting structure. Feature teams generally aren't rewarded for investing in long-term code and production health. If you report to a feature team manager there will often be downward pressure to focus less on production health and more on helping features ship quickly. Giving SRE a degree of independent oversight serves as a system of checks and balances against those feature team pressures.
kodah 4y ago
> SRE has an identity crisis. It means too many things to too many people. In some companies, it is an operational help desk. In a few companies, SRE is rumored to be a different aspect of software engineering. SWE has an identity crisis. It means too many things to too many people. Just kidding, our entire industry is filled with titles that mean nothing. I agree, we should fix that.
brundolf 4y ago
> I strongly oppose organizations that have SRE separate from Software Engineers Can confirm, we have an SRE team that's totally separate from engineering and it's a huge and constant bottleneck. We've literally re-architected parts of the application to avoid having to go through SRE for things.
samstave 4y ago
I feel like you haven't heard of BOFH. Bastard Operator From Hell -- A famous IT trope. Which was typically a sysad for a mainframe/Mini-Computer... Usually on a University Campus.... Think Lazlo from Wierd Science, but an asshole. I actually worked with a BOFH. (Sadly he had a heart attack on the CalTrain Station in Mountain View, and died.) He was the AS/400 admin for a place where we manufactured all of SUNs Software (meaning made the SUNos CDs manuals packaging etc) and this guy was a BOFH Sales folk would call up to the helpdesk and request a password reset. This one time, he set the users password to "STUPID!" ... I fired him for that. He had a heart attack shortly thereafter.... :-(
SassyGrapefruit 4y ago
We use an "embedded model" with separate teams. I run an SRE organization with 10 individuals. At any given moment 3-4 are embedded on teams pursuing some objective. The rest are doing incident response and SRE specific projects from the big pile of sprint work. We did this because in our case most development problems didn't have a reliability dimension. We build mathematical models. Do I need an SRE involved in the discussion of switching from one statistical approach to another? In most cases no. But when that team deems themselves limited by their current walled garden and want to explore/operationalize new technology then we need an embedded SRE to facilitate that.

dilyevsky 4y ago

Imo embedded mode is no better - usually just a token ops person that everyone on the team throws over the fence to. I’d say better signal is if sre has its own separate org structure which roots at the cto/vp of eng level. Soon as you have sres reporting to product eng managers it’s beginning of an end.

aluminussoma 4y ago
In the several examples of separate SRE orgs that I've seen, SRE builds their own little empires with no alignment to the product. A cynic would guess that the goal of SRE management is to hire enough SREs so there can be senior managers and directors and eventually a VP with pay packages to boot. They come to Product from the mountain-top with their holy scriptures (the Google SRE book), prescribing surface solutions to problems with deep roots (pray it away with SLOs). SRE 2.0 needs to make the embedded model work.
- joshuamorton 4y ago
  The entire point of the unembedded model is to create the management structure that isn't strictly beholden to the whims of product. An embedded model can't work unless the company leadership wants to make it work, and product leadership won't care enough (as demonstrated by: everything pre-sre). An embedded model is designed to fail, because the incentives are misaligned. Like, what your asking is "hey, random engineer or engineering manager who is an 'SRE', make the VP of product prioritize your work and staff your team correctly!". That's not something anyone can fix, no matter how good, unless the management is willing to listen, and continue to listen and prioritize appropriately.
  - arminiusreturns 4y ago
    Greybeard sysadmin turned SRE-mask ops dude here, and this is correct. I've seen inside literally hundreds of orgs in my career, and almost always the real disconnect exists at the c-suite and board level. For a long time I blamed them for this dysfunction, but it tooks some years and maturity to start thinking more about how I can speak their language to reach their lofty, double-scheduled minds. Its a failure on both ends. This is where a good CTO/CIO with actual leadership qualities will make or break an org, as they should usually be the SRE-esque visionaries, leaders, defenders. When that doesnt happen, and lets face it, it doesnt happen very often, you get cycles of "try this other ops/admin/swe/sre structure and management method" until it falls apart and new management tries something else or they start rolling heads right under the CTO/CIO until it becomes an 1000 and 1 nights. One of the biggest warning signs of this is when tech competent middle managers (worth their weight in gold) start jumping ship enmasse.
  - donavanm 4y ago
    Isnt this a bit reductive? The other side of the fence is “How would an independent SRE org be able to fund/influence/prioritize inside the product org.” And Im pretty sure the answer there is > A dedicated model can't work unless the company leadership wants to make it work. Product reliability and performance is not something anyone can fix, no matter how good, unless the management is willing to listen, and continue to listen and prioritize appropriately. So you have the same answer, incentives and top down buy in. And if you have incentives at management layers, and top down buy in, then youve solved the root people + business problem. And now how/why do you choose between options?
pramodbiligiri 4y ago
A recent talk I saw suggested that SREs report into the COO (Chief Operations Officer) and not the CTO or VP of Engg. I thought that’s a good idea. The COO is interested in the stability of ongoing operations whereas the other two roles are about launching new things (Unless I have gotten the definition of COO wrong).
- nunez 4y ago
  organizationally that's a bad idea at a lot of places because the CTO will usually have more oversight and a bigger budget to match, and SREs are closer to SWEs than traditional ops

dijit 4y ago

> SRE has an identity crisis. It means too many things to too many people. In some companies, it is an operational help desk. In a few companies, SRE is rumored to be a different aspect of software engineering. SRE is Sysadmins but with better contracts between teams, and automation focused. I haven't had much trouble with that definition, does away with all the cruft of the "Production" talk by Benjamin Treynor Sloss and the majority of the SRE books (which are a prescription of a method of creating those contracts). That said: "DevOps" and Systems Administration in general has had an identity crisis for a while. People don't like the idea of "Sysadmin" for some unknown reason, but give the same role to someone and call them "DevOps" and suddenly all is well. Same is now true of SRE. Someone on HN said to me: "DevOps is how you get Engineering to respect Systems Administration".. Maybe the same is true of SRE? I definitely feel the identity crisis of DevOps is much, much worse than that of SRE- but I guess that depends on your circles.

kodah 4y ago
> SRE is Sysadmins but with better contracts between teams, and automation focused. That's not even close to true, which I think is OPs point. For instance, I have an extensive SRE-SE background, but I'm also a software engineer. If I was tasked with "sysadmin" stuff, or shepherding peoples Jenkins jobs every day I'd probably lose my mind.
- Rapzid 4y ago
  Same, background in cloud/systems engineering, SRE, and staff full stack engineer. Or, FullStack++ as I sometimes like to call it haha.
  - pojzon 4y ago
    How do you call an Engineer that can do everything beside Frontend ?
- dijit 4y ago
  I think we misunderstand each other completely. There are many types of systems administrators; when we talk about as SRE in particular there some overlap with what I was doing as a “Linux Systems Administrator” in a Platform Engineering team, in 2009. Automating migrations, distributed job control, instrumenting the message queue system, managing cache invalidation across our entire fleet of memcache servers (which was ugly hacked with MySQL, not proud), as well as “maintenance” programs to upgrade “versions” (multi-tenant SaaS) and warm caches, dark launches, canary deployments… well you get the picture. Of course I was responsible for the reliability of the systems these things ran on, but there was a really-really strong focus on automation as part of the job, and this was when puppet was quite new, we were stuck with tools like cfengine. Coddling Jenkins sounds like build or release engineering. Which is also a separate thing. People seem to forget that systems administration has a focus on software development too- I mean for goodness sake terraform, ansible and salt all come from sysadmins; I suspect it’s because of some very foolish admins proudly exclaiming they don’t code. But they were always the bottom rung and end up being “promoted” to management at some point.

throwaway894345 4y ago

One of the functions that SRE provides in our organization is building out the platform and solving cross-cutting problems. How do you do that without some reporting structure? Shared ownership and guilds can only go so far since each team has its own priorities and incentives (and will rarely choose to yield engineering time to platform things if they can devote it to product things). I’m not arguing against an embedded model, but I do think there’s tension that’s not easily resolved.

aluminussoma 4y ago
In my division, there is an infrastructure team and product teams. Those platform type projects are handled by the infrastructure team. In a product only team, it will always be difficult to work on such projects. Ultimately, the answer ties back into the identity question: what is SRE?
- throwaway894345 4y ago
  In our org, we have SRE who handle platform and SRE who are embedded on product teams, but all report up through the SRE org. One of the goals is to spread more infrastructure knowledge to the engineering teams to reduce their dependence on SRE so that SRE is mostly doing platform work and special infra-intense projects. I’m not saying this is the one true way or even a particularly good way, and I agree about the identity crisis (same as with DevOps half a decade ago).

daenz 4y ago

Former Senior SRE here. Was doing everything: working with engineering teams to build a microservice authoring framework. Putting out fires and responding to outages. Adding monitoring/metrics/logging to all the things. Building productivity tools for engineers. Spinning up new software in sandboxed environments for evaluation. Onboarding and offboarding new engineers. Coordinating downtime for software upgrades. Refactoring our infra and making it more IaC. Security audits. Our team was basically a dumping ground for anything that required experience in many different types systems and/or privileged access. On top of that, we had projects were shoe-horned into roadmaps. So while 90% of our duties were reactive to what the org needed, we also had the pressure of trying to meet deadlines for normal software engineering work. It was unhealthy and I wanted very much to not be defined as an "SRE"

samstave 4y ago
Director of Ops title for a decade here: Add on top of this managing multiples of you and having to do the CFOs work for the CEO and litterally manage to the penny the instances we spun up and dealing with all the expectations of ops WRT to developing tooling on top of tooling.

hiddenwaffle 4y ago

My company expects SRE to be synonymous with cloud administrator, acting as the gateway to doing anything AWS-related. They've also asked us to troubleshoot for database backup processes, write Crystal Reports on user activity, and be Slack admins. Some of the developers believe that we do manual production deployments of their builds (we don't). I get requests all the time from users asking if I can make accounts on various systems for them. I was given a programming test for this job :(

dub 4y ago

Agreed about having SRE fully integrated into feature teams in terms of day-to-day work, seating charts, team building events, etc. From a management perspective I think it still makes sense for SRE to have a different reporting structure. Feature teams generally aren't rewarded for investing in long-term code and production health. If you report to a feature team manager there will often be downward pressure to focus less on production health and more on helping features ship quickly. Giving SRE a degree of independent oversight serves as a system of checks and balances against those feature team pressures.

kodah 4y ago

> SRE has an identity crisis. It means too many things to too many people. In some companies, it is an operational help desk. In a few companies, SRE is rumored to be a different aspect of software engineering. SWE has an identity crisis. It means too many things to too many people. Just kidding, our entire industry is filled with titles that mean nothing. I agree, we should fix that.

brundolf 4y ago

> I strongly oppose organizations that have SRE separate from Software Engineers Can confirm, we have an SRE team that's totally separate from engineering and it's a huge and constant bottleneck. We've literally re-architected parts of the application to avoid having to go through SRE for things.

samstave 4y ago

I feel like you haven't heard of BOFH. Bastard Operator From Hell -- A famous IT trope. Which was typically a sysad for a mainframe/Mini-Computer... Usually on a University Campus.... Think Lazlo from Wierd Science, but an asshole. I actually worked with a BOFH. (Sadly he had a heart attack on the CalTrain Station in Mountain View, and died.) He was the AS/400 admin for a place where we manufactured all of SUNs Software (meaning made the SUNos CDs manuals packaging etc) and this guy was a BOFH Sales folk would call up to the helpdesk and request a password reset. This one time, he set the users password to "STUPID!" ... I fired him for that. He had a heart attack shortly thereafter.... :-(

SassyGrapefruit 4y ago

We use an "embedded model" with separate teams. I run an SRE organization with 10 individuals. At any given moment 3-4 are embedded on teams pursuing some objective. The rest are doing incident response and SRE specific projects from the big pile of sprint work. We did this because in our case most development problems didn't have a reliability dimension. We build mathematical models. Do I need an SRE involved in the discussion of switching from one statistical approach to another? In most cases no. But when that team deems themselves limited by their current walled garden and want to explore/operationalize new technology then we need an embedded SRE to facilitate that.