5 ms·

What I have seen in the companies that I have worked in - Initially every engineer does everything - Growth gives birth to independent SRE teams. Now we have

What I have seen in the companies that I have worked in - Initially every engineer does everything - Growth gives birth to independent SRE teams. Now we have engineers who does only SRE - We realised SRE is bottle neck for product teams. SRE teams are removed and all SRE are embedded into the product teams to do dedicated work for them. - Other team members are now required to learn SRE stuff, especially now that terraform etc are there so we are back to everyone is a SRE note that FAANG level companies go back to independent infra, SRE etc maybe because of niche and custom softwares they have.

p_l 6y ago
I suspect the last one is the size of workforce, and even then "embedding in product teams" seems a common mantra mentioned from FAANG companies.
andriosr 6y ago
I have a very similar view. But I see more and more companies trying to build a Google-like SRE structure, where the team actually operates more mature services highly critical to the business. Tricky not to become traditional ops teams, but seems doable. SRE becoming bottleneck with scale is something I see a lot. Usually regular product engineers that are more familiar with infra start doing it full-time for all the company.
Moto7451 6y ago
This matches my experience at my work. I am an early employee and we did one of everything back then, was told we had dedicated Ops/SRE and lost permissions to do stuff about half way through my tenure, and now this year it’s back to everyone is doing their own Ops/SRE. That’s been over the course of about a decade.
- C1sc0cat 6y ago
  cough back in the 80's I was employed by a division of British Telecom explicitly partially as DEV/OP on a billing system as I had SYSAD experience on the platform. Later on mid 90's the BT World wide intra net was run by a full service team that handled hosting admin and development
vsareto 6y ago
It's great to know that we still have no idea what we're all doing in tech.
draw_down 6y ago
> We realised SRE is bottle neck for product teams It isn’t clear what to do at this point. Embedding is one way, I don’t prefer it personally. My current employer keeps the teams separate but they expose platforms that other teams consume. Teams have a runner who’s responsible for helping other teams and responding to timely issues. Run is a rotation, not a permanent role. I think this is a good way, but it might require a bigger size company/tech org than “non FAANG” implies.
codingdave 6y ago
That is not necessarily an unhealthy cycle... Organizations succeed by delivering customer value. Re-orgs of teams to remove pain points and blockers is a sign of an organization flexible enough to handle their growth. Even if you go back and forth a few times between similar states, the larger point is that you are willing to do what is needed to continue to effectively deliver value to your customers.
- vsareto 6y ago
  >the larger point is that you are willing to do what is needed They seem to be unable to organize and delegate work effectively even when it's needed and instead rely on polyglot-engineers to bridge that gap by overloading their responsibilities.
- tupputuppu 6y ago
  Going back and forth isn't progress or keeping up with growth.
saalweachter 6y ago
Is it, uhh, common knowledge that the important thing isn't how the team is organized or how they exactly do their job but that there are engineers paid and incentivized and empowered to keep the status quo running flawlessly (or as close there to as possible) rather than being paid and incentivized and empowered to launch cool new features or make the money graphs go up? The problem at most companies is that if you have two engineers, Alice and Betsy, and Alice launches a change that generated marketing buzz that gets spun into sales, and Betsy launches a change that prevented the site from going down over the seasonal rush, Alice gets promoted and Betsy gets told to work on more impactful projects. Which isn't to say that Alice's work isn't important or promotion worthy, but you need an incentive structure that rewards both.
- mjayhn 6y ago
  In my experience, Alice moves on to a cool new project to greenfield while Betsy gets stuck supporting Alices creation until they eventually get on-call burnout and quit. Alice checks in only when there's enough noise made either by angry customers, cto or angry SREs. That's my 10yrs+ ops experience, at least, and this is hugely exacerbated if Alice is put on such a high pedestal that they don't even have to be on-call with Betty at any point of the lifecycle.
  - CoolGuySteve 6y ago
    I think ultimately this dynamic is why Google can't keep most products around for longer than 5 years.
gowld 6y ago
> We realised SRE is bottle neck for product teams. This means that management decided velocity is more important than reliability?
- dragonwriter 6y ago
  > > We realised SRE is bottle neck for product teams. > This means that management decided velocity is more important than reliability? No, given the organizational response ("SRE teams are removed and all SRE are embedded into the product teams to do dedicated work for them.") It seems to be like the motivation for Agile cross-functional teams instead of separate Analysis, Design, Coding, Testing, ... teams, and like DevOps, and like DevSecOps, etc., yet another instance of realizing that throwing products over internal walls in organizational handoffs produces inefficiencies.

p_l 6y ago

I suspect the last one is the size of workforce, and even then "embedding in product teams" seems a common mantra mentioned from FAANG companies.

andriosr 6y ago

I have a very similar view. But I see more and more companies trying to build a Google-like SRE structure, where the team actually operates more mature services highly critical to the business. Tricky not to become traditional ops teams, but seems doable. SRE becoming bottleneck with scale is something I see a lot. Usually regular product engineers that are more familiar with infra start doing it full-time for all the company.

Moto7451 6y ago

This matches my experience at my work. I am an early employee and we did one of everything back then, was told we had dedicated Ops/SRE and lost permissions to do stuff about half way through my tenure, and now this year it’s back to everyone is doing their own Ops/SRE. That’s been over the course of about a decade.

C1sc0cat 6y ago
cough back in the 80's I was employed by a division of British Telecom explicitly partially as DEV/OP on a billing system as I had SYSAD experience on the platform. Later on mid 90's the BT World wide intra net was run by a full service team that handled hosting admin and development

vsareto 6y ago

It's great to know that we still have no idea what we're all doing in tech.

draw_down 6y ago

> We realised SRE is bottle neck for product teams It isn’t clear what to do at this point. Embedding is one way, I don’t prefer it personally. My current employer keeps the teams separate but they expose platforms that other teams consume. Teams have a runner who’s responsible for helping other teams and responding to timely issues. Run is a rotation, not a permanent role. I think this is a good way, but it might require a bigger size company/tech org than “non FAANG” implies.

codingdave 6y ago

That is not necessarily an unhealthy cycle... Organizations succeed by delivering customer value. Re-orgs of teams to remove pain points and blockers is a sign of an organization flexible enough to handle their growth. Even if you go back and forth a few times between similar states, the larger point is that you are willing to do what is needed to continue to effectively deliver value to your customers.

vsareto 6y ago
>the larger point is that you are willing to do what is needed They seem to be unable to organize and delegate work effectively even when it's needed and instead rely on polyglot-engineers to bridge that gap by overloading their responsibilities.
tupputuppu 6y ago
Going back and forth isn't progress or keeping up with growth.

saalweachter 6y ago

Is it, uhh, common knowledge that the important thing isn't how the team is organized or how they exactly do their job but that there are engineers paid and incentivized and empowered to keep the status quo running flawlessly (or as close there to as possible) rather than being paid and incentivized and empowered to launch cool new features or make the money graphs go up? The problem at most companies is that if you have two engineers, Alice and Betsy, and Alice launches a change that generated marketing buzz that gets spun into sales, and Betsy launches a change that prevented the site from going down over the seasonal rush, Alice gets promoted and Betsy gets told to work on more impactful projects. Which isn't to say that Alice's work isn't important or promotion worthy, but you need an incentive structure that rewards both.

mjayhn 6y ago
In my experience, Alice moves on to a cool new project to greenfield while Betsy gets stuck supporting Alices creation until they eventually get on-call burnout and quit. Alice checks in only when there's enough noise made either by angry customers, cto or angry SREs. That's my 10yrs+ ops experience, at least, and this is hugely exacerbated if Alice is put on such a high pedestal that they don't even have to be on-call with Betty at any point of the lifecycle.
- CoolGuySteve 6y ago
  I think ultimately this dynamic is why Google can't keep most products around for longer than 5 years.

gowld 6y ago

> We realised SRE is bottle neck for product teams. This means that management decided velocity is more important than reliability?

dragonwriter 6y ago
> > We realised SRE is bottle neck for product teams. > This means that management decided velocity is more important than reliability? No, given the organizational response ("SRE teams are removed and all SRE are embedded into the product teams to do dedicated work for them.") It seems to be like the motivation for Agile cross-functional teams instead of separate Analysis, Design, Coding, Testing, ... teams, and like DevOps, and like DevSecOps, etc., yet another instance of realizing that throwing products over internal walls in organizational handoffs produces inefficiencies.