5 ms·

I worked at a major software company that was implementing an “SRE” org and it was frankly pathetic. It basically had its genesis in some higher up hearing that

by pram 6y ago

I worked at a major software company that was implementing an “SRE” org and it was frankly pathetic. It basically had its genesis in some higher up hearing that Google does this thing, so we probably should too. That then later turned into a bunch of random DevOps/Sys Admins drafted into the new org reading the Google SRE book. It was a shamelessly cargo cult approach. After a year all we really had was hundreds of Confluence pages specifying policy for this and that, a bunch of ceremony added around releases, and arbitrary SLA/SLOs defined. Almost zero actual technical work was accomplished. They then started filling the team with new grads who had neither devops or sysadmin experience, kind of cementing the perception that it was a non-technical role. Either way I’m not a fan of the entire concept after being subjected to that suffering!

andriosr 6y ago
Blindly replicating what Google is doing is really problematic, I saw something similar at a few companies. The principles are interesting, but they have to consider each company's reality. Google is putting a lot of marketing money on the topic, but from what I saw they are trying to alert that there is no one size fits all for this.
- jeffbee 6y ago
  So far I haven't seen any companies replicating what Google SRE does, blindly or intentionally. Much as this article describes, organizations I've seen are just inventing a role they perceive to exist in their organization and slapping the label "SRE" upon it. Often these roles are directly contradictory to the way Google SRE works: they are just carrying pagers for products they aren't allowed to improve, they own build/test/release for some reason even though that clearly falls in the domain of the developer, SREs are setting service levels which is obviously the responsibility of product. I think people should realize that when you go into the hiring market looking for SREs, you are about to pay more, not less, for a highly qualified software engineer who is also a systems expert. You want these people to not just operate product, but to be closely involved in architecture and development so the product turns out to be operable. Your SREs should be working in the product code on the bits that ordinary developers might overlook or not be qualified to write, because they lack expertise in system or distributed systems behavior. What this article is calling "real world" is just an attempt to redefine the term.
  - sumtechguy 6y ago
    An SRE with no authority to do anything is just an overpaid totally frustrated paperweight. They have to be able to change things on both sides of the 'fence' if they can only do one or the other or neither then their role is not an SRE no matter what sorts of titles you give them. Any org I go into that says 'we want an SRE because google has one' I can not help myself I blurt out 'you are not google. what works for them will not work for you, you need to find your own way of doing this that fits your org. Trying to mimic google without the understanding of why they way they are will damage your org'
    - user5994461 6y ago
      >>> An SRE with no authority to do anything is just an overpaid totally frustrated paperweight. That matches my experience at most companies, the SRE/DevOps is just the sucker who is on call.
      - dodobirdlord 6y ago
        I think authority is a critical differentiator between the SRE described in the Google book and SRE as it’s practiced elsewhere. Google SREs can impose moratoriums on feature development or deployments if they judge that a product is becoming critically unreliable (which has a well-defined meaning based on SLO breaches). SRE teams will also sometimes “offboard” a product that is insufficiently respectful of the finite attention that an SRE team needs to spread across multiple products, returning full production control to the development team along with the pager and all responsibilities thereof.
        sumtechguy 6y ago
        I think one more differentiator is the dedicated position. Rotation is not SRE. It can be helpful to help people understand both sides. But just rotating your team into the role every few weeks with no real 'group in charge' just seems to make them mad.
    - apple4ever 6y ago
- the_70x 6y ago
  and the fact that not all the companies run the same workloads or scale as google
- zimbatm 6y ago
  I feel like there is an analogy with self-help books. Every personality and company is different and they have to do the work to find out what works best for them. Internalizing the meaning of things and leading your own thinking is probably as important as the practices themselves. Adopting another company's practices is often lazy thinking. There are no silver bullets.
- gowld 6y ago
  Marketing money? besides paying people to publish their work?
tannhaeuser 6y ago
Sounds truly horrible, and meets my experience of SOA around 2005, but let's not forget that younger devs don't have the luxury of having tinkered with stuff hands-on as it was introduced, such as SQL, pre-jquery js, SOA, Unix/Linux to form an intuition.
- C1sc0cat 6y ago
  Is that not a problem with the training / education of entry level. This was the problem the raspberry pi was designed for GCSE and A Level students to actually get hands on experience, so that when you go to to university you had at least the basics.
SV_BubbleTime 6y ago
> Almost zero actual technical work was accomplished. This is 100% my problem with this article. At no point is it ever implied WHAT or WHY or WHERE actual work is being done. Only HOW they organize it. It looks to me from the article they are using every “cool” tool and seem to care more about the how than the why. And that’s fine if it’s the point of the article which I guess it is, but it seems like it’s “reliably” structuring a husk. I defend the costs of IT and in this case the costs of an SRE approach - but these aren’t the things that make you money and keep your business afloat. When I spend a couple weeks making a system wide event bus to get messages from n to m that’s cool and I need it... but customers don’t pay me for that. They pay me to fix a problem they have. I’m very cautious of articles like this that say all “the right” things but don’t even hint at solving customer problems.
dilyevsky 6y ago
You just stumbled upon same issue google had - sre-swe is really tough to hire for. The concept is good but you need software engineers w/ systems background and they’re relatively rare and most good ones are already at faang
- mjayhn 6y ago
  I mean Google could stop being completely elitist with SREs needing to come from SWE backgrounds and hire the thousands of SRE/Sysadmin/Ops types who do know systems very well but have a lot/bit less SWE background and let their SWEs contribute to mentoring them on SWE stuff. Pipe dream of mine, for sure. There's been at least 3 G SREs that have posted in here about what they do day to day and it's literally no different from any other non-FAANG SRE/Ops role any of us do at various scales except for the gatekeeping.
  - dilyevsky 6y ago
    Google does hire SRE SEs and I interviewed many - the focus on coding is a lot less and they get grilled more on systems etc. it’s important to have a healthy mixture of both or you run into eng culture issues like described by many posters here
    - jldugger 6y ago
      > Google does hire SRE SEs and I interviewed many For those who aren't familiar with the google acroynm soup: SE here means systems engineer not software engineer(SWE).
    - kazen44 6y ago
      Also, a systems engineer usually has a different way of thinking about a problem then a software engineer and vice versa. This helps tremendously in troubleshooting and building stable products. Having a combination of systems, networking and programming "perspective" on a team seems to result in very robust systems all around.
  - joshuamorton 6y ago
krooj 6y ago
Is this TEAM? There's a nasty issue with companies trying to emulate Google on "all the things" and it usually doesn't work, since often times these organizational structures are a natural evolution of where a company came from, but you aren't that company... Best SRE/QE model I've experienced was with a couple of these guys being embedded into a dev team (reporting structure didn't really matter), such that they participated in the team meetings and rituals, becoming domain experts. One fellow eventually learned the SAML2.0 spec inside out and was invaluable for catching bugs. I suppose being embedded but reporting up through a different org did have it's benefit: namely, the consistent dissemination of best practices across the company, but with a judicious eye to applicable context that the current do-it-all-swe cannot with the current SRE ivory tower model. The do-it-all-swe model has yielded trash QE/observability in my experience.
- pram 6y ago
  Yes, lol

by pram 6y ago

andriosr 6y ago

Blindly replicating what Google is doing is really problematic, I saw something similar at a few companies. The principles are interesting, but they have to consider each company's reality. Google is putting a lot of marketing money on the topic, but from what I saw they are trying to alert that there is no one size fits all for this.

jeffbee 6y ago
So far I haven't seen any companies replicating what Google SRE does, blindly or intentionally. Much as this article describes, organizations I've seen are just inventing a role they perceive to exist in their organization and slapping the label "SRE" upon it. Often these roles are directly contradictory to the way Google SRE works: they are just carrying pagers for products they aren't allowed to improve, they own build/test/release for some reason even though that clearly falls in the domain of the developer, SREs are setting service levels which is obviously the responsibility of product. I think people should realize that when you go into the hiring market looking for SREs, you are about to pay more, not less, for a highly qualified software engineer who is also a systems expert. You want these people to not just operate product, but to be closely involved in architecture and development so the product turns out to be operable. Your SREs should be working in the product code on the bits that ordinary developers might overlook or not be qualified to write, because they lack expertise in system or distributed systems behavior. What this article is calling "real world" is just an attempt to redefine the term.
- sumtechguy 6y ago
  An SRE with no authority to do anything is just an overpaid totally frustrated paperweight. They have to be able to change things on both sides of the 'fence' if they can only do one or the other or neither then their role is not an SRE no matter what sorts of titles you give them. Any org I go into that says 'we want an SRE because google has one' I can not help myself I blurt out 'you are not google. what works for them will not work for you, you need to find your own way of doing this that fits your org. Trying to mimic google without the understanding of why they way they are will damage your org'
  - user5994461 6y ago
    >>> An SRE with no authority to do anything is just an overpaid totally frustrated paperweight. That matches my experience at most companies, the SRE/DevOps is just the sucker who is on call.
    - dodobirdlord 6y ago
      I think authority is a critical differentiator between the SRE described in the Google book and SRE as it’s practiced elsewhere. Google SREs can impose moratoriums on feature development or deployments if they judge that a product is becoming critically unreliable (which has a well-defined meaning based on SLO breaches). SRE teams will also sometimes “offboard” a product that is insufficiently respectful of the finite attention that an SRE team needs to spread across multiple products, returning full production control to the development team along with the pager and all responsibilities thereof.
      - sumtechguy 6y ago
        I think one more differentiator is the dedicated position. Rotation is not SRE. It can be helpful to help people understand both sides. But just rotating your team into the role every few weeks with no real 'group in charge' just seems to make them mad.
  - apple4ever 6y ago
the_70x 6y ago
and the fact that not all the companies run the same workloads or scale as google
zimbatm 6y ago
I feel like there is an analogy with self-help books. Every personality and company is different and they have to do the work to find out what works best for them. Internalizing the meaning of things and leading your own thinking is probably as important as the practices themselves. Adopting another company's practices is often lazy thinking. There are no silver bullets.
gowld 6y ago
Marketing money? besides paying people to publish their work?

tannhaeuser 6y ago

Sounds truly horrible, and meets my experience of SOA around 2005, but let's not forget that younger devs don't have the luxury of having tinkered with stuff hands-on as it was introduced, such as SQL, pre-jquery js, SOA, Unix/Linux to form an intuition.

C1sc0cat 6y ago
Is that not a problem with the training / education of entry level. This was the problem the raspberry pi was designed for GCSE and A Level students to actually get hands on experience, so that when you go to to university you had at least the basics.

SV_BubbleTime 6y ago

> Almost zero actual technical work was accomplished. This is 100% my problem with this article. At no point is it ever implied WHAT or WHY or WHERE actual work is being done. Only HOW they organize it. It looks to me from the article they are using every “cool” tool and seem to care more about the how than the why. And that’s fine if it’s the point of the article which I guess it is, but it seems like it’s “reliably” structuring a husk. I defend the costs of IT and in this case the costs of an SRE approach - but these aren’t the things that make you money and keep your business afloat. When I spend a couple weeks making a system wide event bus to get messages from n to m that’s cool and I need it... but customers don’t pay me for that. They pay me to fix a problem they have. I’m very cautious of articles like this that say all “the right” things but don’t even hint at solving customer problems.

dilyevsky 6y ago

You just stumbled upon same issue google had - sre-swe is really tough to hire for. The concept is good but you need software engineers w/ systems background and they’re relatively rare and most good ones are already at faang

mjayhn 6y ago
I mean Google could stop being completely elitist with SREs needing to come from SWE backgrounds and hire the thousands of SRE/Sysadmin/Ops types who do know systems very well but have a lot/bit less SWE background and let their SWEs contribute to mentoring them on SWE stuff. Pipe dream of mine, for sure. There's been at least 3 G SREs that have posted in here about what they do day to day and it's literally no different from any other non-FAANG SRE/Ops role any of us do at various scales except for the gatekeeping.
- dilyevsky 6y ago
  Google does hire SRE SEs and I interviewed many - the focus on coding is a lot less and they get grilled more on systems etc. it’s important to have a healthy mixture of both or you run into eng culture issues like described by many posters here
  - jldugger 6y ago
    > Google does hire SRE SEs and I interviewed many For those who aren't familiar with the google acroynm soup: SE here means systems engineer not software engineer(SWE).
  - kazen44 6y ago
    Also, a systems engineer usually has a different way of thinking about a problem then a software engineer and vice versa. This helps tremendously in troubleshooting and building stable products. Having a combination of systems, networking and programming "perspective" on a team seems to result in very robust systems all around.
- joshuamorton 6y ago

krooj 6y ago

Is this TEAM? There's a nasty issue with companies trying to emulate Google on "all the things" and it usually doesn't work, since often times these organizational structures are a natural evolution of where a company came from, but you aren't that company... Best SRE/QE model I've experienced was with a couple of these guys being embedded into a dev team (reporting structure didn't really matter), such that they participated in the team meetings and rituals, becoming domain experts. One fellow eventually learned the SAML2.0 spec inside out and was invaluable for catching bugs. I suppose being embedded but reporting up through a different org did have it's benefit: namely, the consistent dissemination of best practices across the company, but with a judicious eye to applicable context that the current do-it-all-swe cannot with the current SRE ivory tower model. The do-it-all-swe model has yielded trash QE/observability in my experience.

pram 6y ago
Yes, lol