6 ms·

It sounds like the lesson here is that tacking on SRE to an out-of-control development process, churning out new services by the boatload, doesn't scale. This

It sounds like the lesson here is that tacking on SRE to an out-of-control development process, churning out new services by the boatload, doesn't scale. This is caused by the typical attitude software companies have towards development. The most common model for software development is simple... rush to push out new features to the market, and pay the costs later--and then everyone is balking at the costs. The solution is kind of brilliant, IMO. You don't have to pay technical debt on projects if you shut them down and delete the code from your repository. Migrate to industry-standard solutions. Use off-the-shelf programs & libraries. Delete all your custom stuff. Replace good solutions with "good enough" solutions. The SREs can help you with that, but they can't help with out-of-control development. As your code base gets larger, the cost of supporting that code base gets larger too. The difficulty of scaling your SRE to match development reflects your out-of-control development process, not a problem with SRE. Keep the costs under control by keeping your code base under control.

lykr0n 5y ago
Yep. SRE is not a substitute for high level, overarching architects and designers. One pattern I see is that, as the company grows the development gets split into different product groups which will organically diverge unless there is rigid enforcement of design patterns. In some places, SRE does this implicitly because they will only support X, Y, or Z but in others each product group will have their own group of SREs. There becomes a point when you need one or a small group of people who are the opinionated developers who can make design decisions and who have the authority to cause everyone else to course correct. If you don't have this, you'll wind up with long migrations and legacy stuff that never seems to go away.
- colechristensen 5y ago
  When you get very large you need both enterprise wide SREs that are responsible for consulting and approving architecture for reliability purposes AND localized per service or other small breakdown SREs to support small sets of services. How you break this down is tricky and there is definitely overlap. Crappy attitudes towards support and reliability don’t scale, what you can get away with a few people keeping things barely held together stops working as you grow.
- lmm 5y ago
  I don't find having high-level architects to be a good pattern. They can make mistakes like anyone else; indeed having people who are no longer day-to-day coding make decisions that they don't feel the effects of makes wrong decisions more likely. SRE exists to support product functions and like everything else should be attached to and understood in terms of those functions. Yes, every product group probably should have their own SREs, so that product group can own its whole lifecycle. Yes, different groups will do their own things and there will be mismatches and duplicated effort. That's less bad than the alternative.
  - tinco 5y ago
    We recently decided as a company that the horizontal responsibilities structure doesn't work well at all, at least not at small scale. This was not in the software/infra teams but in our operations but I think there's some general truth here. The more vertically responsible your teams are, the better the final product is, and the more inefficiencies and impedance mismatches you can track down and fix. For us it meant that the data processing teams have been made part of the drone operators team, so whenever we fly a mission a photography/3d rendering expert will also be part of the team that operates the drone. On paper it's more expensive to have office workers in the field, but in practice it leads to fewer reflights and happier and more productive employees. I imagine that for the software departments, it could mean that every app development team has at least one member that has good operating system and network infrastructure knowledge, and/or maybe database expertise so that the team as a whole can largely operate a feature largely without having to depend on an outside SRE specialist. And then the SRE's that you do have can focus on the site reliability, instead of having to constantly tell developers how the way they coded something is bad or whatever.
  - lykr0n 5y ago
    I'm not saying that they should not be active developers, but people who can enforce change across the entire organization. Previous Job (2500+ devs by the time I left) had an in-house RPc system that was being moved over to gRPC. That project was taking years because teams had no coordination on this process. The decision was made at some level and trickled out to everyone else. There was no single person or group who was in charge of: - How services would be discovered - Implementation Patterns of how Services & Methods will be defined - Standardization of which libraries to use - Examples and Pre-build shared libraries that provide the stuff like tracing, monitoring, retries, etc... - Advocating for the changes SRE seems to fall into the position of advocating business value for development practices that compete with business objectives that can provide value as well. At large organizations, if you don't have a central point that can set development objectives and be the one who teams can go to with "this pattern doesn't work for us, we can do this but we need buy in from other teams" issues and have directives handed down. Unless you operate in an environment where the only cross-team communication is well versioned public APIs, then you will run into issues where you have to conflicting needs between teams and need someone to set a vision (this can be a group of people, rotating people, or a single person. how is not the issue)
- mavelikara 5y ago
  > There becomes a point when you need one or a small group of people who are the opinionated developers who can make design decisions and who have the authority to cause everyone else to course correct. How many times have you been in the "everyone else" camp and was course corrected? How did those efforts work out for the firm in the long run? Any experiences to share that could be useful for the community?
  - lykr0n 5y ago
    Previous Job had an in-house RPC system and there was a desire to move to gRPC. This process largely turned into "name and shame" because there was no incentive for some teams to put in the effort to make the changes. They had other objectives to complete, and swapping RPC frameworks was not one of them. The only way the change happened was putting a hard deadline before the old system was shutdown (by SRE), which is not the right way to do it. There were a lot of stories like this. One team owned user information, but the business needs shifted this ownership to another team. This resulted in the ownership being split between three teams, and applications turning into transparent proxies for other applications. One service was a REST interface that provided a bit of logic on top of forwarding the request to a gRPC service. The make up of the company was a bunch of loosely coupled product teams, and the only common connection was SRE and Data who worked with everyone. SRE became the team that had to work to resolve these "what the fuck, why can't you just sit down and figure out who should own this" issues. There really needed to be an architect or someone who could look at the big picture that could say "Why do we have this one internal REST service? Ok. Team A and B. You have this quarter to stop using Service Q and migrate to Service W." $NewCompany, SRE is doing the course correcting (just due to small size), but we have a Principal Developer who is dictating that "Yes, we're going to implement new business logic in lambdas following this pattern." And they work to make sure that everything is done in a standard way, but at the same time take ideas for new patterns and make sure they are done in a smart way and don't conflict with anything else. He doesn't stand as a roadblock, but someone who can make sure teams are not going off and doing weird things (like use MySQL when we're a Postgres shop) that can cause issues later.
rektide 5y ago
My read on the article was that much more was related to each team being on their own to set up & drive their pipelines, operate their own services, and there being a lack of commonality/shared experience. A vast number of the software engineers don't get the ops (running software) stuff hardly at all & half of them can sort of play along, hack stuff into place. The engineers on product teams who do know how to do things meanwhile don't get all the constraints, best practices, ideas that other various DevOps folk have done & have their own wants/desires/expected ways of doing things, so they end up creating their own very unique sub-ways of doing things within the org. None of these practices converge on regularity or consistency with what DevOps machinery ends up being built. What we do have often is just a random pile of containers and scripts that a couple people sort of know decently & everyone else suffers through & survives within. Almost never does it look like any other company's devops kitchen. SRE doesn't scale because it's an every now and then thing, and few people notice or care about the difference between a well-built corporate citizen that runs well & is monitored & operated according to whatever the in-power SRE cabal wants. People start to care only if things are going bad, either via services not building/integrating/deploying/running as well as they should, or from too much confoundedness/general head scratching by either the SRE or regular engineers. SRE is not a priority, it's not practiced regular, it's only an every-now-and-then thing, so we don't have the chance to get good, to institutionalize the right ways of doing things. That's what the articles is discussing. Not the rest of the everyday normal software development rushing-bedlam you describe.
- klodolph 5y ago
  > SRE doesn't scale because it's an every now and then thing, ... That's the part that doesn't scale... tacking on SRE at the end, or doing it every now and then. The reason people don't care about the software being a "well-built corporate citizen" is because they care more about shipping features. If you have an SRE team that will say "no" to you when you try to ship new stuff, you'll eventually figure out a way to build new things in a way that the SRE team will say "yes". When I say "no", that could be a hard pushback like "no, that's not getting shipped" or it could be an answer like, "no, the SRE team will not support that, yet." These kind of decisions need to be made at a high level, because everyone in the institution is typically operating with the wrong incentives. That's why you end up with a random pile of containers and scripts. It doesn't have to end up that way, even when you have microservices. > That's what the articles is discussing. Not the rest of the everyday normal software development rushing-bedlam you describe. I disagree with the article, so necessarily there are going to be differences between what I'm saying and what the article is saying.
bcrosby95 5y ago
Funny, I read it differently. They talk about frameworks, libraries, and best practices. Effectively, they're talking about standardization across your teams/services so they don't fuck things up. Essentially, you're taking away some of the purported freedoms of microservices (complete independence - eg I can write this service in brainfuck if I want!) and reigning it in a bit so you don't build a pile of trash.
- klodolph 5y ago
  I think of that kind of standardization kind of like deleting code. Stuff like, "We are deprecating support for Python in SRE, no new projects may be shipped in Python."
- jayd16 5y ago
  Now all your trash sorta looks the same.
  - zdragnar 5y ago
    Anecdatum time: a friend once worked at a company that had separate teams building small services in a small variety of common languages (mostly python and golang). One individual decided that a particular new service that was going to be really critical just had to be written in erlang, and went ahead and did it. Fast forward to a few years later, when my friend started there. The erlang dev was long gone, and nobody knew erlang or OTP or anything about the service well enough to take ownership of it. My friend was constantly awoken during his on-call weeks because the damned thing kept repeatedly tipping itself over now that the company had grown. He couldn't maintain it other than restarting and praying, and couldn't rewrite it- the company had bigger priorities. That was the shortest lived job of his career so far.
deleted 5y ago
[deleted]
emmelaich 5y ago
It must be possible for SREs to refuse a service that doesn't meet certain criteria. AFAIK this doesn't happen anywhere. Although I believe G has something like this.
- pkhuong 5y ago
  Google SREs definitely have that stick.
- dilyevsky 5y ago
  In order to make this possible you need an entirely separate chain of command for SRE like what google has or very influential SREs either of which is exceedingly rare so not surprised op thinks sre doesn’t scale
- jedmeyers 5y ago
  > I believe G has something like this It does. And until the service meets the criteria, SWEs on the project are actually partially playing the SRE roles under the guidance of the SRE org.
  - tayo42 5y ago
    is sre at google just a maintenance team? what do they do then?
    - tremon 5y ago
      SRE is the first-responder team. They are on-call 24/7 (the team, not each person), perform systems and service monitoring, triage failures and mitigate outages. That doesn't mean it's all handwork, I'm sure SREs at Google employ a boatload of automated event handling and custom response scripts. But "keeping the service up" requires different skills than "building a service", and Google chose to separate their Dev and Ops this way. As others said in this thread, if some service isn't up to SRE standards (in terms of monitoring, logging, or robustness), the SRE team won't accept it and Devs would have to do their own Ops.
    - 5y ago
eikenberry 5y ago
> Use off-the-shelf programs & libraries. Delete all your custom stuff. Replace good solutions with "good enough" solutions. Be very careful following this advice. Only use an off-the-shelf if it is pretty close to what you need or if you can change the need to fit the tool. Trying to shoehorn processes (or whatever) that don't fit a tool to a tool often end up being a lot worse than a custom coded solution. I've seen this multiple times at enterprises that preferred vendor solutions as you suggest and they were worse train wrecks than places that used more custom solutions.
- geodel 5y ago
  Manager: We need API gateway and Service Mesh in Kubernetes for our service Engineer: I have come up with solution that wouldn't need all these frameworks. Manager: Good. See how soon you can integrate with API Gateway and Service Mesh. Heard at friend's workplace.
  - mey 5y ago
    Question is, was it use AN api gateway or use THIS api gateway.
    - geodel 5y ago
      It was use THIS api gateway. Mainly it was exercise in moving from technical work to year long vendor negotiation and setup. The OTHER API gateway blew up despite being used at Netflix. But THIS one is NextGen® so obviously much better.
  - laumars 5y ago
    Without more context either party could be in the wrong there. - The developer for building a non-standard solution that wasn’t asked for and is impossible for the next person to maintain (like those who decide to roll their own database, VCS, encryption, etc) - or the manager for not understanding the requirements of the build and then expecting developers to redo existing work rather than spending 5 minutes learning the tech stack. Either way, this is why some orgs refuse to allow any development without a technical design. When you go into a build with a design that everyone agrees on you ensure that expensive development time is focused and less time is wasted rewriting code due to missing requirements.
pjc50 5y ago
This is making me realise how strong the analogy between Conway's Law and Coase's Theory of the Firm is. (recap: https://en.wikipedia.org/wiki/Theory_of_the_firm https://en.wikipedia.org/wiki/Theory_of_the_firm especially "Why is the boundary between firms and the market located exactly there with relation to size and output variety? Which transactions are performed internally and which are negotiated on the market?") The answer involves looking at informational and coordination costs. Microservices are extremely Coasian: rather than have a part of the software/company under direct, traditional command-and-control management, it's kept at arms length and interacted with via a software contract. Using microservices theoretically reduces the coordination costs between parts of the team, but in practice risks just shovels them around. After all, the easiest way to reduce a cost is to dump the externality on to someone else. If you were deploying a monolith, it would obviously have only one set of SRE practices; with microservices, you end up duplicating those and introducing costly variation.

lykr0n 5y ago

Yep. SRE is not a substitute for high level, overarching architects and designers. One pattern I see is that, as the company grows the development gets split into different product groups which will organically diverge unless there is rigid enforcement of design patterns. In some places, SRE does this implicitly because they will only support X, Y, or Z but in others each product group will have their own group of SREs. There becomes a point when you need one or a small group of people who are the opinionated developers who can make design decisions and who have the authority to cause everyone else to course correct. If you don't have this, you'll wind up with long migrations and legacy stuff that never seems to go away.

colechristensen 5y ago
When you get very large you need both enterprise wide SREs that are responsible for consulting and approving architecture for reliability purposes AND localized per service or other small breakdown SREs to support small sets of services. How you break this down is tricky and there is definitely overlap. Crappy attitudes towards support and reliability don’t scale, what you can get away with a few people keeping things barely held together stops working as you grow.
lmm 5y ago
I don't find having high-level architects to be a good pattern. They can make mistakes like anyone else; indeed having people who are no longer day-to-day coding make decisions that they don't feel the effects of makes wrong decisions more likely. SRE exists to support product functions and like everything else should be attached to and understood in terms of those functions. Yes, every product group probably should have their own SREs, so that product group can own its whole lifecycle. Yes, different groups will do their own things and there will be mismatches and duplicated effort. That's less bad than the alternative.
- tinco 5y ago
  We recently decided as a company that the horizontal responsibilities structure doesn't work well at all, at least not at small scale. This was not in the software/infra teams but in our operations but I think there's some general truth here. The more vertically responsible your teams are, the better the final product is, and the more inefficiencies and impedance mismatches you can track down and fix. For us it meant that the data processing teams have been made part of the drone operators team, so whenever we fly a mission a photography/3d rendering expert will also be part of the team that operates the drone. On paper it's more expensive to have office workers in the field, but in practice it leads to fewer reflights and happier and more productive employees. I imagine that for the software departments, it could mean that every app development team has at least one member that has good operating system and network infrastructure knowledge, and/or maybe database expertise so that the team as a whole can largely operate a feature largely without having to depend on an outside SRE specialist. And then the SRE's that you do have can focus on the site reliability, instead of having to constantly tell developers how the way they coded something is bad or whatever.
- lykr0n 5y ago
  I'm not saying that they should not be active developers, but people who can enforce change across the entire organization. Previous Job (2500+ devs by the time I left) had an in-house RPc system that was being moved over to gRPC. That project was taking years because teams had no coordination on this process. The decision was made at some level and trickled out to everyone else. There was no single person or group who was in charge of: - How services would be discovered - Implementation Patterns of how Services & Methods will be defined - Standardization of which libraries to use - Examples and Pre-build shared libraries that provide the stuff like tracing, monitoring, retries, etc... - Advocating for the changes SRE seems to fall into the position of advocating business value for development practices that compete with business objectives that can provide value as well. At large organizations, if you don't have a central point that can set development objectives and be the one who teams can go to with "this pattern doesn't work for us, we can do this but we need buy in from other teams" issues and have directives handed down. Unless you operate in an environment where the only cross-team communication is well versioned public APIs, then you will run into issues where you have to conflicting needs between teams and need someone to set a vision (this can be a group of people, rotating people, or a single person. how is not the issue)
mavelikara 5y ago
> There becomes a point when you need one or a small group of people who are the opinionated developers who can make design decisions and who have the authority to cause everyone else to course correct. How many times have you been in the "everyone else" camp and was course corrected? How did those efforts work out for the firm in the long run? Any experiences to share that could be useful for the community?
- lykr0n 5y ago
  Previous Job had an in-house RPC system and there was a desire to move to gRPC. This process largely turned into "name and shame" because there was no incentive for some teams to put in the effort to make the changes. They had other objectives to complete, and swapping RPC frameworks was not one of them. The only way the change happened was putting a hard deadline before the old system was shutdown (by SRE), which is not the right way to do it. There were a lot of stories like this. One team owned user information, but the business needs shifted this ownership to another team. This resulted in the ownership being split between three teams, and applications turning into transparent proxies for other applications. One service was a REST interface that provided a bit of logic on top of forwarding the request to a gRPC service. The make up of the company was a bunch of loosely coupled product teams, and the only common connection was SRE and Data who worked with everyone. SRE became the team that had to work to resolve these "what the fuck, why can't you just sit down and figure out who should own this" issues. There really needed to be an architect or someone who could look at the big picture that could say "Why do we have this one internal REST service? Ok. Team A and B. You have this quarter to stop using Service Q and migrate to Service W." $NewCompany, SRE is doing the course correcting (just due to small size), but we have a Principal Developer who is dictating that "Yes, we're going to implement new business logic in lambdas following this pattern." And they work to make sure that everything is done in a standard way, but at the same time take ideas for new patterns and make sure they are done in a smart way and don't conflict with anything else. He doesn't stand as a roadblock, but someone who can make sure teams are not going off and doing weird things (like use MySQL when we're a Postgres shop) that can cause issues later.

rektide 5y ago

My read on the article was that much more was related to each team being on their own to set up & drive their pipelines, operate their own services, and there being a lack of commonality/shared experience. A vast number of the software engineers don't get the ops (running software) stuff hardly at all & half of them can sort of play along, hack stuff into place. The engineers on product teams who do know how to do things meanwhile don't get all the constraints, best practices, ideas that other various DevOps folk have done & have their own wants/desires/expected ways of doing things, so they end up creating their own very unique sub-ways of doing things within the org. None of these practices converge on regularity or consistency with what DevOps machinery ends up being built. What we do have often is just a random pile of containers and scripts that a couple people sort of know decently & everyone else suffers through & survives within. Almost never does it look like any other company's devops kitchen. SRE doesn't scale because it's an every now and then thing, and few people notice or care about the difference between a well-built corporate citizen that runs well & is monitored & operated according to whatever the in-power SRE cabal wants. People start to care only if things are going bad, either via services not building/integrating/deploying/running as well as they should, or from too much confoundedness/general head scratching by either the SRE or regular engineers. SRE is not a priority, it's not practiced regular, it's only an every-now-and-then thing, so we don't have the chance to get good, to institutionalize the right ways of doing things. That's what the articles is discussing. Not the rest of the everyday normal software development rushing-bedlam you describe.

klodolph 5y ago
> SRE doesn't scale because it's an every now and then thing, ... That's the part that doesn't scale... tacking on SRE at the end, or doing it every now and then. The reason people don't care about the software being a "well-built corporate citizen" is because they care more about shipping features. If you have an SRE team that will say "no" to you when you try to ship new stuff, you'll eventually figure out a way to build new things in a way that the SRE team will say "yes". When I say "no", that could be a hard pushback like "no, that's not getting shipped" or it could be an answer like, "no, the SRE team will not support that, yet." These kind of decisions need to be made at a high level, because everyone in the institution is typically operating with the wrong incentives. That's why you end up with a random pile of containers and scripts. It doesn't have to end up that way, even when you have microservices. > That's what the articles is discussing. Not the rest of the everyday normal software development rushing-bedlam you describe. I disagree with the article, so necessarily there are going to be differences between what I'm saying and what the article is saying.

bcrosby95 5y ago

Funny, I read it differently. They talk about frameworks, libraries, and best practices. Effectively, they're talking about standardization across your teams/services so they don't fuck things up. Essentially, you're taking away some of the purported freedoms of microservices (complete independence - eg I can write this service in brainfuck if I want!) and reigning it in a bit so you don't build a pile of trash.

klodolph 5y ago
I think of that kind of standardization kind of like deleting code. Stuff like, "We are deprecating support for Python in SRE, no new projects may be shipped in Python."
jayd16 5y ago
Now all your trash sorta looks the same.
- zdragnar 5y ago
  Anecdatum time: a friend once worked at a company that had separate teams building small services in a small variety of common languages (mostly python and golang). One individual decided that a particular new service that was going to be really critical just had to be written in erlang, and went ahead and did it. Fast forward to a few years later, when my friend started there. The erlang dev was long gone, and nobody knew erlang or OTP or anything about the service well enough to take ownership of it. My friend was constantly awoken during his on-call weeks because the damned thing kept repeatedly tipping itself over now that the company had grown. He couldn't maintain it other than restarting and praying, and couldn't rewrite it- the company had bigger priorities. That was the shortest lived job of his career so far.

deleted 5y ago

[deleted]

emmelaich 5y ago

It must be possible for SREs to refuse a service that doesn't meet certain criteria. AFAIK this doesn't happen anywhere. Although I believe G has something like this.

pkhuong 5y ago
Google SREs definitely have that stick.
dilyevsky 5y ago
In order to make this possible you need an entirely separate chain of command for SRE like what google has or very influential SREs either of which is exceedingly rare so not surprised op thinks sre doesn’t scale
jedmeyers 5y ago
> I believe G has something like this It does. And until the service meets the criteria, SWEs on the project are actually partially playing the SRE roles under the guidance of the SRE org.
- tayo42 5y ago
  is sre at google just a maintenance team? what do they do then?
  - tremon 5y ago
    SRE is the first-responder team. They are on-call 24/7 (the team, not each person), perform systems and service monitoring, triage failures and mitigate outages. That doesn't mean it's all handwork, I'm sure SREs at Google employ a boatload of automated event handling and custom response scripts. But "keeping the service up" requires different skills than "building a service", and Google chose to separate their Dev and Ops this way. As others said in this thread, if some service isn't up to SRE standards (in terms of monitoring, logging, or robustness), the SRE team won't accept it and Devs would have to do their own Ops.
  - 5y ago

eikenberry 5y ago

> Use off-the-shelf programs & libraries. Delete all your custom stuff. Replace good solutions with "good enough" solutions. Be very careful following this advice. Only use an off-the-shelf if it is pretty close to what you need or if you can change the need to fit the tool. Trying to shoehorn processes (or whatever) that don't fit a tool to a tool often end up being a lot worse than a custom coded solution. I've seen this multiple times at enterprises that preferred vendor solutions as you suggest and they were worse train wrecks than places that used more custom solutions.

geodel 5y ago
Manager: We need API gateway and Service Mesh in Kubernetes for our service Engineer: I have come up with solution that wouldn't need all these frameworks. Manager: Good. See how soon you can integrate with API Gateway and Service Mesh. Heard at friend's workplace.
- mey 5y ago
  Question is, was it use AN api gateway or use THIS api gateway.
  - geodel 5y ago
    It was use THIS api gateway. Mainly it was exercise in moving from technical work to year long vendor negotiation and setup. The OTHER API gateway blew up despite being used at Netflix. But THIS one is NextGen® so obviously much better.
- laumars 5y ago
  Without more context either party could be in the wrong there. - The developer for building a non-standard solution that wasn’t asked for and is impossible for the next person to maintain (like those who decide to roll their own database, VCS, encryption, etc) - or the manager for not understanding the requirements of the build and then expecting developers to redo existing work rather than spending 5 minutes learning the tech stack. Either way, this is why some orgs refuse to allow any development without a technical design. When you go into a build with a design that everyone agrees on you ensure that expensive development time is focused and less time is wasted rewriting code due to missing requirements.

pjc50 5y ago

This is making me realise how strong the analogy between Conway's Law and Coase's Theory of the Firm is. (recap: https://en.wikipedia.org/wiki/Theory_of_the_firm https://en.wikipedia.org/wiki/Theory_of_the_firm especially "Why is the boundary between firms and the market located exactly there with relation to size and output variety? Which transactions are performed internally and which are negotiated on the market?") The answer involves looking at informational and coordination costs. Microservices are extremely Coasian: rather than have a part of the software/company under direct, traditional command-and-control management, it's kept at arms length and interacted with via a software contract. Using microservices theoretically reduces the coordination costs between parts of the team, but in practice risks just shovels them around. After all, the easiest way to reduce a cost is to dump the externality on to someone else. If you were deploying a monolith, it would obviously have only one set of SRE practices; with microservices, you end up duplicating those and introducing costly variation.