12 ms·

SRE Doesn’t Scale

162 points by kiyanwang 5y ago

ilrwbwrkhv 5y ago
Microservices should be used as an optimization. Not as a starting point.
mbesto 5y ago
Isn't this more of a comment about microservices than it is about SRE? It reads to me like "once you hit a number of microservices it ends up looking like a monolith": http://highscalability.com/blog/2020/4/8/one-team-at-uber-is-moving-from-microservices-to-macroservic.html http://highscalability.com/blog/2020/4/8/one-team-at-uber-is...
- willcipriano 5y ago
  It's always microservices, whatever you build. You just decide what kind of IPC you use between some of the pieces.
- wara23arish 5y ago
  dumb question time but what exactly makes something a micro service. Is the separation of a specific functionality from a wider array of functions to its own vm make it a microservice? When does something stop being a microservice i guess?
  - notyourday 5y ago
    > dumb question time but what exactly makes something a micro service. This leftpad as a service, over HTTPS
    - wara23arish 5y ago
      I had to google that to understand the context. What a wild story
  - forty 5y ago
    I remember asking a candidate whether they were doing microservices at her current job. She answered "I don't know if we have microservices, but we do have services that don't do much" It's since then that's my definition of a microservice :)
    - rhizome 5y ago
      Productized class methods.
  - thecleaner 5y ago
    My definition is separation of infrastructure and deployment cycles. Everything that always in one deployment is one service or stuff thats part of your code-base is definitely not a different service.
  - igetspam 5y ago
    It stops being a microservice when a developer starts saying, "oh! We can do X in service Y too! It already does ${similar work} and reads/writes from/to ${data source}, so why not?" The intended model is to do one thing, thus enabling surgical changes to functionality without having to rebuild everything. As long as you stick to your API contracts, you can muck around with the internals without effecting anything else.
    - chousuke 5y ago
      That's not really answering the question. How do you define what the "one thing" that the service handles actually is? I have this half-serious opinion that interface design is the hard (technical) part of software development, and the rest is mostly easy.
      - wara23arish 5y ago
        I remember coming to the same realization when the Google Vs Oracle court case was going on. Which decided that API’s are not copyrightable. To me it felt like the wrong decision, but what do i know.
        pjc50 5y ago
        Copyright in interfaces entrenches monopoly.
        chousuke 5y ago
        I would definitely argue that public interfaces should not be copyrightable, since you need to be able to freely reimplement them to interoperate, which in my opinion should be preferred over granting even temporary monopolies to software interfaces. However, public interfaces are only a small part of software interfaces in total. It's getting the system's internal interfaces right that takes most of the work.
        wara23arish 5y ago
        I totally get it from the sense of avoiding monopolies for sure. In my view, I was lumping internal/public interfaces in the same basket. I haven’t really considered public vs internal aspect of this.
  - chippiewill 5y ago
    I think it's the same line of thinking as RISC and CISC (reduced/complex instruction set computer). There's no such thing as a RISC machine or a CISC machine, all computer ISAs live on a spectrum that can be described as RISC-like or CISC-like. There's no such thing as a micro service or a macro service (in terms of there being a perfect delineation). They're simply terms to describe certain design goals. Any given service might have several qualities that can be described as befitting a macro or a micro service. It's all too easy to fall into the no true scotsman trap.
  - deleted 5y ago
    [deleted]
  - jayd16 5y ago
    I would say its best to only think about any deployable thing as a service. "Microservices" is just the idea that those things do not have a minimum size. Essentially, "is this enough functionality to make a new service?" is the wrong question.
  - deleted 5y ago
    [deleted]
  - rmbyrro 5y ago
    A microservice is deployed independently, has its own hardware and can scale independently from the rest of the system. If there are multiple services and they all share hardware, they're not really microservices.
    - alexeldeib 5y ago
      > A microservice is deployed independently, has its own hardware and can scale independently from the rest of the system. Nicely articulated > If there are multiple services and they all share hardware, they're not really microservices. You just described most microservice deployments on Kubernetes ;)
      - rmbyrro 5y ago
        In this case, they run in containers, so they can actually scale independently. For this purpose, it's as if they were running on independent hardware. Even if we had separate cloud servers - such as EC2 on AWS - for each service, technically they are not really independent hardware. But for the purposes of deploying, hardware capacity allocation and scaling, you can see them as such. The same goes for containers orchestrated by K8s.
    - cnst 5y ago
      Don't we have a term for that — software coupling? A microservice architecture has weak coupling.
      - anton-107 5y ago
        it's coupling via network calls instead of in-memory calls
  - snug 5y ago
    I like to think of it more of a separation of components needed to do the job. Need a database, use a database as it's own service, don't couple the database to the app, so if you need to scale the app horizontally you can add more replicas. If you need to scale the database vertically, you can scale the database vertically. If these are coupled, you scale only whats needed. On top of that use the language or tool that is meant for the job. If you need something written in go, build your service with go, if you need nodejs, do it with node, etc. Microservices don't need to be micro or small
- iamstupidsimple 5y ago
  Forgive me but aren't 'macroservices' just... services? I don't see the difference.
  - steveBK123 5y ago
    unless its a gigaservice
pram 5y ago
“Hiring experienced, qualified SREs is difficult and costly. Despite enormous effort from the recruiting organization, there are never enough SREs to support all the services that need their expertise.” Uh huh. Maybe what isn't scaling is their onerous recruiting filters.
- bostonsre 5y ago
  It's good to be an sre. From my experience, there is high demand, low supply and schools don't really teach for sre roles so the pipeline stays small.
  - outworlder 5y ago
    Yes. And then there's the unrealistic demands from companies that didn't even understand what DevOps was supposed to be, and are misunderstanding SRE even more. Many companies still treats this the same way they did "ops".
- anonporridge 5y ago
  A different element of this is that software professionals are often paid so well that a lot of people can take early retirement relatively easily if they're good about saving and investing. And the FIRE movement is only growing. I sometimes wonder if the software world has relatively greater 'leakage' via early retirement than other fields, creating a constant problem of not enough highly experienced people who remain stuck as wage slaves throughout a 40 year career.
- zaptheimpaler 5y ago
  Listen, if you can't write a working text justification algorithm in 45 minutes a decade after college, you have no business doing completely unrelated SRE stuff like monitoring services or scaling databases. Everybody knows leetcode is the key to software. If you can't do that, maybe you just don't have the right IQ to build the next chat app @ Google.
  - GauntletWizard 5y ago
klodolph 5y ago
It sounds like the lesson here is that tacking on SRE to an out-of-control development process, churning out new services by the boatload, doesn't scale. This is caused by the typical attitude software companies have towards development. The most common model for software development is simple... rush to push out new features to the market, and pay the costs later--and then everyone is balking at the costs. The solution is kind of brilliant, IMO. You don't have to pay technical debt on projects if you shut them down and delete the code from your repository. Migrate to industry-standard solutions. Use off-the-shelf programs & libraries. Delete all your custom stuff. Replace good solutions with "good enough" solutions. The SREs can help you with that, but they can't help with out-of-control development. As your code base gets larger, the cost of supporting that code base gets larger too. The difficulty of scaling your SRE to match development reflects your out-of-control development process, not a problem with SRE. Keep the costs under control by keeping your code base under control.
- lykr0n 5y ago
  Yep. SRE is not a substitute for high level, overarching architects and designers. One pattern I see is that, as the company grows the development gets split into different product groups which will organically diverge unless there is rigid enforcement of design patterns. In some places, SRE does this implicitly because they will only support X, Y, or Z but in others each product group will have their own group of SREs. There becomes a point when you need one or a small group of people who are the opinionated developers who can make design decisions and who have the authority to cause everyone else to course correct. If you don't have this, you'll wind up with long migrations and legacy stuff that never seems to go away.
  - colechristensen 5y ago
    When you get very large you need both enterprise wide SREs that are responsible for consulting and approving architecture for reliability purposes AND localized per service or other small breakdown SREs to support small sets of services. How you break this down is tricky and there is definitely overlap. Crappy attitudes towards support and reliability don’t scale, what you can get away with a few people keeping things barely held together stops working as you grow.
igetspam 5y ago
> Google enforces standards and opinions around things like programming languages, instrumentation and metrics, logging, and control systems surrounding traffic and load management. I think the author read this as more of a problem than a solution. This concept is supported by the DevOps model too. Your infrastructure is just as much a part of your product and the teams providing the infrastructure just as responsible for the service levels and API contracts as any customer facing product team.
gautamdivgi 5y ago
> And that move to microservices—in combination with cloud—unleashes a whole new level of autonomy and empowerment for developers who, often coming from a more restrictive ops-controlled environment on prem, introduce all sorts of new programming languages, compute platforms, databases, and other technologies. You need standards, without that SRE is pointless. Everything needs a standard method of monitoring. As an e.g. - stick to Java/Spring Boot, MariaDB and K8S. That will generally cover 85% of your use cases. The automation and advantage of SRE is derived through standards and familiarity with the tool chain.
slaymaker1907 5y ago
I kind of disagree on the Cambrian explosion of languages being a bad thing. While it has downsides, it has a massive upside of discouraging stuff like crazy custom build pipelines. Big companies love to burn money and time by reinventing everything. Bigco isn't going to try and implement their own build system if they need to support a dozen languages.
- high_5 5y ago
  Microservices have a marvelous long-term potential to trash the service with a myriad of legacy and obsolete languages instead of just one. So you just have to run faster to stay in the same place.
kodah 5y ago
The thing that made Google SRE scale was in the fine print and covered in this article. SRE is a premium service, and in order to get the premium service you had to meet some criteria to show you weren't about to turn highly paid, highly knowledgeable folks into professional firefighters. During my time as a SRE at multiple organizations what I witnessed was executives that owned Operations teams renaming the team, paying for a Python course, and calling it "change". Operations Engineers have a culture unto themselves that will not be broken in a years time. A culture of firefighting and heroism as opposed to proactivity and dull standards. They're rarely, if ever, programmers which was a prerequisite for a SRE-SE and SRE-SWE. You don't need to be a master in logic to know that if you dress your old pig up with lipstick and a new title that it's still a pig. I agree with the article: read the SRE book and understand why they did what they did, and if you hire SRE-SE or SRE-SWE types, don't colocate them with Operations types or make them a 1:1 map with your dev teams that haven't earned them.
- Cthulhu_ 5y ago
  I heard about that; amongst other things, if the amount of times SRE gets pinged because of a software fault exceeds X amount, they basically give the pager back to the team that built the software. They aren't taking responsibility for reliability if the software is not reliable. I really appreciated working with a team of SRE types, it gave me a newfound appreciation for quality in software development. I remember one instance where a colleague in my team went up to the 'ops' team and wanted SSH access to a production server to check on some settings (environment variables, I believe). He thought it was a trivial thing, you know, "just lemme have a peek" kinda thing, but the ops team flat-out told him no, if he needs to print env vars, he can do it in his own code - we did continuous releases, a patch could be deployed within minutes if passing all the checks. I loved that the ops team had the mandate from higher up to say no to requests like this, and I found them a lot more professional than the software developers, including myself.
  - kodah 5y ago
    Ideally what that developer was wanting should be a function of the platform. That's likely not in an operations teams scope because they're mostly just fire fighters. Also ideally in a situation like SOX where devs can't have access to production their preproduction environments share the same interface that production has and if their values differ that much is documented.
nateberkopec 5y ago
At some point, it feels like every company is basically re implementing a kind of internal Heroku. I feel like there’s a billion dollar business in putting that on rails for medium to large enterprises.
- jacques_chester 5y ago
  There are many products that aim to fill this niche. I worked on one for years (Cloud Foundry). There is a lot of money to be made, but it's surprisingly stony ground. Oblig. self-promoted thread: https://twitter.com/jacques_chester/status/1435738825347645444 https://twitter.com/jacques_chester/status/14357388253476454...
cletus 5y ago
It actually sounds more like what I'd expect: microservices don't scale. I've never been a fan of microservices for this simple reason: every boundary, every interface, every connection is friction and a potential source of bugs, performance issues and problems. It's now been several years since I worked at Google so I can't speak to the current state of things but when I was there, monoliths ruled the roost. It seems possible there was some experimentation with microservices given the rise of Docker and Kubernetes in the last few years. I'd be surprised if this made serious inroads into high-traffic core services. It's also worth mentioning that generally Google has (had?) SRE teams rather than SREs embedded on SWE teams. You'd never have one SRE supporting a service as there'd have to be enough to maintain a healthy oncall rotation (typically 8-12). I can't help but feel like the author is projecting his own experiences onto what Google has said here.
- jatins 5y ago
  I previously worked in a BigCo that did microservices and currently work at BigCo that leans towards monoliths. It's completely true that microservices makes development _very_ hard. Just being able to run things locally was very hard since instead of understanding how to build/run one service, you had to figure our how to build/run N services. Testing was another major headache. Monoliths help with technical aspects but organizationally they make your life hell and can severely limit your ability to move fast.
- Cthulhu_ 5y ago
  Microservices are a cargo cult thing. It tries to solve a problem that 99% of companies that apply it wish they had.
75dvtwin 5y ago
Hiring SREs is costly... Hiring narrowly focused people with significant pre-existing experience -- Is always costly. ... and I also think, it is practically always a wrong strategy... I am fundamentally against micro-managing labor pools by federal government. However, there are need to be economic incentives (including immigration policies) -- that make it more difficult for employers to hire for 'tool-centric' positions, unless those tools are very expensive physical devices (eg telescopes, quantum computers, etc). The incentives need to direct employers at training on the job (not at after-work online classes). When I see on HN's hire threads -- 'if you have Azure experience -- you will get on top of the pile' -- I cringe. This is absolutely ridiculous. Same pretty much with Ruby/Php Rust/C++ Haskell/Scala modalities of the same problem. Yes hire people who understand process/idioms/patters, but invest in the f..ing training -- if you need somebody 'yesterday' to help, get consultants -- and have them help and at the same time hire for full time roles -- and train your internal stuff (possibly, even, have them learn from the consultants). Afraid of people leaving after acquiring the highly thought-out skills ? -- implement meaningful compensation/retention policies that reflect effort that you spend on training, and risks that you are taking if a person who was just trained -- leaves.
- winrid 5y ago
  What you are saying is generalists are cheaper to hire and that you can train them. That's true. That's why, as a seller (worker) you should try to specialize in at least one area to sell yourself better.
  - 75dvtwin 5y ago
    That's part of it. I am also saying that there needs to be a system of incentives, that then, produces, organizational/hiring practices that favor non-tool-specific employment process.
    - winrid 5y ago
      Well, for that your company has to admit it it doesn't have specific problems. Because if you do have specific problems, it's probably more efficient to hire people that can handle those.
stefanve 5y ago
To me SRE + Cloud means SRE is responsible for the cloud foundation/ platform but the software teams should be DevOps teams and they are responsible for the (micro)services they create. SRE helps out by providing the tools, libs, dashboard , general principles etc. So SRE in responsible for the health of Kubernetes, networking etc and providing libs, support and infra but the DevOps teams are responsible for the health of the services/ software that is running on the cloud platform. A DevOPs teams should not consist of developers and one ops person but the team as a whole is responsible for the stuff they create. They are on call just like SRE is, if something go's down SRE + DevOps work together to fix it
KaiserPro 5y ago
> this person typically just becomes the “whipping boy” for the team. That sounds like a culture problem to me. SRE is just a fancy name for a class of people that make stuff more reliable. However it appears the reader of this book forgot about the other part: making stuff easy to use. Everywhere I have worked in the last 10 years has had an embedded sysadmin/devop/SRE/PE of some sort. They are there to take the raw, difficult to use tools and adapt them so that developers are able to deploy without hand holding. This means evangelising metrics, showing how to manage alerting, coordinating standards and leading incident responses. The symptoms the writer describes are of a dysfunctional dev environment. Ideally your SRE should be able to rotate teams quickly. That means standards. Once a team is setup and running, they shouldn't really need ongoing SRE functions, unless they are making massive changes. This can only happen if developers are re-using tools, rather than making new wheels out of space age materials and getting bored and moving onto new things.
karmakaze 5y ago
To expand on the brief title > Google [...] says the SRE model [...] does not scale with microservices. Instead, they go on to describe a more tractable, framework-oriented model to address this through things like codified best practices, reusable solutions, standardization of tools and patterns Basically anyone planning on microservices should define and monitor bounds on which frameworks, tools and diversity of design patterns in use. Good advice at any scale.

162 points by kiyanwang 5y ago

Microservices should be used as an optimization. Not as a starting point.

Isn't this more of a comment about microservices than it is about SRE? It reads to me like "once you hit a number of microservices it ends up looking like a monolith": http://highscalability.com/blog/2020/4/8/one-team-at-uber-is-moving-from-microservices-to-macroservic.html http://highscalability.com/blog/2020/4/8/one-team-at-uber-is...

willcipriano 5y ago
It's always microservices, whatever you build. You just decide what kind of IPC you use between some of the pieces.
wara23arish 5y ago
dumb question time but what exactly makes something a micro service. Is the separation of a specific functionality from a wider array of functions to its own vm make it a microservice? When does something stop being a microservice i guess?
- notyourday 5y ago
  > dumb question time but what exactly makes something a micro service. This leftpad as a service, over HTTPS
  - wara23arish 5y ago
    I had to google that to understand the context. What a wild story
- forty 5y ago
  I remember asking a candidate whether they were doing microservices at her current job. She answered "I don't know if we have microservices, but we do have services that don't do much" It's since then that's my definition of a microservice :)
  - rhizome 5y ago
    Productized class methods.
- thecleaner 5y ago
  My definition is separation of infrastructure and deployment cycles. Everything that always in one deployment is one service or stuff thats part of your code-base is definitely not a different service.
- igetspam 5y ago
  It stops being a microservice when a developer starts saying, "oh! We can do X in service Y too! It already does ${similar work} and reads/writes from/to ${data source}, so why not?" The intended model is to do one thing, thus enabling surgical changes to functionality without having to rebuild everything. As long as you stick to your API contracts, you can muck around with the internals without effecting anything else.
  - chousuke 5y ago
    That's not really answering the question. How do you define what the "one thing" that the service handles actually is? I have this half-serious opinion that interface design is the hard (technical) part of software development, and the rest is mostly easy.
    - wara23arish 5y ago
      I remember coming to the same realization when the Google Vs Oracle court case was going on. Which decided that API’s are not copyrightable. To me it felt like the wrong decision, but what do i know.
      - pjc50 5y ago
        Copyright in interfaces entrenches monopoly.
      - chousuke 5y ago
        I would definitely argue that public interfaces should not be copyrightable, since you need to be able to freely reimplement them to interoperate, which in my opinion should be preferred over granting even temporary monopolies to software interfaces. However, public interfaces are only a small part of software interfaces in total. It's getting the system's internal interfaces right that takes most of the work.
        wara23arish 5y ago
        I totally get it from the sense of avoiding monopolies for sure. In my view, I was lumping internal/public interfaces in the same basket. I haven’t really considered public vs internal aspect of this.
- chippiewill 5y ago
  I think it's the same line of thinking as RISC and CISC (reduced/complex instruction set computer). There's no such thing as a RISC machine or a CISC machine, all computer ISAs live on a spectrum that can be described as RISC-like or CISC-like. There's no such thing as a micro service or a macro service (in terms of there being a perfect delineation). They're simply terms to describe certain design goals. Any given service might have several qualities that can be described as befitting a macro or a micro service. It's all too easy to fall into the no true scotsman trap.
- deleted 5y ago
  [deleted]
- jayd16 5y ago
  I would say its best to only think about any deployable thing as a service. "Microservices" is just the idea that those things do not have a minimum size. Essentially, "is this enough functionality to make a new service?" is the wrong question.
- deleted 5y ago
  [deleted]
- rmbyrro 5y ago
  A microservice is deployed independently, has its own hardware and can scale independently from the rest of the system. If there are multiple services and they all share hardware, they're not really microservices.
  - alexeldeib 5y ago
    > A microservice is deployed independently, has its own hardware and can scale independently from the rest of the system. Nicely articulated > If there are multiple services and they all share hardware, they're not really microservices. You just described most microservice deployments on Kubernetes ;)
    - rmbyrro 5y ago
      In this case, they run in containers, so they can actually scale independently. For this purpose, it's as if they were running on independent hardware. Even if we had separate cloud servers - such as EC2 on AWS - for each service, technically they are not really independent hardware. But for the purposes of deploying, hardware capacity allocation and scaling, you can see them as such. The same goes for containers orchestrated by K8s.
  - cnst 5y ago
    Don't we have a term for that — software coupling? A microservice architecture has weak coupling.
    - anton-107 5y ago
      it's coupling via network calls instead of in-memory calls
- snug 5y ago
  I like to think of it more of a separation of components needed to do the job. Need a database, use a database as it's own service, don't couple the database to the app, so if you need to scale the app horizontally you can add more replicas. If you need to scale the database vertically, you can scale the database vertically. If these are coupled, you scale only whats needed. On top of that use the language or tool that is meant for the job. If you need something written in go, build your service with go, if you need nodejs, do it with node, etc. Microservices don't need to be micro or small
iamstupidsimple 5y ago
Forgive me but aren't 'macroservices' just... services? I don't see the difference.
- steveBK123 5y ago
  unless its a gigaservice

pram 5y ago

“Hiring experienced, qualified SREs is difficult and costly. Despite enormous effort from the recruiting organization, there are never enough SREs to support all the services that need their expertise.” Uh huh. Maybe what isn't scaling is their onerous recruiting filters.

bostonsre 5y ago
It's good to be an sre. From my experience, there is high demand, low supply and schools don't really teach for sre roles so the pipeline stays small.
- outworlder 5y ago
  Yes. And then there's the unrealistic demands from companies that didn't even understand what DevOps was supposed to be, and are misunderstanding SRE even more. Many companies still treats this the same way they did "ops".
anonporridge 5y ago
A different element of this is that software professionals are often paid so well that a lot of people can take early retirement relatively easily if they're good about saving and investing. And the FIRE movement is only growing. I sometimes wonder if the software world has relatively greater 'leakage' via early retirement than other fields, creating a constant problem of not enough highly experienced people who remain stuck as wage slaves throughout a 40 year career.
zaptheimpaler 5y ago
Listen, if you can't write a working text justification algorithm in 45 minutes a decade after college, you have no business doing completely unrelated SRE stuff like monitoring services or scaling databases. Everybody knows leetcode is the key to software. If you can't do that, maybe you just don't have the right IQ to build the next chat app @ Google.
- GauntletWizard 5y ago

klodolph 5y ago

It sounds like the lesson here is that tacking on SRE to an out-of-control development process, churning out new services by the boatload, doesn't scale. This is caused by the typical attitude software companies have towards development. The most common model for software development is simple... rush to push out new features to the market, and pay the costs later--and then everyone is balking at the costs. The solution is kind of brilliant, IMO. You don't have to pay technical debt on projects if you shut them down and delete the code from your repository. Migrate to industry-standard solutions. Use off-the-shelf programs & libraries. Delete all your custom stuff. Replace good solutions with "good enough" solutions. The SREs can help you with that, but they can't help with out-of-control development. As your code base gets larger, the cost of supporting that code base gets larger too. The difficulty of scaling your SRE to match development reflects your out-of-control development process, not a problem with SRE. Keep the costs under control by keeping your code base under control.

lykr0n 5y ago
Yep. SRE is not a substitute for high level, overarching architects and designers. One pattern I see is that, as the company grows the development gets split into different product groups which will organically diverge unless there is rigid enforcement of design patterns. In some places, SRE does this implicitly because they will only support X, Y, or Z but in others each product group will have their own group of SREs. There becomes a point when you need one or a small group of people who are the opinionated developers who can make design decisions and who have the authority to cause everyone else to course correct. If you don't have this, you'll wind up with long migrations and legacy stuff that never seems to go away.
- colechristensen 5y ago
  When you get very large you need both enterprise wide SREs that are responsible for consulting and approving architecture for reliability purposes AND localized per service or other small breakdown SREs to support small sets of services. How you break this down is tricky and there is definitely overlap. Crappy attitudes towards support and reliability don’t scale, what you can get away with a few people keeping things barely held together stops working as you grow.

igetspam 5y ago

> Google enforces standards and opinions around things like programming languages, instrumentation and metrics, logging, and control systems surrounding traffic and load management. I think the author read this as more of a problem than a solution. This concept is supported by the DevOps model too. Your infrastructure is just as much a part of your product and the teams providing the infrastructure just as responsible for the service levels and API contracts as any customer facing product team.

gautamdivgi 5y ago

> And that move to microservices—in combination with cloud—unleashes a whole new level of autonomy and empowerment for developers who, often coming from a more restrictive ops-controlled environment on prem, introduce all sorts of new programming languages, compute platforms, databases, and other technologies. You need standards, without that SRE is pointless. Everything needs a standard method of monitoring. As an e.g. - stick to Java/Spring Boot, MariaDB and K8S. That will generally cover 85% of your use cases. The automation and advantage of SRE is derived through standards and familiarity with the tool chain.

slaymaker1907 5y ago

I kind of disagree on the Cambrian explosion of languages being a bad thing. While it has downsides, it has a massive upside of discouraging stuff like crazy custom build pipelines. Big companies love to burn money and time by reinventing everything. Bigco isn't going to try and implement their own build system if they need to support a dozen languages.

high_5 5y ago
Microservices have a marvelous long-term potential to trash the service with a myriad of legacy and obsolete languages instead of just one. So you just have to run faster to stay in the same place.

kodah 5y ago

The thing that made Google SRE scale was in the fine print and covered in this article. SRE is a premium service, and in order to get the premium service you had to meet some criteria to show you weren't about to turn highly paid, highly knowledgeable folks into professional firefighters. During my time as a SRE at multiple organizations what I witnessed was executives that owned Operations teams renaming the team, paying for a Python course, and calling it "change". Operations Engineers have a culture unto themselves that will not be broken in a years time. A culture of firefighting and heroism as opposed to proactivity and dull standards. They're rarely, if ever, programmers which was a prerequisite for a SRE-SE and SRE-SWE. You don't need to be a master in logic to know that if you dress your old pig up with lipstick and a new title that it's still a pig. I agree with the article: read the SRE book and understand why they did what they did, and if you hire SRE-SE or SRE-SWE types, don't colocate them with Operations types or make them a 1:1 map with your dev teams that haven't earned them.

Cthulhu_ 5y ago
I heard about that; amongst other things, if the amount of times SRE gets pinged because of a software fault exceeds X amount, they basically give the pager back to the team that built the software. They aren't taking responsibility for reliability if the software is not reliable. I really appreciated working with a team of SRE types, it gave me a newfound appreciation for quality in software development. I remember one instance where a colleague in my team went up to the 'ops' team and wanted SSH access to a production server to check on some settings (environment variables, I believe). He thought it was a trivial thing, you know, "just lemme have a peek" kinda thing, but the ops team flat-out told him no, if he needs to print env vars, he can do it in his own code - we did continuous releases, a patch could be deployed within minutes if passing all the checks. I loved that the ops team had the mandate from higher up to say no to requests like this, and I found them a lot more professional than the software developers, including myself.
- kodah 5y ago
  Ideally what that developer was wanting should be a function of the platform. That's likely not in an operations teams scope because they're mostly just fire fighters. Also ideally in a situation like SOX where devs can't have access to production their preproduction environments share the same interface that production has and if their values differ that much is documented.

nateberkopec 5y ago

At some point, it feels like every company is basically re implementing a kind of internal Heroku. I feel like there’s a billion dollar business in putting that on rails for medium to large enterprises.

jacques_chester 5y ago
There are many products that aim to fill this niche. I worked on one for years (Cloud Foundry). There is a lot of money to be made, but it's surprisingly stony ground. Oblig. self-promoted thread: https://twitter.com/jacques_chester/status/1435738825347645444 https://twitter.com/jacques_chester/status/14357388253476454...

cletus 5y ago

It actually sounds more like what I'd expect: microservices don't scale. I've never been a fan of microservices for this simple reason: every boundary, every interface, every connection is friction and a potential source of bugs, performance issues and problems. It's now been several years since I worked at Google so I can't speak to the current state of things but when I was there, monoliths ruled the roost. It seems possible there was some experimentation with microservices given the rise of Docker and Kubernetes in the last few years. I'd be surprised if this made serious inroads into high-traffic core services. It's also worth mentioning that generally Google has (had?) SRE teams rather than SREs embedded on SWE teams. You'd never have one SRE supporting a service as there'd have to be enough to maintain a healthy oncall rotation (typically 8-12). I can't help but feel like the author is projecting his own experiences onto what Google has said here.

jatins 5y ago
I previously worked in a BigCo that did microservices and currently work at BigCo that leans towards monoliths. It's completely true that microservices makes development _very_ hard. Just being able to run things locally was very hard since instead of understanding how to build/run one service, you had to figure our how to build/run N services. Testing was another major headache. Monoliths help with technical aspects but organizationally they make your life hell and can severely limit your ability to move fast.
Cthulhu_ 5y ago
Microservices are a cargo cult thing. It tries to solve a problem that 99% of companies that apply it wish they had.

75dvtwin 5y ago

Hiring SREs is costly... Hiring narrowly focused people with significant pre-existing experience -- Is always costly. ... and I also think, it is practically always a wrong strategy... I am fundamentally against micro-managing labor pools by federal government. However, there are need to be economic incentives (including immigration policies) -- that make it more difficult for employers to hire for 'tool-centric' positions, unless those tools are very expensive physical devices (eg telescopes, quantum computers, etc). The incentives need to direct employers at training on the job (not at after-work online classes). When I see on HN's hire threads -- 'if you have Azure experience -- you will get on top of the pile' -- I cringe. This is absolutely ridiculous. Same pretty much with Ruby/Php Rust/C++ Haskell/Scala modalities of the same problem. Yes hire people who understand process/idioms/patters, but invest in the f..ing training -- if you need somebody 'yesterday' to help, get consultants -- and have them help and at the same time hire for full time roles -- and train your internal stuff (possibly, even, have them learn from the consultants). Afraid of people leaving after acquiring the highly thought-out skills ? -- implement meaningful compensation/retention policies that reflect effort that you spend on training, and risks that you are taking if a person who was just trained -- leaves.

winrid 5y ago
What you are saying is generalists are cheaper to hire and that you can train them. That's true. That's why, as a seller (worker) you should try to specialize in at least one area to sell yourself better.
- 75dvtwin 5y ago
  That's part of it. I am also saying that there needs to be a system of incentives, that then, produces, organizational/hiring practices that favor non-tool-specific employment process.
  - winrid 5y ago
    Well, for that your company has to admit it it doesn't have specific problems. Because if you do have specific problems, it's probably more efficient to hire people that can handle those.

stefanve 5y ago

To me SRE + Cloud means SRE is responsible for the cloud foundation/ platform but the software teams should be DevOps teams and they are responsible for the (micro)services they create. SRE helps out by providing the tools, libs, dashboard , general principles etc. So SRE in responsible for the health of Kubernetes, networking etc and providing libs, support and infra but the DevOps teams are responsible for the health of the services/ software that is running on the cloud platform. A DevOPs teams should not consist of developers and one ops person but the team as a whole is responsible for the stuff they create. They are on call just like SRE is, if something go's down SRE + DevOps work together to fix it

KaiserPro 5y ago

> this person typically just becomes the “whipping boy” for the team. That sounds like a culture problem to me. SRE is just a fancy name for a class of people that make stuff more reliable. However it appears the reader of this book forgot about the other part: making stuff easy to use. Everywhere I have worked in the last 10 years has had an embedded sysadmin/devop/SRE/PE of some sort. They are there to take the raw, difficult to use tools and adapt them so that developers are able to deploy without hand holding. This means evangelising metrics, showing how to manage alerting, coordinating standards and leading incident responses. The symptoms the writer describes are of a dysfunctional dev environment. Ideally your SRE should be able to rotate teams quickly. That means standards. Once a team is setup and running, they shouldn't really need ongoing SRE functions, unless they are making massive changes. This can only happen if developers are re-using tools, rather than making new wheels out of space age materials and getting bored and moving onto new things.

karmakaze 5y ago

To expand on the brief title > Google [...] says the SRE model [...] does not scale with microservices. Instead, they go on to describe a more tractable, framework-oriented model to address this through things like codified best practices, reusable solutions, standardization of tools and patterns Basically anyone planning on microservices should define and monitor bounds on which frameworks, tools and diversity of design patterns in use. Good advice at any scale.