6 ms·

Does anyone have any reading material on the reliability of systems that use microservices? I've had a bit of basic probability rattling around in the back of m

Does anyone have any reading material on the reliability of systems that use microservices? I've had a bit of basic probability rattling around in the back of my head that makes me suspect microservices are in general less reliable. I'd be interested in seeing a real-world analysis. My thinking goes like this, with some simplifying assumptions. Let's say you have a monolith with 99% uptime that you rearchitect into 5 microservices, each with 99% uptime, and if any one of those services goes down your whole system is down. Let's also assume for the sake of simplicity that these microservices are completely independent, although they are almost assuredly not. From basic probability, 99% uptime means there is some chunk of time t for which P(monolith goes down) = 1%. But P(microservice system goes down) = P(service A down or service B down or...) = P(service A down) + P(service B down) + ... = 5% In reality P(microservice system goes down) < 5% because they aren't independent and the chunk of time in which service A can go down will overlap that of service B. But still, that means the upper bound of the whole system going down is higher than for a monolith. But microservices are pretty popular, and I'm sure someone has thought along these lines before. One potential rebuttal is that each microservice is in fact more reliable than the monolith, although from what I've seen in my career I am skeptical that's truly the case. Where's the hole in my reasoning? (Or maybe I'm right. That would be fine too.)

dmattia 3y ago
In general, one of the goals of microservices should be that if one of the five services goes down, the other four should be able to operate in some capacity still. In practice, this can make the math quite a bit messier, but I don't think it necessarily has been worse overall from my perspective. So instead of having your system be up or down 99% of the time in a monolith, you'll have it fully up 95% of the time (using your numbers), but of that 5% of downtime, 20% of the time one of your products will be running slowly, or 10% of the time some new feature you launched won't work for specific customers in some specific region, etc. At my company it makes things like SLA/SLO guarantees for "our services" pretty complicated in that it's hard to define what uptime truly means, but overall I think the five microservice approach, when done well, should have less than 1% of complete downtime, at the cost of more partial downtime
- VyseofArcadia 3y ago
  > In general, one of the goals of microservices should be that if one of the five services goes down, the other four should be able to operate in some capacity still. This is an excellent point, but what brought this to my mind was that the microservices in the Netflix article I don't think have this property. It looks to me if any of the VIS, CAS, LGS, or VES go down, then the whole service is effectively down. Indeed, in my own career what I've seen is that if one microservice goes down the user won't be seeing 500 errors or friends, but the service will be completely useless to the user. You've just gone from a hard error to a spinning load icon, which might in fact be an even worse user experience. It could be argued that this is just "you're doing microservices wrong", but then we start getting into no true Scotsman territory.
  - geodel 3y ago
    > Indeed, in my own career what I've seen is that if one microservice goes down the user won't be seeing 500 errors or friends Exactly what it does is that first few hours of triage call goes with people claiming "well my service is up and issue is somewhere else". So find which service failed itself take crucial hours instead of fixing the failing service. But in a world where Micro Service Incident Commanders can pinpoint failing a service among 1000 micro service within seconds on their vast 80 inch monitoring consoles and direct resolution admirals to fix in next 15 mins. It might just all work fine.
    - fragmede 3y ago
      the problem comes when it's a distributed system, and it's the interaction between multiple systems that's causing the problem, and not a specific microservice being down. something got upgraded and the message size changed in an unexpected and incompatible way that worked fine in testing.
  - threeseed 3y ago
    > It looks to me if any of the VIS, CAS, LGS, or VES go down, But the whole point is that by splitting it into micro-services you can efficiently and optimally scale each component individually. So it's extremely rare that VIS for example would entirely go down. And because Netflix has tools like Hystrix if one instance is unavailable it will seamlessly route to another one. And Even if you push bad code there are techniques like blue/green and canary releases which can be used.
jolux 3y ago
I don’t know if our architecture truly qualifies as microservices but in my experience one of the advantages is that the system is able to limp along in a degraded state much more effectively when one service goes down whereas it’s a lot easier to bring the whole system to its knees with a single change in a monolith. This suggests an addition to your model which is that not all outages are equally costly.
bick_nyers 3y ago
I'm not an expert in this discussion by any means, but my two cents is that collaboration on larger teams is easier on the microservice path than the monolith path. I would expect that over the course of years, the uptime and feature velocity of the monolith will decline at a rate faster than that of microservices, if a monolith has 99% reliability, then that inflection point might take a while to occur due to the calculation that you provided. However, as a company you might be willing to go from 99% to 95% reliability (or 3 9's to 2 9's) to double the speed of feature development. To my understanding, microservices are primarily implemented as an architecture for collaboration due to the inherent inefficiencies and communication difficulties of large teams. This is why they are often not recommended unless you could be categorized as "Big Tech".
- VyseofArcadia 3y ago
  Anecdata incoming. This was when I was working in "Big Tech". When I was in web dev, my experience was that there was actually no good separation, and collaboration with microservices was in fact more difficult. Every single feature I worked on required changes across several microservices, and having Team A run service A and Team B run service B etc. just meant that I had to get buy in from every team. My team was easy enough to work with, but then for the work needed on service B I would have to learn and start using their processes, attend their standups and meetings in addition to mine, and so on. Frankly it was a nightmare. But in a monolith, the same work is usually just a few quick arguments during code reviews. Maybe inviting a few extra people to design reviews. In my experience, microservices just make teams more territorial.
zmgsabst 3y ago
Your assumptions elide the benefits: - micro services don’t always block the pipeline; often the failing one can catch up later - scaling can happen for each micro service - removing faulty components from the main path means the key services are less likely to crash - you haven’t explained why feature X is more likely to crash in a micro service than a monolith, eg, you’re assuming components A and B have 0.5% crash in a monolith but 1% when run independently Your model ignores that most of your crashes come from the same code paths between the two models; only a small contribution is to the crash is from hosting.
geodel 3y ago
I did about something same. If there are 5 services each can fail independently (unlikely but lets assume) and uptime is 99%. Then P(All up at same time) = 0.99^5 which gives any one is down at any moment is 1- 0.99^5 ~5%. So this increase the failure 5 times of original 1% downtime. And with 100s of micro services in overall architecture with some indirect connections between many of them I think this number could possibly go much higher. Further at least where I work it is clearly that failure rate is higher than 5%. But with cottage industry of observability tools, cloud native solutions ..blah..blah.. telling basic maths to people in responsible positions is sure shot way to get fired. I am already being marked as someone opposed to progress so I can basically take my statistics and shove up mine. There is million times more data about reliability of micro services and they all can't be wrong.
com2kid 3y ago
> My thinking goes like this, with some simplifying assumptions. Let's say you have a monolith with 99% uptime that you rearchitect into 5 microservices, each with 99% uptime, and if any one of those services goes down your whole system is down. Let's also assume for the sake of simplicity that these microservices are completely independent, although they are almost assuredly not. I've worked on Microservices at HBO. IIRC over 2 years my team's multiple services only had 1 complete outage, and 2 or 3 impactful incidents. Also a nice benefit of Microservices is that you can shove queues and retry logic between services, and replay messages later on when a service is back up. Obviously not appropriate for anything that needs to give real time results, but there are a surprising number of features that don't mind a 30 second or even 5 minute delay so long as success is guaranteed eventually.
trevor-e 3y ago
The flaw in your logic/math is assuming that the resulting 5 microservices each have the same 99% uptime as the original monolith. In practice some microservices are much simpler and therefore more reliable than others, especially when broken out.
- andy_ppp 3y ago
  Well, even that depends, the overhead to do microservices well is a lot - versioning of what services are deployed and guarantees around compatibility between said deployments can be a huge amount of work. I’ve seen systems where 2 month old containers are just floating around still being sent incompatible messages. Then there’s teaching developers about CAP theory properly, at Netflix great everyone is sharp enough to get it but at AverageCompanyX, well my experience is 30% of developers actually can’t reason about eventual consistency. Therefore I would question the assumption that things are simpler, code certainly, infrastructure, debugging and deployment certainly not. I would say breaking things down into services, slowly, as it makes sense is enough. They don’t need to be micro.
  - trevor-e 3y ago
    Yea totally agreed with your points, a bunch of new problems arise that most orgs aren't equipped to solve and should probably stick with monolith. I was merely pointing out the probability part and how nobody would use microservices if it meant (.99 ^ 5) reliability, although to your point that can definitely happen at some places!
- tester756 3y ago
  >In practice some microservices are much simpler and therefore more reliable than others, especially when broken out. Much simpler in what way? You mean less close = less crash/segfault potential? If yes, then oh c'mon, modern stacks are incredibly reliable that they almost never crash. More microservices = more infra level stuff needed = WAY more potential problems
jameshart 3y ago
Several reasons this doesn’t pan out in practice. 1) Retries. When one replica of a microservice is down, the calling service can retry, get service from an up replica, and the outage is routed around 2) Queues. Microservices lend themselves to queue and worker patterns where downtime on individual services has less effect on overall service availability 4) Outages have narrower impact. One microservice losing access to its database breaks the functionality that relies on that microservice; other functionality runs fine. 5) Changes have smaller blast radius. Most outages are caused by changes; changes in monoliths that cause outages are more likely to take the whole system offline (eg stack overflows and infinite loops crash processes). Changes that cause outages in microservices can’t knock other services offline.

dmattia 3y ago

In general, one of the goals of microservices should be that if one of the five services goes down, the other four should be able to operate in some capacity still. In practice, this can make the math quite a bit messier, but I don't think it necessarily has been worse overall from my perspective. So instead of having your system be up or down 99% of the time in a monolith, you'll have it fully up 95% of the time (using your numbers), but of that 5% of downtime, 20% of the time one of your products will be running slowly, or 10% of the time some new feature you launched won't work for specific customers in some specific region, etc. At my company it makes things like SLA/SLO guarantees for "our services" pretty complicated in that it's hard to define what uptime truly means, but overall I think the five microservice approach, when done well, should have less than 1% of complete downtime, at the cost of more partial downtime

VyseofArcadia 3y ago
> In general, one of the goals of microservices should be that if one of the five services goes down, the other four should be able to operate in some capacity still. This is an excellent point, but what brought this to my mind was that the microservices in the Netflix article I don't think have this property. It looks to me if any of the VIS, CAS, LGS, or VES go down, then the whole service is effectively down. Indeed, in my own career what I've seen is that if one microservice goes down the user won't be seeing 500 errors or friends, but the service will be completely useless to the user. You've just gone from a hard error to a spinning load icon, which might in fact be an even worse user experience. It could be argued that this is just "you're doing microservices wrong", but then we start getting into no true Scotsman territory.
- geodel 3y ago
  > Indeed, in my own career what I've seen is that if one microservice goes down the user won't be seeing 500 errors or friends Exactly what it does is that first few hours of triage call goes with people claiming "well my service is up and issue is somewhere else". So find which service failed itself take crucial hours instead of fixing the failing service. But in a world where Micro Service Incident Commanders can pinpoint failing a service among 1000 micro service within seconds on their vast 80 inch monitoring consoles and direct resolution admirals to fix in next 15 mins. It might just all work fine.
  - fragmede 3y ago
    the problem comes when it's a distributed system, and it's the interaction between multiple systems that's causing the problem, and not a specific microservice being down. something got upgraded and the message size changed in an unexpected and incompatible way that worked fine in testing.
- threeseed 3y ago
  > It looks to me if any of the VIS, CAS, LGS, or VES go down, But the whole point is that by splitting it into micro-services you can efficiently and optimally scale each component individually. So it's extremely rare that VIS for example would entirely go down. And because Netflix has tools like Hystrix if one instance is unavailable it will seamlessly route to another one. And Even if you push bad code there are techniques like blue/green and canary releases which can be used.

jolux 3y ago

I don’t know if our architecture truly qualifies as microservices but in my experience one of the advantages is that the system is able to limp along in a degraded state much more effectively when one service goes down whereas it’s a lot easier to bring the whole system to its knees with a single change in a monolith. This suggests an addition to your model which is that not all outages are equally costly.

bick_nyers 3y ago

I'm not an expert in this discussion by any means, but my two cents is that collaboration on larger teams is easier on the microservice path than the monolith path. I would expect that over the course of years, the uptime and feature velocity of the monolith will decline at a rate faster than that of microservices, if a monolith has 99% reliability, then that inflection point might take a while to occur due to the calculation that you provided. However, as a company you might be willing to go from 99% to 95% reliability (or 3 9's to 2 9's) to double the speed of feature development. To my understanding, microservices are primarily implemented as an architecture for collaboration due to the inherent inefficiencies and communication difficulties of large teams. This is why they are often not recommended unless you could be categorized as "Big Tech".

VyseofArcadia 3y ago
Anecdata incoming. This was when I was working in "Big Tech". When I was in web dev, my experience was that there was actually no good separation, and collaboration with microservices was in fact more difficult. Every single feature I worked on required changes across several microservices, and having Team A run service A and Team B run service B etc. just meant that I had to get buy in from every team. My team was easy enough to work with, but then for the work needed on service B I would have to learn and start using their processes, attend their standups and meetings in addition to mine, and so on. Frankly it was a nightmare. But in a monolith, the same work is usually just a few quick arguments during code reviews. Maybe inviting a few extra people to design reviews. In my experience, microservices just make teams more territorial.

zmgsabst 3y ago

Your assumptions elide the benefits: - micro services don’t always block the pipeline; often the failing one can catch up later - scaling can happen for each micro service - removing faulty components from the main path means the key services are less likely to crash - you haven’t explained why feature X is more likely to crash in a micro service than a monolith, eg, you’re assuming components A and B have 0.5% crash in a monolith but 1% when run independently Your model ignores that most of your crashes come from the same code paths between the two models; only a small contribution is to the crash is from hosting.

geodel 3y ago

I did about something same. If there are 5 services each can fail independently (unlikely but lets assume) and uptime is 99%. Then P(All up at same time) = 0.99^5 which gives any one is down at any moment is 1- 0.99^5 ~5%. So this increase the failure 5 times of original 1% downtime. And with 100s of micro services in overall architecture with some indirect connections between many of them I think this number could possibly go much higher. Further at least where I work it is clearly that failure rate is higher than 5%. But with cottage industry of observability tools, cloud native solutions ..blah..blah.. telling basic maths to people in responsible positions is sure shot way to get fired. I am already being marked as someone opposed to progress so I can basically take my statistics and shove up mine. There is million times more data about reliability of micro services and they all can't be wrong.

com2kid 3y ago

> My thinking goes like this, with some simplifying assumptions. Let's say you have a monolith with 99% uptime that you rearchitect into 5 microservices, each with 99% uptime, and if any one of those services goes down your whole system is down. Let's also assume for the sake of simplicity that these microservices are completely independent, although they are almost assuredly not. I've worked on Microservices at HBO. IIRC over 2 years my team's multiple services only had 1 complete outage, and 2 or 3 impactful incidents. Also a nice benefit of Microservices is that you can shove queues and retry logic between services, and replay messages later on when a service is back up. Obviously not appropriate for anything that needs to give real time results, but there are a surprising number of features that don't mind a 30 second or even 5 minute delay so long as success is guaranteed eventually.

trevor-e 3y ago

The flaw in your logic/math is assuming that the resulting 5 microservices each have the same 99% uptime as the original monolith. In practice some microservices are much simpler and therefore more reliable than others, especially when broken out.

andy_ppp 3y ago
Well, even that depends, the overhead to do microservices well is a lot - versioning of what services are deployed and guarantees around compatibility between said deployments can be a huge amount of work. I’ve seen systems where 2 month old containers are just floating around still being sent incompatible messages. Then there’s teaching developers about CAP theory properly, at Netflix great everyone is sharp enough to get it but at AverageCompanyX, well my experience is 30% of developers actually can’t reason about eventual consistency. Therefore I would question the assumption that things are simpler, code certainly, infrastructure, debugging and deployment certainly not. I would say breaking things down into services, slowly, as it makes sense is enough. They don’t need to be micro.
- trevor-e 3y ago
  Yea totally agreed with your points, a bunch of new problems arise that most orgs aren't equipped to solve and should probably stick with monolith. I was merely pointing out the probability part and how nobody would use microservices if it meant (.99 ^ 5) reliability, although to your point that can definitely happen at some places!
tester756 3y ago
>In practice some microservices are much simpler and therefore more reliable than others, especially when broken out. Much simpler in what way? You mean less close = less crash/segfault potential? If yes, then oh c'mon, modern stacks are incredibly reliable that they almost never crash. More microservices = more infra level stuff needed = WAY more potential problems

jameshart 3y ago

Several reasons this doesn’t pan out in practice. 1) Retries. When one replica of a microservice is down, the calling service can retry, get service from an up replica, and the outage is routed around 2) Queues. Microservices lend themselves to queue and worker patterns where downtime on individual services has less effect on overall service availability 4) Outages have narrower impact. One microservice losing access to its database breaks the functionality that relies on that microservice; other functionality runs fine. 5) Changes have smaller blast radius. Most outages are caused by changes; changes in monoliths that cause outages are more likely to take the whole system offline (eg stack overflows and infinite loops crash processes). Changes that cause outages in microservices can’t knock other services offline.