4 ms·

I used to be monolith-curious, but what sold me on micro-services is the distribution of risk. When you work for a company where uptime matters having a regress

I used to be monolith-curious, but what sold me on micro-services is the distribution of risk. When you work for a company where uptime matters having a regression that takes down everything is not acceptable. Simply using separate services greatly reduces the chances of a full outage and justifies all other overhead.

habibur 4y ago
> Simply using separate services greatly reduces the chances of a full outage and justifies all other overhead. or maybe run redundant monolith fail over servers. should work the same as micro services.
- anthonypasq 4y ago
  then you need to hotfix you need to re build a giant monolith that probably has thousands of tests and a 20-30 minute regression suite easily.
  - therealdrag0 4y ago
    I have seen exactly this. Waiting for a 60+ min CI build during an outage is not a good look.
    - JAlexoid 4y ago
      However, there's another "bonus here" is that you have integration tests that have a better coverage. Microservices don't make builds radically faster for the majority. People still split systems into larger services.
yCombLinks 4y ago
How do microservices help here? You can deploy a monolith 10 times and have the same risk distribution.
- therealdrag0 4y ago
  It's not about replication, it's about compartmentalizing code changes / bugs / resource utilization. If you deploy a bug that causes a service crash loop or resource exhaustion isolating that to a small service reduces impact to other services. And if that service isn't core to app then the app can still function.
whiplash451 4y ago
Why would using microservices reduce the chance of outages? If you break a microservice that is vital for the system, you are as screwed as with a monolyth.
- lowbloodsugar 4y ago
  You have fifty (or 10,000) servers running your critical microservice in multiple AZs. You start a deployment to a single host. The shit hits the fan. You rollback that one host. If it looks fine, you leave it running for a few hours while various canaries and integration tests all hit it. If no red flags occur, you deploy another two, etc. You deploy to different AZs on different days. You can fail over to your critical service in different AZs because you previously ensured that the AZs are scaled so that they can handle that influx of traffic (didn't you?). You've tested that. And that is if it makes it to production. Here is your fleet of test hosts using production data and being verified against the output of production servers.
- therealdrag0 4y ago
  Sure, but not all micro-services are vital. If your "email report" service has a memory leak (or many other noisy-neighbor issues) and is in a crash loop then that wont take down the "search service" or the "auth service", etc. Many other user paths will remain active and usable. It compartmentalizes risk.
  - whiplash451 4y ago
    Proper design in a monolith would also protect you from failures of non-vital services (e.g. through exception capture). So it seems like we’re trying to compensate bad design with microservices. It’s orthogonal IMO.
    - therealdrag0 4y ago
      How does exception capture protect from all failures? The most obvious one I don't see it relating to is resource utilization, CPU, memory, threadpools, db connection pools, etc etc. > we’re trying to compensate bad design No I think we're trying to compensate for developer mistakes and naivety. When you have dozens to hundreds of devs working on an application many of them are juniors and all of them are human and impactful mistakes happen. Just catching the right exceptions and handling them the right way does not protect against devs not catching the right exceptions and not handling them the right way, but microservices does. Maybe you call that compensating for bad design, which is fair and in that case yes it is! And that compensation helps a large team move faster without perfecting design on every change.
lemmsjid 4y ago
If you have a truly modularized monolith, you can have a directed graph of dependent libraries, the leaves of which are different services that can start up. You can individually deploy leaf services and only their dependent code will go out. You can then rationalize which services can go down based on their dependency tree. If email is close to a root library, then yes a regression in it could bring everything down. If email is a leaf service, its code won’t even be deployed to most of the parallel services. You can then have a pretty flexible trade off between the convenience of having email be a rooted library against the trade off of keeping it a lead service (the implication being that leaf services can talk to one another over the network via service stubs, rest, what have you). This is SOA (Service Oriented Architecture), which should be considered in the midst of the microservice / monolith conversation.

habibur 4y ago

> Simply using separate services greatly reduces the chances of a full outage and justifies all other overhead. or maybe run redundant monolith fail over servers. should work the same as micro services.

anthonypasq 4y ago
then you need to hotfix you need to re build a giant monolith that probably has thousands of tests and a 20-30 minute regression suite easily.
- therealdrag0 4y ago
  I have seen exactly this. Waiting for a 60+ min CI build during an outage is not a good look.
  - JAlexoid 4y ago
    However, there's another "bonus here" is that you have integration tests that have a better coverage. Microservices don't make builds radically faster for the majority. People still split systems into larger services.

yCombLinks 4y ago

How do microservices help here? You can deploy a monolith 10 times and have the same risk distribution.

therealdrag0 4y ago
It's not about replication, it's about compartmentalizing code changes / bugs / resource utilization. If you deploy a bug that causes a service crash loop or resource exhaustion isolating that to a small service reduces impact to other services. And if that service isn't core to app then the app can still function.

whiplash451 4y ago

Why would using microservices reduce the chance of outages? If you break a microservice that is vital for the system, you are as screwed as with a monolyth.

lowbloodsugar 4y ago
You have fifty (or 10,000) servers running your critical microservice in multiple AZs. You start a deployment to a single host. The shit hits the fan. You rollback that one host. If it looks fine, you leave it running for a few hours while various canaries and integration tests all hit it. If no red flags occur, you deploy another two, etc. You deploy to different AZs on different days. You can fail over to your critical service in different AZs because you previously ensured that the AZs are scaled so that they can handle that influx of traffic (didn't you?). You've tested that. And that is if it makes it to production. Here is your fleet of test hosts using production data and being verified against the output of production servers.
therealdrag0 4y ago
Sure, but not all micro-services are vital. If your "email report" service has a memory leak (or many other noisy-neighbor issues) and is in a crash loop then that wont take down the "search service" or the "auth service", etc. Many other user paths will remain active and usable. It compartmentalizes risk.
- whiplash451 4y ago
  Proper design in a monolith would also protect you from failures of non-vital services (e.g. through exception capture). So it seems like we’re trying to compensate bad design with microservices. It’s orthogonal IMO.
  - therealdrag0 4y ago
    How does exception capture protect from all failures? The most obvious one I don't see it relating to is resource utilization, CPU, memory, threadpools, db connection pools, etc etc. > we’re trying to compensate bad design No I think we're trying to compensate for developer mistakes and naivety. When you have dozens to hundreds of devs working on an application many of them are juniors and all of them are human and impactful mistakes happen. Just catching the right exceptions and handling them the right way does not protect against devs not catching the right exceptions and not handling them the right way, but microservices does. Maybe you call that compensating for bad design, which is fair and in that case yes it is! And that compensation helps a large team move faster without perfecting design on every change.

lemmsjid 4y ago

If you have a truly modularized monolith, you can have a directed graph of dependent libraries, the leaves of which are different services that can start up. You can individually deploy leaf services and only their dependent code will go out. You can then rationalize which services can go down based on their dependency tree. If email is close to a root library, then yes a regression in it could bring everything down. If email is a leaf service, its code won’t even be deployed to most of the parallel services. You can then have a pretty flexible trade off between the convenience of having email be a rooted library against the trade off of keeping it a lead service (the implication being that leaf services can talk to one another over the network via service stubs, rest, what have you). This is SOA (Service Oriented Architecture), which should be considered in the midst of the microservice / monolith conversation.