7 ms·

How the hell does microservices 'auto-heals' ? Do they autogenerate bug fixes and patch themselves ?

DeathArrow 3y ago
Simple, if a service is not answering for a certain amount of time, a new Kubernetes pod is brought up and the old one is killed. :)
- fvdessen 3y ago
  That also works with monoliths ... Usually you would make your monolith stateless and distribute the incoming requests / events across many instances that can be spawned / killed depending on volume of requests and health status of instances.
  - tomthumb 3y ago
    The same logic holds true for micro services. Statelessness is the key. But most micro services implementations end up being distributed monoliths
  - philjohn 3y ago
    In a past job, the benefit of microservices was that some of the operations performed by the system were fare more CPU intensive than others - by having them in their own service that could be scaled independently led to lower overall hardware requirements, and made keeping latency of the other services sensible much easier.
    - treis 3y ago
      You can scale monoliths independently too. Depending on the language that means paying some additional memory overhead for unused code but practically it's small compared to the typical amount of ram on a server these days.
      - ir123 3y ago
        No, this is not possible always. - consider case where the task is CPU intensive but not so critical as to eat into other parts of the code - consider the case where the task needs some data loaded for it to work. I don't think it is a good idea to have that data loaded into the monolith.
      - RicDan 3y ago
        Please elaborate on that. Without spawning a new monolith, how do you scale it. Add more resources?
        charcircuit 3y ago
        You run another instance of it.
        RicDan 3y ago
        So you have to take care of routing etc. I see how it works, and I completely agree that to start out, so going from PoC to first business implementation, a monolith is the way to go (unless the goal from the start is 100 million concurrent users I guess). But after that initial phase, does it really matter if you use one or the other? You can overengineer both and make them a timesink, or you can keep both simple. I do agree on things like network latency adding up, but being able to completely isolate business logic seems like a nice gain. But Im also not talking about real micro level (i.e. auth login and registration being different services), but more macro (i.e. security is one, printing another(pdf,csv,word etc), BI another one
        fvdessen 3y ago
        Interestingly Meta went with a django monolith for their new app and their goal was definitely in the order of 100 million concurrent users.
        RicDan 3y ago
  - jameshart 3y ago
    When you kill a monolith you kill a random selection of inflight tasks from every part of your application. So a rare bug in your mailing list signup workflow that hangs the process and causes it to be killed causes a random selection of inflight webpage requests, payment transactions, message handlers and business processes to fail. And if those failures aren’t all cleanly handled, your mailing list signup bug could propagate into a much wider issue. Whereas if you have a ‘mailing list service’ that has its own processes that can be killed and respawned, that bug only takes our mailing list processing. Which is good because the bug was probably made by the team who owns mailing list processing. And they can roll back their code and be on their way, with nobody else needing to know or care.
    - mostlysimilar 3y ago
      The original statement was "service is not answering for a certain amount of time". If the instance of your monolith is not responding you're probably already in a bad state and can reasonably kill it.
      - joshuamorton 3y ago
        Sure, but "if the instance of your monolith is not responding" probably means the app is down. That's only going to be true for a small subset of the microservices.
      - jamjamjamjamjam 3y ago
        What are you monitoring your monolith for? For microservices you can monitor specific metrics related to the exact function, and perform health checks, scaling events accordingly. For monoliths you cant be as specific. “Is the response a 500” doesn’t really cut it. “Average request latency” for scaling doesn’t cut it when some of your queries are reads and then some are completely unrelated mass joins.
    - lightbendover 3y ago
      Generally when people argue in favor of a monolith over micro-services, it's not for completely/mostly isolated business functions (i.e. BI pipelines vs. CMS CRUD), it's more for when responding to a single request of group of requests already implicates many services that must all work together to respond to the request(s); in this case, you're still smoked if any one of those services handling a part of the request chokes, you're in fact multiplying your opportunities for failure if you're using micro-services. Monoliths should be stateless (if achievable) and have no concept of partial success in cases where you would like atomicity unless everything is truly idempotent (easier said than achieved). If those criteria are met then callers just need to retry in the event of failure which can be set up for basically free in most frameworks. If you're pushing fatal recurring bugs into production, then that is a separate problem wider than the scope of a monolith vs. micro.

Simple, if a service is not answering for a certain amount of time, a new Kubernetes pod is brought up and the old one is killed. :)

fvdessen 3y ago
That also works with monoliths ... Usually you would make your monolith stateless and distribute the incoming requests / events across many instances that can be spawned / killed depending on volume of requests and health status of instances.
- tomthumb 3y ago
  The same logic holds true for micro services. Statelessness is the key. But most micro services implementations end up being distributed monoliths
- philjohn 3y ago
  In a past job, the benefit of microservices was that some of the operations performed by the system were fare more CPU intensive than others - by having them in their own service that could be scaled independently led to lower overall hardware requirements, and made keeping latency of the other services sensible much easier.
  - treis 3y ago
    You can scale monoliths independently too. Depending on the language that means paying some additional memory overhead for unused code but practically it's small compared to the typical amount of ram on a server these days.
    - ir123 3y ago
      No, this is not possible always. - consider case where the task is CPU intensive but not so critical as to eat into other parts of the code - consider the case where the task needs some data loaded for it to work. I don't think it is a good idea to have that data loaded into the monolith.
    - RicDan 3y ago
      Please elaborate on that. Without spawning a new monolith, how do you scale it. Add more resources?
      - charcircuit 3y ago
        You run another instance of it.
        RicDan 3y ago
        So you have to take care of routing etc. I see how it works, and I completely agree that to start out, so going from PoC to first business implementation, a monolith is the way to go (unless the goal from the start is 100 million concurrent users I guess). But after that initial phase, does it really matter if you use one or the other? You can overengineer both and make them a timesink, or you can keep both simple. I do agree on things like network latency adding up, but being able to completely isolate business logic seems like a nice gain. But Im also not talking about real micro level (i.e. auth login and registration being different services), but more macro (i.e. security is one, printing another(pdf,csv,word etc), BI another one
        fvdessen 3y ago
        Interestingly Meta went with a django monolith for their new app and their goal was definitely in the order of 100 million concurrent users.
        RicDan 3y ago
- jameshart 3y ago
  When you kill a monolith you kill a random selection of inflight tasks from every part of your application. So a rare bug in your mailing list signup workflow that hangs the process and causes it to be killed causes a random selection of inflight webpage requests, payment transactions, message handlers and business processes to fail. And if those failures aren’t all cleanly handled, your mailing list signup bug could propagate into a much wider issue. Whereas if you have a ‘mailing list service’ that has its own processes that can be killed and respawned, that bug only takes our mailing list processing. Which is good because the bug was probably made by the team who owns mailing list processing. And they can roll back their code and be on their way, with nobody else needing to know or care.
  - mostlysimilar 3y ago
    The original statement was "service is not answering for a certain amount of time". If the instance of your monolith is not responding you're probably already in a bad state and can reasonably kill it.
    - joshuamorton 3y ago
      Sure, but "if the instance of your monolith is not responding" probably means the app is down. That's only going to be true for a small subset of the microservices.
    - jamjamjamjamjam 3y ago
      What are you monitoring your monolith for? For microservices you can monitor specific metrics related to the exact function, and perform health checks, scaling events accordingly. For monoliths you cant be as specific. “Is the response a 500” doesn’t really cut it. “Average request latency” for scaling doesn’t cut it when some of your queries are reads and then some are completely unrelated mass joins.
  - lightbendover 3y ago
    Generally when people argue in favor of a monolith over micro-services, it's not for completely/mostly isolated business functions (i.e. BI pipelines vs. CMS CRUD), it's more for when responding to a single request of group of requests already implicates many services that must all work together to respond to the request(s); in this case, you're still smoked if any one of those services handling a part of the request chokes, you're in fact multiplying your opportunities for failure if you're using micro-services. Monoliths should be stateless (if achievable) and have no concept of partial success in cases where you would like atomicity unless everything is truly idempotent (easier said than achieved). If those criteria are met then callers just need to retry in the event of failure which can be set up for basically free in most frameworks. If you're pushing fatal recurring bugs into production, then that is a separate problem wider than the scope of a monolith vs. micro.