7 ms·

I'd be interested in a related "microservices failure stories". Must be a big overlap with this.

by tnolet 8y ago

coredog64 8y ago
Microservices failure stories? “All of them. The End”
- tnolet 8y ago
  I'm not convinced all microservices ventures are failures. Having worked in the space a bit as a founder/CTO of a vendor in that space I've just seen many examples of misguided attempts due to fashion / CV-driven development / hype driven development. The pattern is valid, just not for everyone at every time.
  - hjacobs 8y ago
    ThoughtWorks even has a term for it: "Microservice envy" https://www.thoughtworks.com/radar/techniques/microservice-envy https://www.thoughtworks.com/radar/techniques/microservice-e...
  - ex_amazon_sde 8y ago
    The good part of "microservices" is "services". The bad part is "micro".
    - tyldum 8y ago
      When the definition turns to some kind of dogma, then it will fail.
- lucidone 8y ago
  I'm consulting on a micro services back end right now with mostly prior experience with monoliths. What is the selling point that drives companies down this direction? It's insane, and my client keeps trying to hire new developers and bring on more consultants to build this thing, but the amount of knowledge required is more than any one person can handle. I have similar issues with their choice of db (nosql) and its inflexibility.
  - stingraycharles 8y ago
    The biggest reason for choosing microservices _should_ be scaling development teams: microservices allow multiple teams to work on different code bases, without stepping on each other's toes. What actually happens, tough, is that (uninformed) people choose it because they think it brings them scalability (wrong), it's more cloud compatible (wrong) or the worst offender, it's more modern.
    - jordanbeiber 8y ago
      If you achieve this scale of teams suggested, it means you have a lot of implicit gains here as well, right? Such as higher quality, higher "velocity", separation of concern and hopefully a clear sense of ownership. These are some of the things in my opinion that allows you to scale.
      - lmm 8y ago
        > If you achieve this scale of teams suggested, it means you have a lot of implicit gains here as well, right? > Such as higher quality, higher "velocity", separation of concern and hopefully a clear sense of ownership. No, it means you pay a huge overhead. Quality and velocity both drop as your day-to-day development requires a lot more setup and faff to do anything, and counterintuitively so does separation of concerns as your interfaces become more rigid. Small organisations should do things that don't scale, turn their size into an advantage. If you think of your overhead as ax + bx^2 where x is the number of developers, microservices are a way to reduce b, but at the cost of a big increase to a. It makes sense when x is huge but not before. My litmus test would be: do you need to do multiple (unrelated) deployments of different services at once? If your organisation is small enough that you can get away with only deploying one thing at a time, you'll probably have less overhead if you work without microservices.
        jordanbeiber 8y ago
        Which is why you’ll need the input and help from experienced sysadmins that like to do development, which actually is ”automation”. This done right reduces overhead and gives you an edge in repeatability and quality. Both are required to scale.
- somethingle234 8y ago
  I worked in a company where micro services succeed. To succeed, the ratio has to make sense. We had more than 2,000 engineers with fewer than a few hundred services. So, of course, it made perfect sense. I've also seen a startup where it had 30 people and used the micro-service architecture. That didn't work very well. To make it worse, they used Node.js. It's like using Micro service wasn't difficult enough... they had to go with a dynamic-typed language.
dehrmann 8y ago
I have two. One was caused by data inconsistency between services and regions. One is more hypothetical: the microservices had gotten to the point that no one knew how to start the system if all services are down, and it's possible that services have circular dependencies to the point that it would be incredibly hard to do a cold start.
- nicobn 8y ago
  I've actually seen your hypothetical in action, but the bug was even more subtle. Assume service A, B and C. A and C both need information from each other which is usually cached. Normally, you'd deploy one service at a time so the call chain would go A -> B -> C -> A or A -> C then A -> B -> C but in this particular instance, A and C's caches were cold, causing an explosion of service calls that took both services down.
  - quickthrower2 8y ago
    > A and C both need information from each other Sounds like a monolith pulled apart :-)
- parasubvert 8y ago
  I’d think the best way to handle a cold start is to have each microservice fail fast and be wrapped in a supervisor. They will converge to uptime. Unless you have a truly circular dependency, at which point they probably should be collapsed into a single service.
- kronin 8y ago
  You handle the hypothetical by validating a new environment can be built and bootstrapped. Doing this on a regular basis, either by tearing down and rebuilding dev, or in a separate environment just for this purpose, is not a "nice to have". This same problem exists with monoliths with complicated dependencies, nothing new here. Just like backups, if infrastructure as code isn't tested, it's worthless.
- sgt101 8y ago
  I think that if there is a genuine circular dependency then the services won't start ever. But I think it is possible to introduce services that assume other services are up and have an apparent dependency circularity. The trick is to have all your services resilient to it's start requirements not being met - basically the service has to back off and wait if information it needs isn't yet in the environment... and then ask again - so that when other services are up everything syncs and comes up.

I'd be interested in a related "microservices failure stories". Must be a big overlap with this.

by tnolet 8y ago

I'd be interested in a related "microservices failure stories". Must be a big overlap with this.

coredog64 8y ago

Microservices failure stories? “All of them. The End”

tnolet 8y ago
I'm not convinced all microservices ventures are failures. Having worked in the space a bit as a founder/CTO of a vendor in that space I've just seen many examples of misguided attempts due to fashion / CV-driven development / hype driven development. The pattern is valid, just not for everyone at every time.
- hjacobs 8y ago
  ThoughtWorks even has a term for it: "Microservice envy" https://www.thoughtworks.com/radar/techniques/microservice-envy https://www.thoughtworks.com/radar/techniques/microservice-e...
- ex_amazon_sde 8y ago
  The good part of "microservices" is "services". The bad part is "micro".
  - tyldum 8y ago
    When the definition turns to some kind of dogma, then it will fail.
lucidone 8y ago
I'm consulting on a micro services back end right now with mostly prior experience with monoliths. What is the selling point that drives companies down this direction? It's insane, and my client keeps trying to hire new developers and bring on more consultants to build this thing, but the amount of knowledge required is more than any one person can handle. I have similar issues with their choice of db (nosql) and its inflexibility.
- stingraycharles 8y ago
  The biggest reason for choosing microservices _should_ be scaling development teams: microservices allow multiple teams to work on different code bases, without stepping on each other's toes. What actually happens, tough, is that (uninformed) people choose it because they think it brings them scalability (wrong), it's more cloud compatible (wrong) or the worst offender, it's more modern.
  - jordanbeiber 8y ago
    If you achieve this scale of teams suggested, it means you have a lot of implicit gains here as well, right? Such as higher quality, higher "velocity", separation of concern and hopefully a clear sense of ownership. These are some of the things in my opinion that allows you to scale.
    - lmm 8y ago
      > If you achieve this scale of teams suggested, it means you have a lot of implicit gains here as well, right? > Such as higher quality, higher "velocity", separation of concern and hopefully a clear sense of ownership. No, it means you pay a huge overhead. Quality and velocity both drop as your day-to-day development requires a lot more setup and faff to do anything, and counterintuitively so does separation of concerns as your interfaces become more rigid. Small organisations should do things that don't scale, turn their size into an advantage. If you think of your overhead as ax + bx^2 where x is the number of developers, microservices are a way to reduce b, but at the cost of a big increase to a. It makes sense when x is huge but not before. My litmus test would be: do you need to do multiple (unrelated) deployments of different services at once? If your organisation is small enough that you can get away with only deploying one thing at a time, you'll probably have less overhead if you work without microservices.
      - jordanbeiber 8y ago
        Which is why you’ll need the input and help from experienced sysadmins that like to do development, which actually is ”automation”. This done right reduces overhead and gives you an edge in repeatability and quality. Both are required to scale.
somethingle234 8y ago
I worked in a company where micro services succeed. To succeed, the ratio has to make sense. We had more than 2,000 engineers with fewer than a few hundred services. So, of course, it made perfect sense. I've also seen a startup where it had 30 people and used the micro-service architecture. That didn't work very well. To make it worse, they used Node.js. It's like using Micro service wasn't difficult enough... they had to go with a dynamic-typed language.

dehrmann 8y ago

I have two. One was caused by data inconsistency between services and regions. One is more hypothetical: the microservices had gotten to the point that no one knew how to start the system if all services are down, and it's possible that services have circular dependencies to the point that it would be incredibly hard to do a cold start.

nicobn 8y ago
I've actually seen your hypothetical in action, but the bug was even more subtle. Assume service A, B and C. A and C both need information from each other which is usually cached. Normally, you'd deploy one service at a time so the call chain would go A -> B -> C -> A or A -> C then A -> B -> C but in this particular instance, A and C's caches were cold, causing an explosion of service calls that took both services down.
- quickthrower2 8y ago
  > A and C both need information from each other Sounds like a monolith pulled apart :-)
parasubvert 8y ago
I’d think the best way to handle a cold start is to have each microservice fail fast and be wrapped in a supervisor. They will converge to uptime. Unless you have a truly circular dependency, at which point they probably should be collapsed into a single service.
kronin 8y ago
You handle the hypothetical by validating a new environment can be built and bootstrapped. Doing this on a regular basis, either by tearing down and rebuilding dev, or in a separate environment just for this purpose, is not a "nice to have". This same problem exists with monoliths with complicated dependencies, nothing new here. Just like backups, if infrastructure as code isn't tested, it's worthless.
sgt101 8y ago
I think that if there is a genuine circular dependency then the services won't start ever. But I think it is possible to introduce services that assume other services are up and have an apparent dependency circularity. The trick is to have all your services resilient to it's start requirements not being met - basically the service has to back off and wait if information it needs isn't yet in the environment... and then ask again - so that when other services are up everything syncs and comes up.