7 ms·

My employer adopted microservices for a very specific reason: it became nearly impossible to deploy the monolith. With hundreds of commits trying to go out ever

My employer adopted microservices for a very specific reason: it became nearly impossible to deploy the monolith. With hundreds of commits trying to go out every day, probability that at least one would break something approached 1. Then everything had to be rolled back. Getting unrelated concerns into separate deployable artifacts rescued our velocity. It came with many of its own challenges, too! A great deal of infrastructure had to be built to get from O(N) to O(1) infrastructure engineering effort per service. But we did build it, and now it works great. There is a reason monoliths were traditionally coupled with quarterly or even annual releases gated by extensive QA.

simiones 7y ago
My company is currently moving to microservices, but for different reasons. The problem you raise was in fact fixed years ago in our org simply by properly decoupling our "monolithic" app into modules. The top-level build was simply a collection of all pre-built modules. After each dependency update, an automated regression would run, and if pass rate was less than X%, the change was not pushed upstream. Really, microservices give you 2 things better than that: 1, the ability to combine more technologies; 2, much simpler (horizontal) scalability, since properly done microservices naturally scale well with multiple copies running on multiple machines. Of course, the costs are there, in terms of more debugging difficulties, more complex logging needs and usually a higher minimum overhead.
- bcrosby95 7y ago
  I've horizontally scaled plenty of monoliths. Tends to be easier. Less to monitor. You don't have to worry about "what" you're scaling. You just scale it all. Just throw another server on the pile, it reduces resources used by everything on the existing servers. If you have a process that takes up a lot of a particular resource that others don't (e.g. disk throughput), maybe spin that into its own service. But in my experience, lots of things are just fine being lumped together and don't really exhibit resource use profiles that are all that different from eachother.
- mypalmike 7y ago
  Modules help some, but being tied to a deployment cadence with say, hundreds of other developers, can be very painful. Having to delay my team's features because another team broke the build doesn't help my customers. Isolating CI pipelines by team can help that somewhat, but you still end up sharing fate with teams with whom you might have no real interdependencies.
  - tsimionescu 7y ago
    I guess this also depends a lot on the type of product you build. For me, we are building a piece of software that we ship to customers, so the release cadence is limited by many factors outside engineering anyway. But I can see how on a live service-type product, microservices would help a lot more on this front.
  - huherto 7y ago
    I think this is pretty much it. You can have one microservice per team. Then you can do whatever you want in terms of tools, languages, deployment workflows, etc. As long as you honor your API contracts.
warpdrive 7y ago
Hi, We are also moving to microservices for similar reasons. Do you have any more info to share on learnings and best practices?
- closeparen 7y ago
  Have a solid, standardized base/framework layer for your services, and a way to roll out changes to it that doesn’t involve manual work for each service. Migrations are the devil. You will have an extraordinarily difficult time evolving cross—cutting concerns like auth, tracing, metrics, profiling, health checking, RPC encoding, discovery and routing, secrets, etc. without it. And definitely don’t start migrating stuff that matters until you have mature implementations and operations around all those things, and probably many others too.
wobblegong 7y ago
Our company migrated for very similar reasons - I'm surprised that people aren't mentioning this pain point more. Maybe the overall execution of microservices is poor, so people generally would rather go back to the old thing.
- 0x445442 7y ago
  I've found it to be much harder to manage and deploy multiple projects that it was to deploy a war. It's also been a huge PITA to test the newer architecture because all these serverless artifacts need to be deployed together. What we have now is a distributed monolith, just like the article refers to.
- aianus 7y ago
  We deployed to prod 30x a day at Coinbase, precisely for this reason. It worked great and I was honestly shocked when I worked for other companies that plan their releases like they're shipping shrink-wrapped CDs in 1999.
ptero 7y ago
If you have the problem for which microservices are the better solution (as you describe in your example), great! But I have seen people try to build a solution using microservices "because this is the way to go". But this often trades the problems of commits and unit verification for problems of architecture. It can take a lot of work and skill to design a robust architecture using microservices and architectural bugs and failures can be a lot harder to debug, understand and resolve. Pick your poison.
notJim 7y ago
Wouldn't continuous deployment and rolling forward be better than switching your entire application architecture? In this scenario, you deploy ~30 times a day. If there's a bad commit, you revert it, and then do another deploy. So there's no rollbacks, you don't lose as much velocity, and since deploys are safe and a reverted commit is just a new commit, the revert is safe too.
- closeparen 7y ago
  It took at least 4 hours to deploy if everything went well; not sure if that was policy or an infrastructure limitation, but in any case more than 2 deploys per workday would have been impossible.
tootie 7y ago
The solution to this is just writing modular code and using an artifact repository. It's a model I've rarely seen attempted even though it's much easier than microservices and serves the same purpose. You can have individual dev teams, with their own repo ,backlogs, own stakeholders, etc all working at their own paces. They build modules (jars, nuget packages, npm modules) and deploy semver versioned artifacts to a repo like Nexus or JFrog. Any frontend/consumer applications can build towards versions of those modules and upgrade on their own schedule. Only the consumers need to worry about deployment. This gives you the organizational flexibility but not the infrastructure overhead. The discriminating factor that makes microservices necessary if these individual services have divergent hardware needs.
- therealdrag0 7y ago
  My favorite project to work on was a modular monolith. It was a single deployable but each component (vertical) had its own maven-module.
  - tootie 7y ago
    When developers complain about being verbose and not a great language for coders, I counter that it doesn't exist to solve programming problems, but rather organizational problems. The killer feature that launched Java wasn't crap like checked exceptions, it was javadoc. Strict, self-documenting APIs are 10X more valuable than any intrinsic language feature.
    - timwaagh 7y ago
      i dont care about it not being great. its good enough as a language. but maven... but websphere... java is a hellish platform that ordinarily would not win from interpreted languages or those which focus on fast compilation. But it runs literally everywhere, including your toaster, but more importantly on mainframes which also run the real mvp aka Cobol. run once, run anywhere remains a killer feature no other platform has replicated as well.
      - 7y ago
stock_toaster 7y ago
> With hundreds of commits trying to go out every day, probability that at least one would break something approached 1. Then everything had to be rolled back. Getting unrelated concerns into separate deployable artifacts rescued our velocity. Software tends to reflect the structure of the organization that creates it, so this makes sense to me. If you have multiple teams contributing to a stack, eventually it is easier to have the teams work on their own (micro-)service(s). I recommend to most people that stacks should start out as monoliths though, and move to microservice architectures only when they encounter enough pain. I think starting out with microservices from the get-go just reduces initial velocity with little payoff until you hit a certain scale.
- andrewprock 7y ago
  I think you are missing the intermediate step of libraries. 1. monolith 2. libraries 3. services If you skip step 2, there is a high probability that the services you end up with are going to be just as disorganized as the monolith which is causing grief.
  - stock_toaster 7y ago
    yup, agreed. I generally would consider your step2 as part of the lifecycle of a well architected monolith, but calling it out as a separate step is certainly clearer, and does reflect more of what I have seen "work well" in the real world too.
chillacy 7y ago
Feature toggles are a solution this. That lets you do feature releases independent of code deployment. Doesn’t work 100% of the time but I can’t imagine doing a large monolith without feature toggles.
tomphoolery 7y ago
This is the right reason why an organization would want to adopt microservices, in my opinion. If you have hundreds of commits going out every day, you're probably dealing with a relatively large team, yes? Microservices makes sense for large companies, because it allows for organization of developers into teams that manage each microservice rather than having them all under the umbrella of the same application. But if your team was smaller, say, 6 people...managing 20 different services becomes a real chore and adds a lot of overhead. In this case, a monolithic application is a way better choice, because it allows all the developers to work together on the same codebase instead of spread out and distant amongst a bunch of different applications with varying degrees of quality.
huherto 7y ago
But how big are your microservices ? How many did you build ? A monolith that has hundreds of commits is very worth separating it. But do you break it on 10, 100, 1000 microservices ? We have a case where we have 20 people managing 40 microservices. Way too granular. In the general literature there is not good consensus finding boundaries and rightsizing. Many authors just duck the question.

simiones 7y ago

My company is currently moving to microservices, but for different reasons. The problem you raise was in fact fixed years ago in our org simply by properly decoupling our "monolithic" app into modules. The top-level build was simply a collection of all pre-built modules. After each dependency update, an automated regression would run, and if pass rate was less than X%, the change was not pushed upstream. Really, microservices give you 2 things better than that: 1, the ability to combine more technologies; 2, much simpler (horizontal) scalability, since properly done microservices naturally scale well with multiple copies running on multiple machines. Of course, the costs are there, in terms of more debugging difficulties, more complex logging needs and usually a higher minimum overhead.

bcrosby95 7y ago
I've horizontally scaled plenty of monoliths. Tends to be easier. Less to monitor. You don't have to worry about "what" you're scaling. You just scale it all. Just throw another server on the pile, it reduces resources used by everything on the existing servers. If you have a process that takes up a lot of a particular resource that others don't (e.g. disk throughput), maybe spin that into its own service. But in my experience, lots of things are just fine being lumped together and don't really exhibit resource use profiles that are all that different from eachother.
mypalmike 7y ago
Modules help some, but being tied to a deployment cadence with say, hundreds of other developers, can be very painful. Having to delay my team's features because another team broke the build doesn't help my customers. Isolating CI pipelines by team can help that somewhat, but you still end up sharing fate with teams with whom you might have no real interdependencies.
- tsimionescu 7y ago
  I guess this also depends a lot on the type of product you build. For me, we are building a piece of software that we ship to customers, so the release cadence is limited by many factors outside engineering anyway. But I can see how on a live service-type product, microservices would help a lot more on this front.
- huherto 7y ago
  I think this is pretty much it. You can have one microservice per team. Then you can do whatever you want in terms of tools, languages, deployment workflows, etc. As long as you honor your API contracts.

warpdrive 7y ago

Hi, We are also moving to microservices for similar reasons. Do you have any more info to share on learnings and best practices?

closeparen 7y ago
Have a solid, standardized base/framework layer for your services, and a way to roll out changes to it that doesn’t involve manual work for each service. Migrations are the devil. You will have an extraordinarily difficult time evolving cross—cutting concerns like auth, tracing, metrics, profiling, health checking, RPC encoding, discovery and routing, secrets, etc. without it. And definitely don’t start migrating stuff that matters until you have mature implementations and operations around all those things, and probably many others too.

wobblegong 7y ago

Our company migrated for very similar reasons - I'm surprised that people aren't mentioning this pain point more. Maybe the overall execution of microservices is poor, so people generally would rather go back to the old thing.

0x445442 7y ago
I've found it to be much harder to manage and deploy multiple projects that it was to deploy a war. It's also been a huge PITA to test the newer architecture because all these serverless artifacts need to be deployed together. What we have now is a distributed monolith, just like the article refers to.
aianus 7y ago
We deployed to prod 30x a day at Coinbase, precisely for this reason. It worked great and I was honestly shocked when I worked for other companies that plan their releases like they're shipping shrink-wrapped CDs in 1999.

ptero 7y ago

If you have the problem for which microservices are the better solution (as you describe in your example), great! But I have seen people try to build a solution using microservices "because this is the way to go". But this often trades the problems of commits and unit verification for problems of architecture. It can take a lot of work and skill to design a robust architecture using microservices and architectural bugs and failures can be a lot harder to debug, understand and resolve. Pick your poison.

notJim 7y ago

Wouldn't continuous deployment and rolling forward be better than switching your entire application architecture? In this scenario, you deploy ~30 times a day. If there's a bad commit, you revert it, and then do another deploy. So there's no rollbacks, you don't lose as much velocity, and since deploys are safe and a reverted commit is just a new commit, the revert is safe too.

closeparen 7y ago
It took at least 4 hours to deploy if everything went well; not sure if that was policy or an infrastructure limitation, but in any case more than 2 deploys per workday would have been impossible.

tootie 7y ago

The solution to this is just writing modular code and using an artifact repository. It's a model I've rarely seen attempted even though it's much easier than microservices and serves the same purpose. You can have individual dev teams, with their own repo ,backlogs, own stakeholders, etc all working at their own paces. They build modules (jars, nuget packages, npm modules) and deploy semver versioned artifacts to a repo like Nexus or JFrog. Any frontend/consumer applications can build towards versions of those modules and upgrade on their own schedule. Only the consumers need to worry about deployment. This gives you the organizational flexibility but not the infrastructure overhead. The discriminating factor that makes microservices necessary if these individual services have divergent hardware needs.

therealdrag0 7y ago
My favorite project to work on was a modular monolith. It was a single deployable but each component (vertical) had its own maven-module.
- tootie 7y ago
  When developers complain about being verbose and not a great language for coders, I counter that it doesn't exist to solve programming problems, but rather organizational problems. The killer feature that launched Java wasn't crap like checked exceptions, it was javadoc. Strict, self-documenting APIs are 10X more valuable than any intrinsic language feature.
  - timwaagh 7y ago
    i dont care about it not being great. its good enough as a language. but maven... but websphere... java is a hellish platform that ordinarily would not win from interpreted languages or those which focus on fast compilation. But it runs literally everywhere, including your toaster, but more importantly on mainframes which also run the real mvp aka Cobol. run once, run anywhere remains a killer feature no other platform has replicated as well.
    - 7y ago

stock_toaster 7y ago

> With hundreds of commits trying to go out every day, probability that at least one would break something approached 1. Then everything had to be rolled back. Getting unrelated concerns into separate deployable artifacts rescued our velocity. Software tends to reflect the structure of the organization that creates it, so this makes sense to me. If you have multiple teams contributing to a stack, eventually it is easier to have the teams work on their own (micro-)service(s). I recommend to most people that stacks should start out as monoliths though, and move to microservice architectures only when they encounter enough pain. I think starting out with microservices from the get-go just reduces initial velocity with little payoff until you hit a certain scale.

andrewprock 7y ago
I think you are missing the intermediate step of libraries. 1. monolith 2. libraries 3. services If you skip step 2, there is a high probability that the services you end up with are going to be just as disorganized as the monolith which is causing grief.
- stock_toaster 7y ago
  yup, agreed. I generally would consider your step2 as part of the lifecycle of a well architected monolith, but calling it out as a separate step is certainly clearer, and does reflect more of what I have seen "work well" in the real world too.

chillacy 7y ago

Feature toggles are a solution this. That lets you do feature releases independent of code deployment. Doesn’t work 100% of the time but I can’t imagine doing a large monolith without feature toggles.

tomphoolery 7y ago

This is the right reason why an organization would want to adopt microservices, in my opinion. If you have hundreds of commits going out every day, you're probably dealing with a relatively large team, yes? Microservices makes sense for large companies, because it allows for organization of developers into teams that manage each microservice rather than having them all under the umbrella of the same application. But if your team was smaller, say, 6 people...managing 20 different services becomes a real chore and adds a lot of overhead. In this case, a monolithic application is a way better choice, because it allows all the developers to work together on the same codebase instead of spread out and distant amongst a bunch of different applications with varying degrees of quality.

huherto 7y ago

But how big are your microservices ? How many did you build ? A monolith that has hundreds of commits is very worth separating it. But do you break it on 10, 100, 1000 microservices ? We have a case where we have 20 people managing 40 microservices. Way too granular. In the general literature there is not good consensus finding boundaries and rightsizing. Many authors just duck the question.