6 ms·

I think that many people look at Microservices from a purely technical standpoint. How much compute power do I get per dollar spent. What about the monitoring c

I think that many people look at Microservices from a purely technical standpoint. How much compute power do I get per dollar spent. What about the monitoring complexities and the logging problems? Microservices are actually often driven by business requirements. According to the Myhtical Man Month communication channels follow n(n − 1) / 2. So a team of 4 developers has 6 possible communication channels, a team of 5 already 10. This eincreases sharply as team sizes increase and is one of the reasons big enterprise projects suffer compare to startups. The business benefit of Microservices (done right) is that you build islands of communication, that communicate using only one outside channel. Now communication increases less sharply as teams are separated by their boundaries. According to the guardian the team size and structure is one of the secret for Amazons success. https://www.theguardian.com/technology/2018/apr/24/the-two-pizza-rule-and-the-secret-of-amazons-success https://www.theguardian.com/technology/2018/apr/24/the-two-p... For smaller teams the communication benefit will often not outweigh the additional complexity.

oftenwrong 6y ago
It is possible to keep teams "on islands" with a modular monolith. Each team builds a module. Logging, monitoring, and other shared infrastructure is there for all teams.
- soonnow 6y ago
  Until you come to the deployment phase where all teams need to communicate and deploy in sync. This can be a significant factor for slow down. Also Microservices don't sync on the datastore, these are (in theory) not shared. So yes Microservices can be emulated in Monolithic architectures, as can Monolithic Microservices be built. Usually you end up with the worst of both worlds. The complexity of Microservices with the speed of Monoliths.
hinkley 6y ago
But what happens is you end up with a ton of them, fulfilling the same prophecy you were trying to avoid, or you keep trying to cram more “volume” behind the same surface area, until you eventually tear a hole in the fabric of space time and/or your business. The way to succeed is to do fewer things but make them count. Except that looks awful on your annual review, so the real way to succeed is to Defect. Be seen doing as much as you can and get out before they figure out you consumed all of the oxygen in the room.
- sriku 6y ago
  I think John Ousterhout's recommendation of 'thin' interfaces and 'thick' implementations is a good heuristic that's worked well in my experience. This acts as a force that resists service proliferation that hides behind "modularity". https://web.stanford.edu/~ouster/cgi-bin/cs190-spring16/lecture.php?topic=modularDesign https://web.stanford.edu/~ouster/cgi-bin/cs190-spring16/lect...
  - hinkley 6y ago
    What I'm saying is that this works great for a while. But if you want to keep it working, four years in, you still have to say No a lot.
asim 6y ago
Good points and it feels like this is what most people miss. They take microservices from a purely technical perspective and then rant about its complexity. You don't need a large scale distributed systems platform when you're a small team or have no scaling requirements. Microservices is largely driven through that organisational need of teams having to ship products/features independently and its just easier when the communication is via a service API vs humans or explicitly when walking over each others code.
- ak39 6y ago
  If communication friction and overhead is the primary thrust for microservices (where you obviate said overhead by keeping to small teams), how do you address and manage the need for communication between teams? Obviating communication costs within teams is one thing, but what of the overhead and costs of communication for the whole system? Eg. how do you avoid the cost of deploying a new team that ends up deploying a duplicate microservice?
  - asim 6y ago
    These things are rarely binary. In that there's no one size fits all solution. It comes down to the culture of the company and how they communicate and then how the technology evolved based on their needs. So if you have an engineering org which practices high bandwidth communication there might be a drive to keep that up by coordinating a meeting once a week by nominating one person from the team to go provide updates and learn about what else is happening. You may also explicitly establish an architecture team that has oversight and tries to ensure everyone's aligned on direction. But mostly if you at least have some shared place where you explicitly communicate the creation of new services or design development then it allows you to avoid some of the issues you're talking about. The other thing is having a way to discover APIs and services so you can go check if something exists before building it. In many cases teams do still build a variant themselves because they need something slightly different and thats ok if they're willing to support it.
  - jmchuster 6y ago
    I would say that, if you're at a point where teams are deploying duplicate services with no idea of each other, you're already so large with so many teams, that you might desire that as the correct outcome. You now have hundreds of teams running in parallel, and there's no point in coordinating with everyone, just coordinate with the ten that you are adjacent to. So, yes, the company is not maximally efficient, but it can take advantage of parallelism, and that's ultimately much more important for speed of progress. I could see an argument that it'd take months to get all the teams to all meet and agree and coordinate on what this shared service should be, when you could have built two variants simultaneously in a couple of weeks.
ak39 6y ago
Good explanation. But I have two questions: 1) So which "team" then takes ownership of the entire katamari ball of sprawling microservices and understands the interconnectnedness and nuances of overlaps and competition between each microservice? 2) If the concept of technical debt at yesterday's monolithic architectures is biting us now, what does the future technical debt of microservices hold?
- asim 6y ago
  1. No one has ownership over all the services, that's shared ownership. A team might own the platform and be responsible for up time, but individual teams own their services. Somewhere higher in the engineering org there's explicit ownership of end to end product and its usually tied with that product team or again the platform team. 2. As you scale, sometimes its easier to create a new service than it is to go modify that old service. When a company scales fast or employees churn its either rewrite that service or leave it where it is and build something new. This can lead to something that looks a lot more organic than computational. I have only seen the likes of Google handle deprecation well, to the dislike of most people external but survival of the fittest is harsh at best.
  - deleted 6y ago
    [deleted]
  - tluyben2 6y ago
    in both 1 & 2, I only see money pits... This sounds like so much overhead... Par for the course for enterprises, but I see too many startups now deploying 50+ microservices and suffering with the overhead (hosting, maintenance, employee-count/churn/training, monitoring etc).
    - krab 6y ago
      > in both 1 & 2, I only see money pits... This sounds like so much overhead... That's the point. The overhead is high but the other options to employ extra developers bring even more overhead. And the company wants extra developers because the faster (or more parallel) development still has a good ROI. If you want low overhead, a small team is the way to go.
jensneuse 6y ago
You're making a point that Microservices help teams to solve the problem of increasing communication needs for larger teams. While your statement makes a lot of sense I'd like to ask why this problem cannot be solved with packages or classes within a monolith? A team can own a Microservice and expose an api to others the same way they could expose an api on a class or package, no? Microservices are a way of deploying small parts of software. It's not about communication across teams. It can help with communication but this is not only true for Microservices.
- politelemon 6y ago
  The reason that I've seen (therefore anecdotal) is even if you attempt to cater for handoffs with packages/libraries/classes/APIs, eventually a point comes where some of these don't exist in a vacuum. Or rather, they exist in a vacuum until they don't. For example a change in one library might cause an unexpected degradation in an unrelated application. The communication at that point is strained as there are now multiple teams firefighting and playing the blame game. Or a slight change in one library caused deployments to fail for everyone. In a perfect world with perfect engineering practices this wouldn't happen but as many have pointed out, development is a reflection of the business, and the business is not perfect. Having mini-services, so even just taking a monolith and separating it out into manageable chunks, reduces the communication problem immensely. Doesn't do away with it completely, but goes a long long way!
  - pjmlp 6y ago
    As if the blame game doesn't take place with microservices as well. It is specially "funny" if the microservices are developed by several contractors.
    - oftenwrong 6y ago
      Yes. For example, a service starts sending bad data due to a bug. Now other services have acted on it, stored it, etc. It becomes a problem for many people.
    - 6y ago
pjmlp 6y ago
My first microservices were written using Sun RPC, the company whose motto was "The network is the computer". Since then we have seen plenty of microservices solutions come and go, what stays is the complexity of dealing with distributed programming. Anyone that isn't able to write modular code using the language features is going to write network spaghetti code, with the added benefit to debug network packets and connection issues on top of everything else.
- extra_rice 6y ago
  > Anyone that isn't able to write modular code using the language features is going to write network spaghetti code, with the added benefit to debug network packets and connection issues on top of everything else. This is the most important bit. To me microservices is just another way to arrange your system into module, not much different in concept to modules and packages. Regardless of whether you’re dealing with monolithic or distributed system, if your abstractions are poor, your work and your system will suffer.
  - tluyben2 6y ago
    > Regardless of whether you’re dealing with monolithic or distributed system Agreeing somewhat, but not totally ; monoliths are much easier to test and stabilise. All bigger microservices projects I know (personally, not what I read from Uber etc) have tremendous overheads compared to their monolithic counterparts. Some of them are really nicely done, but there still is a lot of overhead in support (usually the services are written by different people (and possible different languages/environments, for instance .net core, .net framework (for Windows specific services) + Typescript/node), so other people need quite a large ramp up to get into them), networking, monitoring, etc. As far as I have seen, it needs a larger team. And it is very different from having you modular built monolith in your IDE for a spin, than doing a little (local/dev) testrun of 25 microservices + the one you are working on. I can feel the benefits but I have yet to encounter them in real life; so far I have seen (smaller; ~100m/year rev) companies going back to monolith and most of the stories I read are from very large companies with massive development+deployment teams where you actually can have 3+ people per microservice. If you are working with 3-4 people managing 20+ (changing, forming a complex system), I don't really seen it happen (but would love to see practical examples of that) while monoliths of the same complexity/functionality (literally; I worked on a few 'just because scaling!' rebuilds from monoliths->MS(->back)) have no issues at all with that.
mytailorisrich 6y ago
Organisation and communication between teams is improved by having good APIs. You can have that with or without microservices. You can screw this up with or without microservices. The main factors for going for microservices are technical but, as always, must be understood.
sandGorgon 6y ago
you can do islands of communications in a monolith as well, with the added advantage of significantly lower deployment complexity as well lower incidental complexity
- tluyben2 6y ago
  And (much) lower deployment costs, lower staffing costs, and lower development costs... In my experience anyway.
- pragmaticdev 6y ago
  This is only true for trivially small products. The pipeline complexity for multi-team monoliths can consume an entire dev team to maintain.
- collyw 6y ago
  I view a working microservices application as a monolith with unreliable network connections between components. So overall adding a lot more complexity for the promise of lowers costs and scale-ability. One advantage that I do see is the ability to deploy components individually, but I see more downsides than benefits overall. I hope that consensus is becoming "start with a (modular, well designed) monolith and scale out the parts as needed".
ex_amazon_sde 6y ago
Ex Amazon SDE here. Amazon uses a service based architecture where one team (e.g. 4 to 8 engineers) maintains one service. The main benefit is organizational and lies in having a small and very measurable interface between services. Crucially, the team the develops the service runs it and provides 24/7 on-call for it. This also include taking decisions on how much to spend on hardware VS optimization as the team is responsible for the overall performance. More importantly, the team is now responsible for service outages. This is very different from having a team write many microservices. It's especially bad when a team writes a bunch of microservices (the so-called distributed monolith) and somebody else has to run them. Sadly the article does not focus on this aspect. As a side note, once developers realize they get paged out of bed by their own code... it's amusing to see how the cool framework of the month is not cool anymore.
- pragmaticdev 6y ago
  It's difficult to put all of the nuances in a blog post. As a developer, I'm a firm believer in "run what you brung".
  - ex_amazon_sde 6y ago
    The blog post is very long and predicted at 18 minutes of reading, yet the word "deploy" appears only once. Plus there is no mention on who has to run it.
- WJW 6y ago
  At my previous company this was introduced as well, but the incentives were very warped. There was no advantage to actually being all that stable, since then you missed out on "firefighting glory". I never saw praise or promotions for maintaining high uptime. I'd be very interested to learn if Amazon managed to overcome this and if so, how.
  - ex_amazon_sde 6y ago
    I cannot talk for the whole company but I don't think I witnessed a culture of "firefighting glory", rather the opposite. There is a formal process to investigate and correct errors after a non-trivial incident that requires collecting evidence and then discuss what happen, when, why and who did what with the whole team. Then people ask the question "why was this not prevented or forseen?" and keep going backward until you exit the technical realm and look into people's choices. The root causes could be, for example, that it was a management decision to prioritize something else over availability concerns... and the engineers are off the hook. Sometimes one engineer was overworked and tired and did an error in good faith that is too difficult to prevent and avoid, and that's also acceptable. Sometimes that the whole team ignored an availability issue in the product architecture, and that's bad. And so on... Needless to say the outcome can impact people's performance review.
soonnow 6y ago
Here is an article where I look at this issue in more detail https://thenewstack.io/microservices-vs-monoliths-an-operational-comparison/ https://thenewstack.io/microservices-vs-monoliths-an-operati...

oftenwrong 6y ago

It is possible to keep teams "on islands" with a modular monolith. Each team builds a module. Logging, monitoring, and other shared infrastructure is there for all teams.

soonnow 6y ago
Until you come to the deployment phase where all teams need to communicate and deploy in sync. This can be a significant factor for slow down. Also Microservices don't sync on the datastore, these are (in theory) not shared. So yes Microservices can be emulated in Monolithic architectures, as can Monolithic Microservices be built. Usually you end up with the worst of both worlds. The complexity of Microservices with the speed of Monoliths.

hinkley 6y ago

But what happens is you end up with a ton of them, fulfilling the same prophecy you were trying to avoid, or you keep trying to cram more “volume” behind the same surface area, until you eventually tear a hole in the fabric of space time and/or your business. The way to succeed is to do fewer things but make them count. Except that looks awful on your annual review, so the real way to succeed is to Defect. Be seen doing as much as you can and get out before they figure out you consumed all of the oxygen in the room.

sriku 6y ago
I think John Ousterhout's recommendation of 'thin' interfaces and 'thick' implementations is a good heuristic that's worked well in my experience. This acts as a force that resists service proliferation that hides behind "modularity". https://web.stanford.edu/~ouster/cgi-bin/cs190-spring16/lecture.php?topic=modularDesign https://web.stanford.edu/~ouster/cgi-bin/cs190-spring16/lect...
- hinkley 6y ago
  What I'm saying is that this works great for a while. But if you want to keep it working, four years in, you still have to say No a lot.

asim 6y ago

Good points and it feels like this is what most people miss. They take microservices from a purely technical perspective and then rant about its complexity. You don't need a large scale distributed systems platform when you're a small team or have no scaling requirements. Microservices is largely driven through that organisational need of teams having to ship products/features independently and its just easier when the communication is via a service API vs humans or explicitly when walking over each others code.

ak39 6y ago
If communication friction and overhead is the primary thrust for microservices (where you obviate said overhead by keeping to small teams), how do you address and manage the need for communication between teams? Obviating communication costs within teams is one thing, but what of the overhead and costs of communication for the whole system? Eg. how do you avoid the cost of deploying a new team that ends up deploying a duplicate microservice?
- asim 6y ago
  These things are rarely binary. In that there's no one size fits all solution. It comes down to the culture of the company and how they communicate and then how the technology evolved based on their needs. So if you have an engineering org which practices high bandwidth communication there might be a drive to keep that up by coordinating a meeting once a week by nominating one person from the team to go provide updates and learn about what else is happening. You may also explicitly establish an architecture team that has oversight and tries to ensure everyone's aligned on direction. But mostly if you at least have some shared place where you explicitly communicate the creation of new services or design development then it allows you to avoid some of the issues you're talking about. The other thing is having a way to discover APIs and services so you can go check if something exists before building it. In many cases teams do still build a variant themselves because they need something slightly different and thats ok if they're willing to support it.
- jmchuster 6y ago
  I would say that, if you're at a point where teams are deploying duplicate services with no idea of each other, you're already so large with so many teams, that you might desire that as the correct outcome. You now have hundreds of teams running in parallel, and there's no point in coordinating with everyone, just coordinate with the ten that you are adjacent to. So, yes, the company is not maximally efficient, but it can take advantage of parallelism, and that's ultimately much more important for speed of progress. I could see an argument that it'd take months to get all the teams to all meet and agree and coordinate on what this shared service should be, when you could have built two variants simultaneously in a couple of weeks.

ak39 6y ago

Good explanation. But I have two questions: 1) So which "team" then takes ownership of the entire katamari ball of sprawling microservices and understands the interconnectnedness and nuances of overlaps and competition between each microservice? 2) If the concept of technical debt at yesterday's monolithic architectures is biting us now, what does the future technical debt of microservices hold?

asim 6y ago
1. No one has ownership over all the services, that's shared ownership. A team might own the platform and be responsible for up time, but individual teams own their services. Somewhere higher in the engineering org there's explicit ownership of end to end product and its usually tied with that product team or again the platform team. 2. As you scale, sometimes its easier to create a new service than it is to go modify that old service. When a company scales fast or employees churn its either rewrite that service or leave it where it is and build something new. This can lead to something that looks a lot more organic than computational. I have only seen the likes of Google handle deprecation well, to the dislike of most people external but survival of the fittest is harsh at best.
- deleted 6y ago
  [deleted]
- tluyben2 6y ago
  in both 1 & 2, I only see money pits... This sounds like so much overhead... Par for the course for enterprises, but I see too many startups now deploying 50+ microservices and suffering with the overhead (hosting, maintenance, employee-count/churn/training, monitoring etc).
  - krab 6y ago
    > in both 1 & 2, I only see money pits... This sounds like so much overhead... That's the point. The overhead is high but the other options to employ extra developers bring even more overhead. And the company wants extra developers because the faster (or more parallel) development still has a good ROI. If you want low overhead, a small team is the way to go.

jensneuse 6y ago

You're making a point that Microservices help teams to solve the problem of increasing communication needs for larger teams. While your statement makes a lot of sense I'd like to ask why this problem cannot be solved with packages or classes within a monolith? A team can own a Microservice and expose an api to others the same way they could expose an api on a class or package, no? Microservices are a way of deploying small parts of software. It's not about communication across teams. It can help with communication but this is not only true for Microservices.

politelemon 6y ago
The reason that I've seen (therefore anecdotal) is even if you attempt to cater for handoffs with packages/libraries/classes/APIs, eventually a point comes where some of these don't exist in a vacuum. Or rather, they exist in a vacuum until they don't. For example a change in one library might cause an unexpected degradation in an unrelated application. The communication at that point is strained as there are now multiple teams firefighting and playing the blame game. Or a slight change in one library caused deployments to fail for everyone. In a perfect world with perfect engineering practices this wouldn't happen but as many have pointed out, development is a reflection of the business, and the business is not perfect. Having mini-services, so even just taking a monolith and separating it out into manageable chunks, reduces the communication problem immensely. Doesn't do away with it completely, but goes a long long way!
- pjmlp 6y ago
  As if the blame game doesn't take place with microservices as well. It is specially "funny" if the microservices are developed by several contractors.
  - oftenwrong 6y ago
    Yes. For example, a service starts sending bad data due to a bug. Now other services have acted on it, stored it, etc. It becomes a problem for many people.
  - 6y ago

pjmlp 6y ago

My first microservices were written using Sun RPC, the company whose motto was "The network is the computer". Since then we have seen plenty of microservices solutions come and go, what stays is the complexity of dealing with distributed programming. Anyone that isn't able to write modular code using the language features is going to write network spaghetti code, with the added benefit to debug network packets and connection issues on top of everything else.

extra_rice 6y ago
> Anyone that isn't able to write modular code using the language features is going to write network spaghetti code, with the added benefit to debug network packets and connection issues on top of everything else. This is the most important bit. To me microservices is just another way to arrange your system into module, not much different in concept to modules and packages. Regardless of whether you’re dealing with monolithic or distributed system, if your abstractions are poor, your work and your system will suffer.
- tluyben2 6y ago
  > Regardless of whether you’re dealing with monolithic or distributed system Agreeing somewhat, but not totally ; monoliths are much easier to test and stabilise. All bigger microservices projects I know (personally, not what I read from Uber etc) have tremendous overheads compared to their monolithic counterparts. Some of them are really nicely done, but there still is a lot of overhead in support (usually the services are written by different people (and possible different languages/environments, for instance .net core, .net framework (for Windows specific services) + Typescript/node), so other people need quite a large ramp up to get into them), networking, monitoring, etc. As far as I have seen, it needs a larger team. And it is very different from having you modular built monolith in your IDE for a spin, than doing a little (local/dev) testrun of 25 microservices + the one you are working on. I can feel the benefits but I have yet to encounter them in real life; so far I have seen (smaller; ~100m/year rev) companies going back to monolith and most of the stories I read are from very large companies with massive development+deployment teams where you actually can have 3+ people per microservice. If you are working with 3-4 people managing 20+ (changing, forming a complex system), I don't really seen it happen (but would love to see practical examples of that) while monoliths of the same complexity/functionality (literally; I worked on a few 'just because scaling!' rebuilds from monoliths->MS(->back)) have no issues at all with that.

mytailorisrich 6y ago

Organisation and communication between teams is improved by having good APIs. You can have that with or without microservices. You can screw this up with or without microservices. The main factors for going for microservices are technical but, as always, must be understood.

sandGorgon 6y ago

you can do islands of communications in a monolith as well, with the added advantage of significantly lower deployment complexity as well lower incidental complexity

tluyben2 6y ago
And (much) lower deployment costs, lower staffing costs, and lower development costs... In my experience anyway.
pragmaticdev 6y ago
This is only true for trivially small products. The pipeline complexity for multi-team monoliths can consume an entire dev team to maintain.
collyw 6y ago
I view a working microservices application as a monolith with unreliable network connections between components. So overall adding a lot more complexity for the promise of lowers costs and scale-ability. One advantage that I do see is the ability to deploy components individually, but I see more downsides than benefits overall. I hope that consensus is becoming "start with a (modular, well designed) monolith and scale out the parts as needed".

ex_amazon_sde 6y ago

Ex Amazon SDE here. Amazon uses a service based architecture where one team (e.g. 4 to 8 engineers) maintains one service. The main benefit is organizational and lies in having a small and very measurable interface between services. Crucially, the team the develops the service runs it and provides 24/7 on-call for it. This also include taking decisions on how much to spend on hardware VS optimization as the team is responsible for the overall performance. More importantly, the team is now responsible for service outages. This is very different from having a team write many microservices. It's especially bad when a team writes a bunch of microservices (the so-called distributed monolith) and somebody else has to run them. Sadly the article does not focus on this aspect. As a side note, once developers realize they get paged out of bed by their own code... it's amusing to see how the cool framework of the month is not cool anymore.

pragmaticdev 6y ago
It's difficult to put all of the nuances in a blog post. As a developer, I'm a firm believer in "run what you brung".
- ex_amazon_sde 6y ago
  The blog post is very long and predicted at 18 minutes of reading, yet the word "deploy" appears only once. Plus there is no mention on who has to run it.
WJW 6y ago
At my previous company this was introduced as well, but the incentives were very warped. There was no advantage to actually being all that stable, since then you missed out on "firefighting glory". I never saw praise or promotions for maintaining high uptime. I'd be very interested to learn if Amazon managed to overcome this and if so, how.
- ex_amazon_sde 6y ago
  I cannot talk for the whole company but I don't think I witnessed a culture of "firefighting glory", rather the opposite. There is a formal process to investigate and correct errors after a non-trivial incident that requires collecting evidence and then discuss what happen, when, why and who did what with the whole team. Then people ask the question "why was this not prevented or forseen?" and keep going backward until you exit the technical realm and look into people's choices. The root causes could be, for example, that it was a management decision to prioritize something else over availability concerns... and the engineers are off the hook. Sometimes one engineer was overworked and tired and did an error in good faith that is too difficult to prevent and avoid, and that's also acceptable. Sometimes that the whole team ignored an availability issue in the product architecture, and that's bad. And so on... Needless to say the outcome can impact people's performance review.

soonnow 6y ago

Here is an article where I look at this issue in more detail https://thenewstack.io/microservices-vs-monoliths-an-operational-comparison/ https://thenewstack.io/microservices-vs-monoliths-an-operati...