13 ms·

Microservices are a solution to a social problem, not a technical one. A team of N engineers requires N² coordination. Large teams get mired in endless meeting

by synack 3y ago

Microservices are a solution to a social problem, not a technical one. A team of N engineers requires N² coordination. Large teams get mired in endless meetings, email, design reviews. Small teams are more effective, but struggle to maintain large systems. Splitting a system into subsystems allows each team to focus on their piece of the puzzle while minimizing the amount of peer-to-peer coordination. Yes, microservices add complexity and overhead, but this approach enables a large organization to build and iterate on large systems quickly.

meowtimemania 3y ago
if you have a well modularized monolith, you can get best of both worlds.
- synack 3y ago
  Agreed! Software modules and libraries can achieve the same independence of subsystems without implying a network topology.
- jayd16 3y ago
  Not really. You can't really reduce the blast radius of crashes or bad deployments. You need to have the discipline of a good CI/CD instead of siloed but decoupled workflows. Just keeping things neat doesn't go nearly as far as a separate process on separate machines. Monolith might be better but I don't think it's a situation where you can have it all.
  - aidos 3y ago
    This argument always confuses me. It depends on what you’re doing but if you’re doing web services, as most here are, the crash is limited to the request being served. The blast radius is a single request, right? In every likelihood it _is_ a separate process on a separate machine.
    - threeseed 3y ago
      They are talking about deployment. With micro-services if you screw up a deployment the service is down and if your platform is well designed the system will be degraded but not down. With a monolith the whole system is down.
      - quickthrower2 3y ago
        There are a lot of deployments options to mitigate that risk.
      - charcircuit 3y ago
        >With a monolith the whole system is down. No, you are just at reduced capacity. The moral of the story is not to roll out to 100% right away.
        threeseed 3y ago
        Monoliths go hand in hand with a centralised database. Bit hard to do schema evolution with a staggered rollout.
        charcircuit 3y ago
        The software should be compatible with both the old and new schema until all of your database servers have moved over to the new schema. All rollouts are going to be somewhat staggered even if you go straight to 100%.
        DougBTX 3y ago
        If schema migrations take non-zero time and you want zero-downtime deployments, then the both service versions need to be compatible with either the new schema or the old schema, regardless of service size.
        ownagefool 3y ago
        The same problem exists in microservoce land, only you now have a load of separate teams managing their own database, perhaps with different solutions.
  - DougBTX 3y ago
    > You need to have the discipline of a good CI/CD Good CD is even more important if there are more services to deploy
- croo 3y ago
  That is a big if that can be easily broken by a new guy who don't know the rules or if the pace is fast enough that you cannot review everything. If the codebase is different you can force the separation not just ask nicely to keep the code well modularized.
  - meowtimemania 3y ago
    You can enforce modularity with git acls, and multi-module workspaces within a single git repository. for example: golang - multi-module workspaces javascript - yarn workspaces With this setup, you can create modules a, b and c. And then add restrictions about which which modules are available to a given module
- mmcnl 3y ago
  Yes, only it takes a while to figure out the right level of abstraction for your organization. It's difficult to start right out of the box with the right type of modularization.
corethree 3y ago
Why do you have to split things into services? How about moving things into different folders. Have you thought about that? Why do you have to modularize it with a whole new repo, a whole new docker set up? Just use a folder bro.
- abhiyerra 3y ago
  I found Django apps to be a good middle ground.
- hot_gril 3y ago
  Usually the most important designation of the separate system is that it uses a separate database. You only have immediate consistency within one service.
  - aledalgrande 3y ago
    lol good luck without centralized data
    - hot_gril 3y ago
      Each service is authoritative on its own data, and it works neatly. Any medium to large size system is going to have natural places to separate that, or else will buckle under complexity or even hardware constraints if you try to keep it all in one DB. I've seen it repeatedly.
      - corethree 3y ago
        Nah this doesn't work out. Because requirements are so chaotic and modular design isn't an axiomatic science things will for sure go wrong with the design both because you got the design wrong and because the requirements shifted to a point where another design was better. You encapsulated data under one service and you find that it's actually utilized much more in another service. This happens a lot. The longer you can centralize your storage the better and easier everything is.
brhsagain 3y ago
> Splitting a system into subsystems allows each team to focus on their piece of the puzzle while minimizing the amount of peer-to-peer coordination. This does not happen at all. When you break a system into subsystems, all the previous connections that get remapped to new connections between subsystems still need to happen, in order to solve the fundamental problem that the system solves — except now instead of just making the connection directly, there has to be a "cross-functional" meeting between teams and a complicated communication layer between the systems. And if somehow you find a breakdown that requires minimal connections between subsystems, then those connections wouldn't have existed in the original system either, and the N² problem doesn't exist.
- jedberg 3y ago
  If that's the experience then you're doing services wrong. Each service should have its own datastore and a single API. The interface between services should be a single connection. There should be maybe one meeting where the caller defines what they need the service to return to them.
  - 10000truths 3y ago
    The problem with this is that you have to be really damned careful how you split things up. If your separate data stores end up having to be joined together later on because some new business feature requires them to be cross-referenced, you're painted into one of two corners: 1. Merge the two services (and their data stores) into one and cause havoc downstream of either service 2. Burn through your network latency/throughput budget trying to reinvent a DB join across RPC boundaries (and god forbid if you can't batch multiple lookups in a single API call!)
    - dahwolf 3y ago
      Exactly. Software developers will never learn that the separation of concerns is a myth. In reality, UI, business logic and data are deeply entangled. Hence, moving these things apart makes everything worse.
  - 3y ago
deleted 3y ago
[deleted]
politelemon 3y ago
Shouldn't it be n(n-1)/2 coordination. Assuming you mean communication channels?
- jjgreen 3y ago
  I'd assumed the OP meant O(n^2), and n(n-1)/2 = O(n^2)
hot_gril 3y ago
Yep, I work in a large org that used to be a monolith (which means single DB really). Was a mess for the reasons you'd expect. Even our subteam of 10 needs to split things up more.
baobabKoodaa 3y ago
I've never seen teams organized around microservices like that. What I've seen, again and again, is one huge team where "everyone is responsible for all the microservices" (meaning, no-one is responsible for anything). On a theory level I would agree with you - I've just never seen that happen in practice.
- misja111 3y ago
  I'm not that much of a supporter of microservices, but my experience is the opposite: in every company I've worked for that used microservices, each team had their own set of microservices they were responsible for.
  - ris58h 3y ago
    It should be like that but in practice sometimes other team just too busy to implement required feature. I can say my PM that we have to wait for a month but in reality I will have to implement it myself.
mirekrusin 3y ago
People seem to forget they can create separate directories in their codebase. They solve "people problem" by converting trivial technical problem into complex distributed system problem. Well, now you have a _Problem_.
- mattacular 3y ago
  The complexities of software development could be solved with this one weird trick - if only programmers remembered FOLDERS. What an absurd thing to suggest.
- mmcnl 3y ago
  Sure teams in large organizations allow other teams to randomly create folders? Also I've rarely seen an internal code base that is easy to grasp. You can create your own microservice and set up an API in 1/5th of the time it takes to understand a foreign code base and make a small adjustment.
chalcolithic 3y ago
I wonder why people ever see it differently?
gscott 3y ago
Maybe a solution to an anti-social problem! FT.com did a recorded seminar session on their microservices architecture and one of the benefits they extolled is if someone wanted to improve on a feature they could just make it all over again and replace the old microservice with a new one. No need to look at the last persons code, just blow it away like it never existed. I gathered their site is actually a black box filled with hundreds of black boxes of microservices. All a mystery, they either work or they don't and if they don't they fail gracefully quickly. https://www.youtube.com/watch?v=_qakAUjXiek https://www.youtube.com/watch?v=_qakAUjXiek
JackMorgan 3y ago
I'm honestly not even super convinced that small teams struggle to maintain large systems. I've been on a team that was only 7 good engineers maintaining a 3.5 million line project that had both a web UI and thick client. It supported 2 different databases and had a horizontally scalable job runner. At one point it was 35 engineers, but layoffs took it down to 7, at which point we started to get a lot more done. There was just so much less time spent keeping everyone aligned. So many fewer meetings, sign-offs, reviews, plannings, retrospectives, management meetings, etc. Developers had a lot more agency, so they just got stuff done. Technical debt repayment became 50% of our time, as we easily knocked out features in the other half of the time. We kept ruthlessly cutting complexity, so it got faster to add new features. I'm sure some projects just need more bodies, but I think there's an upper bound to how much complexity can be added to a system in a given unit of time. Adding developers over a threshold will result in the same amount of features per week, just everyone does a little less and spends a little more time on communication. Repeat up to thousands of developers where adding a single field takes months.
- tonyedgecombe 3y ago
  >At one point it was 35 engineers, but layoffs took it down to 7, at which point we started to get a lot more done. Years ago I did two back to back contracts for two different pharmaceutical companies. They were both about the same size but one had an IT group that was ten times the size of the other. You can guess which project was late and painful.
crabbone 3y ago
Microservices are not a solution to the problem you describe. Nothing in your problem description requires the micro part. Microservices are about splitting the application into very fine-grained sub-applications. It's not about modularity in general, it's about making things as modular as possible (up to one function per service). That's why they are micro. Otherwise we'd just call them "services" and nobody would have any problem with that.
nijave 3y ago
>Splitting a system into subsystems allows each team to focus on their piece of the puzzle while minimizing the amount of peer-to-peer coordination. Assuming coupling is reasonable. If you have a "distributed monolith", you still end up with all the meetings because every microservice change risks breaking interfaces other people are using. In the context of coupling, I'd argue the same applies to monoliths. Multiple teams can successfully work on a monolith given architecture where they're not constantly stepping in each other's toes (each team works mostly in their own modules/classes/packages)
DrScientist 3y ago
I get the point about creating boundaries to document the dependencies - however if your language supports packages and private keywords you can do that without having microservices. And once you've split up your app into lots of independent microservices who owns the arrows between the microservice boxes? ( The actual app ).
vinay_ys 3y ago
Ummm, extremely large teams develop monolith (single binary) things called OSes or databases etc. The modularity for scaling developers cross-communication comes from.... modules! duh! Aka libraries.
mmcnl 3y ago
Exactly, I missed this completely in the article. In my experience, microservices attempt to solve organizational problems and not technical problems. There are technical downsides to microservices that may be outweighed by organizational benefits. With monoliths you might have a large amount of hidden opportunity cost that technically never surface. I do agree that this is less of a problem for startups so it doesn't make sense to start with a complex microservice architecture. But in large organizations, especially corporates, it absolutely makes sense.
dahwolf 3y ago
"Splitting a system into subsystems allows each team to focus on their piece of the puzzle while minimizing the amount of peer-to-peer coordination." The coordination is still there because a microservice team does not live in a vacuum. They build services based on demand from other teams that typically build web apps, mobile apps, sometimes server-to-server. Hence, the "independent" team now becomes a roadblock for higher order features.

by synack 3y ago

meowtimemania 3y ago

if you have a well modularized monolith, you can get best of both worlds.

synack 3y ago
Agreed! Software modules and libraries can achieve the same independence of subsystems without implying a network topology.
jayd16 3y ago
Not really. You can't really reduce the blast radius of crashes or bad deployments. You need to have the discipline of a good CI/CD instead of siloed but decoupled workflows. Just keeping things neat doesn't go nearly as far as a separate process on separate machines. Monolith might be better but I don't think it's a situation where you can have it all.
- aidos 3y ago
  This argument always confuses me. It depends on what you’re doing but if you’re doing web services, as most here are, the crash is limited to the request being served. The blast radius is a single request, right? In every likelihood it _is_ a separate process on a separate machine.
  - threeseed 3y ago
    They are talking about deployment. With micro-services if you screw up a deployment the service is down and if your platform is well designed the system will be degraded but not down. With a monolith the whole system is down.
    - quickthrower2 3y ago
      There are a lot of deployments options to mitigate that risk.
    - charcircuit 3y ago
      >With a monolith the whole system is down. No, you are just at reduced capacity. The moral of the story is not to roll out to 100% right away.
      - threeseed 3y ago
        Monoliths go hand in hand with a centralised database. Bit hard to do schema evolution with a staggered rollout.
        charcircuit 3y ago
        The software should be compatible with both the old and new schema until all of your database servers have moved over to the new schema. All rollouts are going to be somewhat staggered even if you go straight to 100%.
        DougBTX 3y ago
        If schema migrations take non-zero time and you want zero-downtime deployments, then the both service versions need to be compatible with either the new schema or the old schema, regardless of service size.
        ownagefool 3y ago
        The same problem exists in microservoce land, only you now have a load of separate teams managing their own database, perhaps with different solutions.
- DougBTX 3y ago
  > You need to have the discipline of a good CI/CD Good CD is even more important if there are more services to deploy
croo 3y ago
That is a big if that can be easily broken by a new guy who don't know the rules or if the pace is fast enough that you cannot review everything. If the codebase is different you can force the separation not just ask nicely to keep the code well modularized.
- meowtimemania 3y ago
  You can enforce modularity with git acls, and multi-module workspaces within a single git repository. for example: golang - multi-module workspaces javascript - yarn workspaces With this setup, you can create modules a, b and c. And then add restrictions about which which modules are available to a given module
mmcnl 3y ago
Yes, only it takes a while to figure out the right level of abstraction for your organization. It's difficult to start right out of the box with the right type of modularization.

corethree 3y ago

Why do you have to split things into services? How about moving things into different folders. Have you thought about that? Why do you have to modularize it with a whole new repo, a whole new docker set up? Just use a folder bro.

abhiyerra 3y ago
I found Django apps to be a good middle ground.
hot_gril 3y ago
Usually the most important designation of the separate system is that it uses a separate database. You only have immediate consistency within one service.
- aledalgrande 3y ago
  lol good luck without centralized data
  - hot_gril 3y ago
    Each service is authoritative on its own data, and it works neatly. Any medium to large size system is going to have natural places to separate that, or else will buckle under complexity or even hardware constraints if you try to keep it all in one DB. I've seen it repeatedly.
    - corethree 3y ago
      Nah this doesn't work out. Because requirements are so chaotic and modular design isn't an axiomatic science things will for sure go wrong with the design both because you got the design wrong and because the requirements shifted to a point where another design was better. You encapsulated data under one service and you find that it's actually utilized much more in another service. This happens a lot. The longer you can centralize your storage the better and easier everything is.

brhsagain 3y ago

> Splitting a system into subsystems allows each team to focus on their piece of the puzzle while minimizing the amount of peer-to-peer coordination. This does not happen at all. When you break a system into subsystems, all the previous connections that get remapped to new connections between subsystems still need to happen, in order to solve the fundamental problem that the system solves — except now instead of just making the connection directly, there has to be a "cross-functional" meeting between teams and a complicated communication layer between the systems. And if somehow you find a breakdown that requires minimal connections between subsystems, then those connections wouldn't have existed in the original system either, and the N² problem doesn't exist.

jedberg 3y ago
If that's the experience then you're doing services wrong. Each service should have its own datastore and a single API. The interface between services should be a single connection. There should be maybe one meeting where the caller defines what they need the service to return to them.
- 10000truths 3y ago
  The problem with this is that you have to be really damned careful how you split things up. If your separate data stores end up having to be joined together later on because some new business feature requires them to be cross-referenced, you're painted into one of two corners: 1. Merge the two services (and their data stores) into one and cause havoc downstream of either service 2. Burn through your network latency/throughput budget trying to reinvent a DB join across RPC boundaries (and god forbid if you can't batch multiple lookups in a single API call!)
  - dahwolf 3y ago
    Exactly. Software developers will never learn that the separation of concerns is a myth. In reality, UI, business logic and data are deeply entangled. Hence, moving these things apart makes everything worse.
- 3y ago

deleted 3y ago

[deleted]

politelemon 3y ago

Shouldn't it be n(n-1)/2 coordination. Assuming you mean communication channels?

jjgreen 3y ago
I'd assumed the OP meant O(n^2), and n(n-1)/2 = O(n^2)

hot_gril 3y ago

Yep, I work in a large org that used to be a monolith (which means single DB really). Was a mess for the reasons you'd expect. Even our subteam of 10 needs to split things up more.

baobabKoodaa 3y ago

I've never seen teams organized around microservices like that. What I've seen, again and again, is one huge team where "everyone is responsible for all the microservices" (meaning, no-one is responsible for anything). On a theory level I would agree with you - I've just never seen that happen in practice.

misja111 3y ago
I'm not that much of a supporter of microservices, but my experience is the opposite: in every company I've worked for that used microservices, each team had their own set of microservices they were responsible for.
- ris58h 3y ago
  It should be like that but in practice sometimes other team just too busy to implement required feature. I can say my PM that we have to wait for a month but in reality I will have to implement it myself.

mirekrusin 3y ago

People seem to forget they can create separate directories in their codebase. They solve "people problem" by converting trivial technical problem into complex distributed system problem. Well, now you have a _Problem_.

mattacular 3y ago
The complexities of software development could be solved with this one weird trick - if only programmers remembered FOLDERS. What an absurd thing to suggest.
mmcnl 3y ago
Sure teams in large organizations allow other teams to randomly create folders? Also I've rarely seen an internal code base that is easy to grasp. You can create your own microservice and set up an API in 1/5th of the time it takes to understand a foreign code base and make a small adjustment.

chalcolithic 3y ago

I wonder why people ever see it differently?

gscott 3y ago

Maybe a solution to an anti-social problem! FT.com did a recorded seminar session on their microservices architecture and one of the benefits they extolled is if someone wanted to improve on a feature they could just make it all over again and replace the old microservice with a new one. No need to look at the last persons code, just blow it away like it never existed. I gathered their site is actually a black box filled with hundreds of black boxes of microservices. All a mystery, they either work or they don't and if they don't they fail gracefully quickly. https://www.youtube.com/watch?v=_qakAUjXiek https://www.youtube.com/watch?v=_qakAUjXiek

JackMorgan 3y ago

I'm honestly not even super convinced that small teams struggle to maintain large systems. I've been on a team that was only 7 good engineers maintaining a 3.5 million line project that had both a web UI and thick client. It supported 2 different databases and had a horizontally scalable job runner. At one point it was 35 engineers, but layoffs took it down to 7, at which point we started to get a lot more done. There was just so much less time spent keeping everyone aligned. So many fewer meetings, sign-offs, reviews, plannings, retrospectives, management meetings, etc. Developers had a lot more agency, so they just got stuff done. Technical debt repayment became 50% of our time, as we easily knocked out features in the other half of the time. We kept ruthlessly cutting complexity, so it got faster to add new features. I'm sure some projects just need more bodies, but I think there's an upper bound to how much complexity can be added to a system in a given unit of time. Adding developers over a threshold will result in the same amount of features per week, just everyone does a little less and spends a little more time on communication. Repeat up to thousands of developers where adding a single field takes months.

tonyedgecombe 3y ago
>At one point it was 35 engineers, but layoffs took it down to 7, at which point we started to get a lot more done. Years ago I did two back to back contracts for two different pharmaceutical companies. They were both about the same size but one had an IT group that was ten times the size of the other. You can guess which project was late and painful.

crabbone 3y ago

Microservices are not a solution to the problem you describe. Nothing in your problem description requires the micro part. Microservices are about splitting the application into very fine-grained sub-applications. It's not about modularity in general, it's about making things as modular as possible (up to one function per service). That's why they are micro. Otherwise we'd just call them "services" and nobody would have any problem with that.

nijave 3y ago

>Splitting a system into subsystems allows each team to focus on their piece of the puzzle while minimizing the amount of peer-to-peer coordination. Assuming coupling is reasonable. If you have a "distributed monolith", you still end up with all the meetings because every microservice change risks breaking interfaces other people are using. In the context of coupling, I'd argue the same applies to monoliths. Multiple teams can successfully work on a monolith given architecture where they're not constantly stepping in each other's toes (each team works mostly in their own modules/classes/packages)

DrScientist 3y ago

I get the point about creating boundaries to document the dependencies - however if your language supports packages and private keywords you can do that without having microservices. And once you've split up your app into lots of independent microservices who owns the arrows between the microservice boxes? ( The actual app ).

vinay_ys 3y ago

Ummm, extremely large teams develop monolith (single binary) things called OSes or databases etc. The modularity for scaling developers cross-communication comes from.... modules! duh! Aka libraries.

mmcnl 3y ago

Exactly, I missed this completely in the article. In my experience, microservices attempt to solve organizational problems and not technical problems. There are technical downsides to microservices that may be outweighed by organizational benefits. With monoliths you might have a large amount of hidden opportunity cost that technically never surface. I do agree that this is less of a problem for startups so it doesn't make sense to start with a complex microservice architecture. But in large organizations, especially corporates, it absolutely makes sense.

dahwolf 3y ago

"Splitting a system into subsystems allows each team to focus on their piece of the puzzle while minimizing the amount of peer-to-peer coordination." The coordination is still there because a microservice team does not live in a vacuum. They build services based on demand from other teams that typically build web apps, mobile apps, sometimes server-to-server. Hence, the "independent" team now becomes a roadblock for higher order features.