10 ms·

This is what I tell engineers. Microservices aren't a performance strategy. They are a POTENTIAL cost saving strategy against performance. And an engineering co

This is what I tell engineers. Microservices aren't a performance strategy. They are a POTENTIAL cost saving strategy against performance. And an engineering coordination strategy. Theoretically If you have a monolith that can be scaled horizontally there isn't any difference between having 10 replicas of your monolith and having 5 replicas of two microservices with the same codebase. UNLESS you are trying to underscale part of your functionality. You can't underscale part of your app with a monolith. Your pipe has to be big enough for all of it. Generally speaking though if you are talking about 10 replicas of something there's very little money to be saved anywhere. Even then though the cost savings only start at large scales. You need to have a minimum of 3 replicas for resiliency. If those 3 replicas are too big for your scale then you are just wasting money. The place where I see any real world benefit for most companies is just engineering coordination. With a single repo for a monolith I can make 1 team own that repo and tell them it's their responsibility to keep it clean. In a shared monolith however 0 people own it because everyone owns it and the repo becomes a disaster faster than you can say "we need enterprise caching".

scient 3y ago
Services, or even microservices, are more of a strategy to allow teams to scale than services or products to scale. I think thats one of the biggest misconceptions for engineers. On the other end you have the monorepo crew, who are doing it for the same reasons. On your note about resiliency and scale - its always a waste of money until shit hits the fan. Then you really pay for it.
- manicennui 3y ago
  Except that most people build microservices in a way that ignores the reality of cloud providers and the fact that they are building (more) distributed systems, and often end up with lower resiliency.
- dns_snek 3y ago
  How do you square that with the fact that shit usually hits the fan precisely because of this complexity, not in spite of it? That's my observation & experience, anyway. Added bits of "resiliency" often add brand new, unexplored failure points that are just ticking time bombs waiting to bring the entire system down.
  - bluGill 3y ago
    Not adding that resiliency isn't the answer though - it just means known failures will get you. Is that better than the unknown failures because of your mitigation? I cannot answer that. I can tell you 100% that eventually a disk will fail. I can tell you 100% that eventually the power will go out. I can tell you 100% that even if you have a computer with redundant power supplies each connected to separate grids, eventually both power supplies will fail at the same time - it just will happen a lot less often than if you have a regular computer not on any redundant/backup power. I can tell you that network cables do break from time to time. I can tell you that buildings are vulnerable to earthquakes, fires, floods, tornadoes and other such disasters). I can tell you that software is not perfect and eventually crashes. I can tell you that upgrades are hard if any protocol changed. I can tell you there is a long list of other known disasters that I didn't list, but a little research will discover. I could look up the odds of the above. In turn this allows calculating the costs of each mitigation against the likely cost of not mitigating it - but this is only statistical you may decide something statistically cannot happen and it does anyway. What I cannot tell you is how much you should mitigate. There is a cost to each mitigation that need to be compared to the value.
    - camgunz 3y ago
      Absolutely yeah, these things are hard enough to test in a controlled environment with a single app (e.g. FoundationDB) but practically impossible to test fully in a microservices architecture. It's so nice to have this complexity managed for you in the storage layer.
  - troupe 3y ago
    > How do you square that with the fact that shit usually hits the fan precisely because of this complexity The theoretical benefit may not be what most teams are going to experience. Usually the fact that microservices are seen as a solution to a problem that could more easily be solved in other much simpler ways, is a pretty good indication that any theoretical benefits are going to be lost through other poor decision making.
  - Eridrus 3y ago
- camgunz 3y ago
  > Services, or even microservices, are more of a strategy to allow teams to scale than services or products to scale. I've never really understood why you couldn't just break up your monolith into modules. So like if there's a "payments" section, why isn't that API stabilized? I think all the potential pitfalls (coupling, no commitment to compatibility) are there for monoliths and microservices, the difference is in the processes. For example, microservices export some kind of API over REST/GraphQL/gRPC which they can have SDKs for, they can version them, etc. Why can't you just define interfaces to modules within your monolith? You can generate API docs, you can version interfaces, you can make completely new versions, etc. I just feel like this would be a huge improvement: - It's so much more engineering work to build the service handler scaffolding (validation, serialization/deserialization, defining errors) - You avoid the runtime overhead of serialiation/deserialization and network latency - You don't need to build SDKs/generate protobufs/generate clients/etc. - You never have the problem of "is anyone using this service?" because you can use code coverage tools - Deployment is much, much simpler - You never have the problem of "we have to support this old--sometimes broken--functionality because this old service we can't modify depends on it". This is a really undersold point: maybe it's true that microservice architectures let engineers build things without regard for other teams, but they can't remove things without regard for other teams, and this dynamic is like a no limit credit card for tech debt. Do you keep that service around as it slowly accretes more and more code it can't delete? Do you fork a new service w/o the legacy code and watch your fleet of microservices grow ever larger? - You never have the problem of "how do we update the version of Node on 50 microservices?"
  - mjr00 3y ago
    > You never have the problem of "how do we update the version of Node on 50 microservices?" And instead you have the problem of "how do we update the version of Node on our 10 million LOC codebase?" Which is, in my experience, an order of magnitude harder. Ease of upgrading the underlying platform versions of Node, Python, Java, etc is one of the biggest benefits of smaller, independent services.
    - naasking 3y ago
      Upgrading the platform also happens at least 10x less frequently, so that math doesn't necessarily work out in your favour though.
- danielovichdk 3y ago
  Microservices is more about organisation than it is about technology. And that is why developers have so much trouble getting it right. They can't without having the organisational fundamentals in place. It is simply not possible. The architectural constraints of microservices will show the organisational weaknesses in a much higher rate because of the pressure it puts on having the organisation be very strict about ownership, communication and autonomy. The takes a higher level of maturity as an organisation to enable the benefits of microservies, which is also why most organisations shouldn't even try. Stop all the technical nonsense because it won't solve the root cause of the matter. It's the organisation. Not the technology
nijave 3y ago
Microservices can help with performance by splitting off performance critical pieces and allowing you to rewrite in a different stack or language (Rust or go instead of Ruby or Python) But yeah, they also tend to explode complexity
- dns_snek 3y ago
  An important point that people seem to forget is that you don't need microservices to invoke performant native code, just a dynamic library and FFI support. The entire Python ecosystem is built around this idea.
  - bunderbunder 3y ago
    This is why Python stole enterprise big data and machine learning from Java. It actually has a higher performance ceiling for certain specific situations because, almost uniquely among garbage collected high-level languages, it can call native code without marshaling or memory pinning.
  - nijave 3y ago
    You don't need it but you can also explode your repo, test, and build complexity when it'd be easier to keep them isolated. For instance, you might not want to require all develops have a C tool chain installed with certain libraries for a tiny bit of performance optimized code that almost never gets updated.
    - dns_snek 3y ago
      I don't know, that seems like confusion of runtime and code organization boundaries. Adding a network boundary in production just to serve some code organization purpose during development seems completely unnecessary to me. For development purposes you could build and distribute binary artifacts just like you would for any other library. Developers who don't touch native code can just fetch the pre-built binaries corresponding to current commit hash (e.g. from CI artifacts).
duped 3y ago
> Microservices aren't a performance strategy Who thinks that more i/o and more frequent cold starts are more performant? Where I see micoservices as very useful is for elasticity and modularization. It's probably slower than a monolith at your scale but you don't want to fallover when loads start increasing and you need to scale horizontally. Microservices with autoscaling can make that very useful. But of course, updating services can be a nightmare. It's a game of tradeoffs.
camgunz 3y ago
> Microservices aren't a performance strategy. They are a POTENTIAL cost saving strategy against performance. Yeah they were a way to handle databases that couldn't scale horizontally. You could move business logic out of the database/SQL and into Java/Python/TypeScript app servers you could spin up. Now that we have databases like BigQuery and CockroachDB we don't need to do this anymore.
- thesnide 3y ago
  If you plan to scale horizontally with a microservice strategy, you'll in a lot of pain. As if you have say 100x more customers, will you divide your application in 100x more services? As others said, microservices scale manpower, not performance. It only comes sometimes as a bonus.
jakey_bakey 3y ago
Surely one could split a shared monolith into many internal libraries and modules, facilitating ownership?
- troupe 3y ago
  Yes, but you are still dealing with situations where other teams are deploying code that you are responsible for. With microservices you can always say, "our microservice wasn't deployed, so it is someone else's problem." But I think you are pointing out one of the reasons most places don't get benefits from microservices. If their culture doesn't let them do as you describe with a monolith, the overhead of microservices just bring in additional complexities to an already dysfunctional team/organization.
- manicennui 3y ago
  It's crazy to me how opposed to building libraries everyone seems these days. We use a fuckton of libraries, but the smallest division of software we'll build is a service.
  - dalyons 3y ago
    its not crazy to me... ive worked at a couple of places that decided to try to use libraries in a web services context. They were all disasters, and the libraries ended up being considered severe tech debt. The practical aspects of updating all the consumers of your library, and how the libraries interact with persistence stores end up being their undoing.
    - manicennui 3y ago
      How do you square this with the fact that you likely used dozens of libraries from third parties?
      - dalyons 3y ago
        probably 1000s hah. It's the speed of change. third party libraries dont change much, or fast - and you dont really want them to. The software you build internally at a product company with 500+ engineers changes much much faster. And you want it to.
wg0 3y ago
You can scale those things with libraries also. The very browser that you're using to read this comment is an example of it. FreeType, Hurfbaz, Pango, Cairo, Uniscribe, GDI, zlib this that and a deep deep dependency tree built by people who never have talked to each other directly other than the documentation of their libraries - works perfectly well. I assure you 98% of the companies have simpler and shallower code base than that of a modern A class browser. Microservices was a wrong turn in our industry for 98% of the use cases.
- Tainnor 3y ago
  > Hurfbaz This made me chuckle. :) It's HarfBuzz. Yours sounds like a goblin.
  - wg0 3y ago
    I could't recall the correct name but originally, it is in Persian حرف باز and in that باز would be transliterated as Baz. But yes, you're right.
    - Tainnor 3y ago
      I wasn't even aware that this came from Persian! Thanks for educating me.
chasd00 3y ago
> And an engineering coordination strategy. i've always felt the org chart defines the microservice architecture. It's a way to keep teams out of each other's hair. When you have the same dev working on more than one service then that's an indication you're headed for trouble.
- wesselbindt 3y ago
  That's not just a feeling, it's a direct consequence of Conway's law. You feel correctly.
mhh__ 3y ago
I view micro services as a risk mainly in a Conway sense rather than technology. Most companies can run on a VPS.
danenania 3y ago
I mostly agree. I'd add though that isolation can be a legitimate reason for a microservice. For example, if you have some non-critical logic that potentially uses a lot of CPU, splitting that out can make sense to be sure it's not competing for CPU with your main service (and bringing it down in the pathological case). Similarly, if you have any critical endpoints that are read-only, splitting those out from the main service where writes occur can improve the reliability of those endpoints.
otabdeveloper4 3y ago
> Your pipe has to be big enough for all of it. What do you mean by "pipe" here? It's easier to share CPU and network bandwidth across monolith threads than it is across microservice instances. (In fact, that is the entire premise of virtualization - a VM host is basically a way to turn lots of disparate services into a monolith.)
jmull 3y ago
> In a shared monolith however 0 people own it because everyone owns it and the repo becomes a disaster faster than you can say "we need enterprise caching". * I give two teams one repo each and tell each one: "This is your repo, keep it clean." -or- * I give two teams one folder each in a repo and tell each one: "This is your folder, keep it clean." If you've got a repo or folder (or anything else) that no one is responsible for, that's a management problem, and micro services won't solve it. Repos (and folders) don't really have anything to do with organizing a complex system of software -- they are just containers whose logical organization should follow from the logical organization of the system. Microservices back you into position where when the code of one team to calls the component of another team, it has to be high-latency, fault-prone, low-granularity, config-heavy. Some stuff falls the way anyway, so no problem in those case, but why burn that limitation in to your software architecture from the start? Just so you don't have to assign teams responsibility for portions of a repo?
- leeoniya 3y ago
  i think doing releases, deployments, and rollbacks is trickier in a monorepo. with multiple services/repos, each team can handle their own release cycles, on-call rotation, and only deal with the bits, tests, and failures they wrote. you lose some CPU/RAM/IO efficiency, but you gain autonomy and resilience, and faster time to resolution. https://blog.nelhage.com/post/efficiency-vs-resiliency/ https://blog.nelhage.com/post/efficiency-vs-resiliency/ e.g. at Grafana we're working through some of these decoupling-to-microservices challenges, because the SREs that deploy to our infra should not need to deal with investigating and rolling back a whole monolith due to some regression introduced by some specific team to a core datasource plugin, or frontend plugin/panel. the pain at scale is very real, and engineering hours are by far more expensive than the small efficiency loss you get by over-provisioning the metal a bit.
  - jmull 3y ago
    I didn't mean to say there are no reasons to have separate repos, just more responding to the post above mine. But I will note that almost all the benefits listed aren't specific to micro services. (Also, to me, it's flawed that a dependency can really push to production without coordination. At the least, dependent components should decide when to "take" a new version. There are lot of ways to manage that -- versioned endpoints, e.g., so that dependents can take the new when they are ready. But it's extra complexity that's tricky in practice. e.g., you really ought to semver the endpoints, and keep versions as long as there could be any dependents... but how many is that, and how do you know? From what I've seen, people don't do that and instead dependencies push new versions, ready or not, and just deal with the problems as breakages in production, which is pretty terrible.)
roenxi 3y ago
> The place where I see any real world benefit for most companies is just engineering coordination. With a single repo for a monolith I can make 1 team own that repo and tell them it's their responsibility to keep it clean. In a shared monolith however 0 people own it because everyone owns it and the repo becomes a disaster faster than you can say "we need enterprise caching". Microservices supposed to be so small that they are much smaller than a team's responsibility; so I don't think that is a factor. The bottleneck in a software engineering company is always leadership and social organisation (think Google - they tried to muscle in on any number of alternative billion dollar businesses and generally failed despite having excellent developers who assembled the required parts rapidly). Microservices won't save you from a a manger who doesn't think code ownership is important. Monoliths won't prevent such a manger from making people responsible for parts of the system. I've worked on monoliths with multiple teams involved. It seems natural that large internal modules start to develop and each team has their own areas. A little bit like an amoeba readying to split, but never actually going through with it.
upupupandaway 3y ago
> You can't underscale part of your app with a monolith. Your pipe has to be big enough for all of it. Not necessarily. I owned one of Amazon’s most traffic-heavy services (3 million TPS). It was a monolith with roughly 20 APIs, all extremely high volume. To keep the monolith simple and be able to scale it up or down independently, the thousands of EC2 instances running it were separated in fleets, which serving a specific use case. Then we could for example scale the website fleet while keeping the payment fleet stable. The only catch is that teams needed to be allowed to call only the APIs representing the use case of that fleet (no calling payment-related APIs on the website fleet). Given the low number of APIs and basic ACL control, it was not a challenge.
begueradj 3y ago
Keep it simple :)

scient 3y ago

Services, or even microservices, are more of a strategy to allow teams to scale than services or products to scale. I think thats one of the biggest misconceptions for engineers. On the other end you have the monorepo crew, who are doing it for the same reasons. On your note about resiliency and scale - its always a waste of money until shit hits the fan. Then you really pay for it.

manicennui 3y ago
Except that most people build microservices in a way that ignores the reality of cloud providers and the fact that they are building (more) distributed systems, and often end up with lower resiliency.
dns_snek 3y ago
How do you square that with the fact that shit usually hits the fan precisely because of this complexity, not in spite of it? That's my observation & experience, anyway. Added bits of "resiliency" often add brand new, unexplored failure points that are just ticking time bombs waiting to bring the entire system down.
- bluGill 3y ago
  Not adding that resiliency isn't the answer though - it just means known failures will get you. Is that better than the unknown failures because of your mitigation? I cannot answer that. I can tell you 100% that eventually a disk will fail. I can tell you 100% that eventually the power will go out. I can tell you 100% that even if you have a computer with redundant power supplies each connected to separate grids, eventually both power supplies will fail at the same time - it just will happen a lot less often than if you have a regular computer not on any redundant/backup power. I can tell you that network cables do break from time to time. I can tell you that buildings are vulnerable to earthquakes, fires, floods, tornadoes and other such disasters). I can tell you that software is not perfect and eventually crashes. I can tell you that upgrades are hard if any protocol changed. I can tell you there is a long list of other known disasters that I didn't list, but a little research will discover. I could look up the odds of the above. In turn this allows calculating the costs of each mitigation against the likely cost of not mitigating it - but this is only statistical you may decide something statistically cannot happen and it does anyway. What I cannot tell you is how much you should mitigate. There is a cost to each mitigation that need to be compared to the value.
  - camgunz 3y ago
    Absolutely yeah, these things are hard enough to test in a controlled environment with a single app (e.g. FoundationDB) but practically impossible to test fully in a microservices architecture. It's so nice to have this complexity managed for you in the storage layer.
- troupe 3y ago
  > How do you square that with the fact that shit usually hits the fan precisely because of this complexity The theoretical benefit may not be what most teams are going to experience. Usually the fact that microservices are seen as a solution to a problem that could more easily be solved in other much simpler ways, is a pretty good indication that any theoretical benefits are going to be lost through other poor decision making.
- Eridrus 3y ago
camgunz 3y ago
> Services, or even microservices, are more of a strategy to allow teams to scale than services or products to scale. I've never really understood why you couldn't just break up your monolith into modules. So like if there's a "payments" section, why isn't that API stabilized? I think all the potential pitfalls (coupling, no commitment to compatibility) are there for monoliths and microservices, the difference is in the processes. For example, microservices export some kind of API over REST/GraphQL/gRPC which they can have SDKs for, they can version them, etc. Why can't you just define interfaces to modules within your monolith? You can generate API docs, you can version interfaces, you can make completely new versions, etc. I just feel like this would be a huge improvement: - It's so much more engineering work to build the service handler scaffolding (validation, serialization/deserialization, defining errors) - You avoid the runtime overhead of serialiation/deserialization and network latency - You don't need to build SDKs/generate protobufs/generate clients/etc. - You never have the problem of "is anyone using this service?" because you can use code coverage tools - Deployment is much, much simpler - You never have the problem of "we have to support this old--sometimes broken--functionality because this old service we can't modify depends on it". This is a really undersold point: maybe it's true that microservice architectures let engineers build things without regard for other teams, but they can't remove things without regard for other teams, and this dynamic is like a no limit credit card for tech debt. Do you keep that service around as it slowly accretes more and more code it can't delete? Do you fork a new service w/o the legacy code and watch your fleet of microservices grow ever larger? - You never have the problem of "how do we update the version of Node on 50 microservices?"
- mjr00 3y ago
  > You never have the problem of "how do we update the version of Node on 50 microservices?" And instead you have the problem of "how do we update the version of Node on our 10 million LOC codebase?" Which is, in my experience, an order of magnitude harder. Ease of upgrading the underlying platform versions of Node, Python, Java, etc is one of the biggest benefits of smaller, independent services.
  - naasking 3y ago
    Upgrading the platform also happens at least 10x less frequently, so that math doesn't necessarily work out in your favour though.
danielovichdk 3y ago
Microservices is more about organisation than it is about technology. And that is why developers have so much trouble getting it right. They can't without having the organisational fundamentals in place. It is simply not possible. The architectural constraints of microservices will show the organisational weaknesses in a much higher rate because of the pressure it puts on having the organisation be very strict about ownership, communication and autonomy. The takes a higher level of maturity as an organisation to enable the benefits of microservies, which is also why most organisations shouldn't even try. Stop all the technical nonsense because it won't solve the root cause of the matter. It's the organisation. Not the technology

nijave 3y ago

Microservices can help with performance by splitting off performance critical pieces and allowing you to rewrite in a different stack or language (Rust or go instead of Ruby or Python) But yeah, they also tend to explode complexity

dns_snek 3y ago
An important point that people seem to forget is that you don't need microservices to invoke performant native code, just a dynamic library and FFI support. The entire Python ecosystem is built around this idea.
- bunderbunder 3y ago
  This is why Python stole enterprise big data and machine learning from Java. It actually has a higher performance ceiling for certain specific situations because, almost uniquely among garbage collected high-level languages, it can call native code without marshaling or memory pinning.
- nijave 3y ago
  You don't need it but you can also explode your repo, test, and build complexity when it'd be easier to keep them isolated. For instance, you might not want to require all develops have a C tool chain installed with certain libraries for a tiny bit of performance optimized code that almost never gets updated.
  - dns_snek 3y ago
    I don't know, that seems like confusion of runtime and code organization boundaries. Adding a network boundary in production just to serve some code organization purpose during development seems completely unnecessary to me. For development purposes you could build and distribute binary artifacts just like you would for any other library. Developers who don't touch native code can just fetch the pre-built binaries corresponding to current commit hash (e.g. from CI artifacts).

duped 3y ago

> Microservices aren't a performance strategy Who thinks that more i/o and more frequent cold starts are more performant? Where I see micoservices as very useful is for elasticity and modularization. It's probably slower than a monolith at your scale but you don't want to fallover when loads start increasing and you need to scale horizontally. Microservices with autoscaling can make that very useful. But of course, updating services can be a nightmare. It's a game of tradeoffs.

camgunz 3y ago

> Microservices aren't a performance strategy. They are a POTENTIAL cost saving strategy against performance. Yeah they were a way to handle databases that couldn't scale horizontally. You could move business logic out of the database/SQL and into Java/Python/TypeScript app servers you could spin up. Now that we have databases like BigQuery and CockroachDB we don't need to do this anymore.

thesnide 3y ago
If you plan to scale horizontally with a microservice strategy, you'll in a lot of pain. As if you have say 100x more customers, will you divide your application in 100x more services? As others said, microservices scale manpower, not performance. It only comes sometimes as a bonus.

jakey_bakey 3y ago

Surely one could split a shared monolith into many internal libraries and modules, facilitating ownership?

troupe 3y ago
Yes, but you are still dealing with situations where other teams are deploying code that you are responsible for. With microservices you can always say, "our microservice wasn't deployed, so it is someone else's problem." But I think you are pointing out one of the reasons most places don't get benefits from microservices. If their culture doesn't let them do as you describe with a monolith, the overhead of microservices just bring in additional complexities to an already dysfunctional team/organization.
manicennui 3y ago
It's crazy to me how opposed to building libraries everyone seems these days. We use a fuckton of libraries, but the smallest division of software we'll build is a service.
- dalyons 3y ago
  its not crazy to me... ive worked at a couple of places that decided to try to use libraries in a web services context. They were all disasters, and the libraries ended up being considered severe tech debt. The practical aspects of updating all the consumers of your library, and how the libraries interact with persistence stores end up being their undoing.
  - manicennui 3y ago
    How do you square this with the fact that you likely used dozens of libraries from third parties?
    - dalyons 3y ago
      probably 1000s hah. It's the speed of change. third party libraries dont change much, or fast - and you dont really want them to. The software you build internally at a product company with 500+ engineers changes much much faster. And you want it to.

wg0 3y ago

You can scale those things with libraries also. The very browser that you're using to read this comment is an example of it. FreeType, Hurfbaz, Pango, Cairo, Uniscribe, GDI, zlib this that and a deep deep dependency tree built by people who never have talked to each other directly other than the documentation of their libraries - works perfectly well. I assure you 98% of the companies have simpler and shallower code base than that of a modern A class browser. Microservices was a wrong turn in our industry for 98% of the use cases.

Tainnor 3y ago
> Hurfbaz This made me chuckle. :) It's HarfBuzz. Yours sounds like a goblin.
- wg0 3y ago
  I could't recall the correct name but originally, it is in Persian حرف باز and in that باز would be transliterated as Baz. But yes, you're right.
  - Tainnor 3y ago
    I wasn't even aware that this came from Persian! Thanks for educating me.

chasd00 3y ago

> And an engineering coordination strategy. i've always felt the org chart defines the microservice architecture. It's a way to keep teams out of each other's hair. When you have the same dev working on more than one service then that's an indication you're headed for trouble.

wesselbindt 3y ago
That's not just a feeling, it's a direct consequence of Conway's law. You feel correctly.

mhh__ 3y ago

I view micro services as a risk mainly in a Conway sense rather than technology. Most companies can run on a VPS.

danenania 3y ago

I mostly agree. I'd add though that isolation can be a legitimate reason for a microservice. For example, if you have some non-critical logic that potentially uses a lot of CPU, splitting that out can make sense to be sure it's not competing for CPU with your main service (and bringing it down in the pathological case). Similarly, if you have any critical endpoints that are read-only, splitting those out from the main service where writes occur can improve the reliability of those endpoints.

otabdeveloper4 3y ago

> Your pipe has to be big enough for all of it. What do you mean by "pipe" here? It's easier to share CPU and network bandwidth across monolith threads than it is across microservice instances. (In fact, that is the entire premise of virtualization - a VM host is basically a way to turn lots of disparate services into a monolith.)

jmull 3y ago

> In a shared monolith however 0 people own it because everyone owns it and the repo becomes a disaster faster than you can say "we need enterprise caching". * I give two teams one repo each and tell each one: "This is your repo, keep it clean." -or- * I give two teams one folder each in a repo and tell each one: "This is your folder, keep it clean." If you've got a repo or folder (or anything else) that no one is responsible for, that's a management problem, and micro services won't solve it. Repos (and folders) don't really have anything to do with organizing a complex system of software -- they are just containers whose logical organization should follow from the logical organization of the system. Microservices back you into position where when the code of one team to calls the component of another team, it has to be high-latency, fault-prone, low-granularity, config-heavy. Some stuff falls the way anyway, so no problem in those case, but why burn that limitation in to your software architecture from the start? Just so you don't have to assign teams responsibility for portions of a repo?

leeoniya 3y ago
i think doing releases, deployments, and rollbacks is trickier in a monorepo. with multiple services/repos, each team can handle their own release cycles, on-call rotation, and only deal with the bits, tests, and failures they wrote. you lose some CPU/RAM/IO efficiency, but you gain autonomy and resilience, and faster time to resolution. https://blog.nelhage.com/post/efficiency-vs-resiliency/ https://blog.nelhage.com/post/efficiency-vs-resiliency/ e.g. at Grafana we're working through some of these decoupling-to-microservices challenges, because the SREs that deploy to our infra should not need to deal with investigating and rolling back a whole monolith due to some regression introduced by some specific team to a core datasource plugin, or frontend plugin/panel. the pain at scale is very real, and engineering hours are by far more expensive than the small efficiency loss you get by over-provisioning the metal a bit.
- jmull 3y ago
  I didn't mean to say there are no reasons to have separate repos, just more responding to the post above mine. But I will note that almost all the benefits listed aren't specific to micro services. (Also, to me, it's flawed that a dependency can really push to production without coordination. At the least, dependent components should decide when to "take" a new version. There are lot of ways to manage that -- versioned endpoints, e.g., so that dependents can take the new when they are ready. But it's extra complexity that's tricky in practice. e.g., you really ought to semver the endpoints, and keep versions as long as there could be any dependents... but how many is that, and how do you know? From what I've seen, people don't do that and instead dependencies push new versions, ready or not, and just deal with the problems as breakages in production, which is pretty terrible.)

roenxi 3y ago

> The place where I see any real world benefit for most companies is just engineering coordination. With a single repo for a monolith I can make 1 team own that repo and tell them it's their responsibility to keep it clean. In a shared monolith however 0 people own it because everyone owns it and the repo becomes a disaster faster than you can say "we need enterprise caching". Microservices supposed to be so small that they are much smaller than a team's responsibility; so I don't think that is a factor. The bottleneck in a software engineering company is always leadership and social organisation (think Google - they tried to muscle in on any number of alternative billion dollar businesses and generally failed despite having excellent developers who assembled the required parts rapidly). Microservices won't save you from a a manger who doesn't think code ownership is important. Monoliths won't prevent such a manger from making people responsible for parts of the system. I've worked on monoliths with multiple teams involved. It seems natural that large internal modules start to develop and each team has their own areas. A little bit like an amoeba readying to split, but never actually going through with it.

upupupandaway 3y ago

> You can't underscale part of your app with a monolith. Your pipe has to be big enough for all of it. Not necessarily. I owned one of Amazon’s most traffic-heavy services (3 million TPS). It was a monolith with roughly 20 APIs, all extremely high volume. To keep the monolith simple and be able to scale it up or down independently, the thousands of EC2 instances running it were separated in fleets, which serving a specific use case. Then we could for example scale the website fleet while keeping the payment fleet stable. The only catch is that teams needed to be allowed to call only the APIs representing the use case of that fleet (no calling payment-related APIs on the website fleet). Given the low number of APIs and basic ACL control, it was not a challenge.

begueradj 3y ago

Keep it simple :)