7 ms·

I think your point while valid, it is probably a lot more nuanced. From the post it's more akin to an Amazon shared build and deployment system than "every libr

by wowohwow 10mo ago

I think your point while valid, it is probably a lot more nuanced. From the post it's more akin to an Amazon shared build and deployment system than "every library update needs to redeploy every time scenario". It's likely there's a single source of truth where you pull libraries or shared resources from, when team A wants to update the pointer to library-latest to 2.0 but the current reference of library-latest is still 1.0, everyone needs to migrate off of it otherwise things will break due to backwards compatibility or whatever. Likewise, if there's a -need- to remove a version for a vulnerability or what have you, then everyone needs to redeploy, sure, but the centralized benefit of this likely outweighs the security cost and complexity of tracking the patching and deployment process for each and every service. I would say those systems -are- and likely would be classified as micro services but from a cost and ease perspective operate within a shared services environment. I don't think it's fair to consider this style of design decision as a distributed monolith. By that level of logic, having a singular business entity vs 140 individual business entities for each service would mean it's a distributed monolith.

mjr00 10mo ago
> It's likely there's a single source of truth for where you pull libraries or shared resources from, when team A wants to update the pointer to library-latest to 2.0 but the current reference of library-latest is still 1.0, everyone needs to migrate off of it otherwise things will break due to backwards compatibility or whatever. No, this misses one of the biggest benefits of services; you explicitly don't need everyone to upgrade library-latest to 2.0 at the same time. If you do find yourself in a situation where you can't upgrade a core library like e.g. SQLAlchemy or Spring, or the underlying Python/Java/Go/etc runtime, without requiring updates to every service, you are back in the realm of a distributed monolith.
- wowohwow 10mo ago
  I disagree. Both can be true at the same time. A good design should not point to library-latest in a production setting, it should point to a stable known good version via direct reference, i.e library-1.0.0-stable. However, the world we live in, people choose pointing to latest, to avoid manual work and trust other teams did the right diligence when updating to the latest version. You can point to a stable version in the model I described and still be distributed and a micro service, while depending on a shared service or repository.
  - vlovich123 10mo ago
    You can do that but you keep missing that you’re no longer a true microservice as originally defined and envisioned, which is that you can deploy the service independently under local control. Can you imagine if Google could only release a new API if all their customers simultaneously updated to that new API? You need loose coupling between services. OP is correct that you are indeed now in a weird hybrid monolith application where it’s deployed piecemeal but can’t really be deployed that way because of tightly coupled dependencies. Be ready for a blog post in ten years how they broke apart the monolith into loosely coupled components because it was too difficult to ship things with a large team and actually have it land in production without getting reverted to an unrelated issue.
    - dmoy 10mo ago
      > Can you imagine if Google could only release a new API if all their customers simultaneously updated to that new API? You need loose coupling between services. Internal Google services: *sweating profusely* (Mostly in jest, it's obviously a different ballgame internal to the monorepo on borg)
    - GeneralMayhem 10mo ago
      Internal and external have wildly different requirements. Google internally can't update a library unless the update is either backward-compatible for all current users or part of the same change that updates all those users, and that's enforced by the build/test harness. That was an explicit choice, and I think an excellent one, for that scenario: it's more important to be certain that you're done when you move forward, so that it's obvious when a feature no longer needs support, than it is to enable moving faster in "isolation" when you all work for the same company anyway. But also, you're conflating code and services. There's a huge difference between libraries that are deployed as part of various binaries and those that are used as remote APIs. If you want to update a utility library that's used by importing code, then you don't need simultaneous deployment, but you would like to update everywhere to get it done with - that's only really possible with a monorepo. If you want to update a remote API without downtime, then you need a multi-phase rollout where you introduce a backward-compatibility mode... but that's true whether you store the code in one place or two.
- mekoka 10mo ago
  You're both right, but talking past each other. You're right that shared dependencies create a problem, but it can be the problem without semantically redefining the services themselves as a shared monolith. Imagine someone came to you with a similar problem and you concluded "distributed monolith", which may lead them to believe that their services should be merged into a single monolith. What if they then told you that it's going to be tough because these were truly separate apps, but that used the same OS wide Python install, one ran on Django/Postgres, another on Flask/SQLite, and another was on Fastapi/Mongo, but they all relied on some of the same underlying libs that are frequently updated. The more accurate finger should point to bad dependency management and you'd tell them about virtualenv or docker.
- deaddodo 10mo ago
  The dependencies they're likely referring to aren't core libraries, they're shared interfaces. If you're using protobufs, for instance, and you share the interfaces in a repo. Updating Service A's interface(s) necessitates all services dependent on communicating with it to be updated as well (whether you utilize those changes or not). Generally for larger systems, but smaller/scrappier teams, a true dependency management tree for something like this is out of scope so they just redeploy everything in a domain.
  - mjr00 10mo ago
    > If you're using protobufs, for instance, and you share the interfaces in a repo. Updating Service A's interface(s) necessitates all services dependent on communicating with it to be updated as well (whether you utilize those changes or not). This is not true! This is one of the core strengths of protobuf. Non-destructive protobuf changes, such as adding new API methods or new fields, do not require clients to update. On the server-side you do need to handle the case when clients don't send you the new data--plus deal with the annoying "was this int64 actually set to 0 or is it just using the default?" problem--but as a whole you can absolutely independently update a protobuf, implement it on the server, and existing clients can keep on calling and be totally fine. Now, that doesn't mean you can go crazy, as doing things like deleting fields, changing field numbering or renaming APIs will break clients, but this is just the reality of building distributed systems.
    - deaddodo 10mo ago
      I was oversimplifying for the sake of example, but yes you are correct. Properly managed protobufs don't require an update on strict interface expansion; so shouldn't always require a redeploy.
    - necovek 10mo ago
      What you are talking about is simply keeping the API (whether a library or a service) backwards-compatible. There are plenty strategies to achieve that, and it can be done with almost any interface layer (HTTP, protobuf, JSON, SQL, ...).
  - 10mo ago
- rbranson 10mo ago
  This is explicitly called out in the blog post in the trade-offs section. I was one of the engineers who helped make the decisions around this migration. There is no one size fits all. We believed in that thinking originally, but after observing how things played out, decided to make different trade-offs.
  - mjr00 10mo ago
    > There is no one size fits all. Totally agree. For what it's worth, based on the limited information in the article, I actually do think it was the right decision to pull all of the per-destination services back into one. The shared library problem can go both ways, after all: maybe the solution is to remove the library so your microservices are fully independent, or maybe they really should have never been independent in the first place and the solution is to put them back together. I don't think either extreme of "every line of code in the company is deployed as one service" or "every function is an independent FaaS" really works in practice, it's all about finding the right balance, which is domain-specific every time.
  - wowohwow 10mo ago
    FWIW, I think it was a great write up. It's clear to me what the rationale was and had good justification. Based on the people responding to all of my comments, it is clear people didn't actually read it and are opining without appropriate context.
  - nine_k 10mo ago
    To me it sounds like so: "We realized that we were not running microservice architecture, but rather a distributed monolith, so it made sense to make it a regular monolith". It's a decision I would wholeheartedly agree with.
    - necovek 10mo ago
      I don't think you read the post carefully enough: they were not running a distributed monolith, and every service was using different dependencies (versions of them). This meant that it was costly to maintain and caused a lot of confusion, especially with internal dependencies (shared libraries): this is the trade-off they did not like and wanted to move away from. They moved away from this in multiple steps, first one of those being making it a "distributed monolith" (as per your implied definition) by putting services in a monorepo and then making them use the same dependency versions (before finally making them a single service too).
- philwelch 10mo ago
  > If you do find yourself in a situation where you can't upgrade a core library like e.g. SQLAlchemy or Spring, or the underlying Python/Java/Go/etc runtime, without requiring updates to every service, you are back in the realm of a distributed monolith. Show me a language runtime or core library that will never have a CVE. Otherwise, by your definition, microservices don’t exist and all service oriented architectures are distributed monoliths.
- lelanthran 10mo ago
  Right, but theres a cost to having to support 12 different versions of a library in your system. Its a tradeoff
  - imtringued 10mo ago
    No, that sounds like you're breaking backwards compatibility too often. Assuming there is an update every week, you can expect all teams to update within two weeks, which means if everything goes well only two versions are active.
    - necovek 10mo ago
      That's not how things play out in practice. Say you do backwards incompatible changes "infrequently", say every 4 months, or 3 times per year. In 5 years, that's 15 versions with backwards incompatible changes. Everybody is time pressured at least sometimes, and you miss one update, and now you've got multiple backwards-incompatible updates and you need to do it carefully the next time around, meaning more time needed, meaning it needs to be scheduled and planned along with other feature work now. And then you end up with something close to a normal distribution of versions across ~140 services: some have the very old versions (v1-v4), majority are on some middle, not ancient versions but still ~7 versions behind on average (v5-v10), and only some are on the latest few versions (v11-v15). "Patch" versions can become even crazier, and yes, there will be bugs in them making them inadvertently not backwards compatible either (as not everybody updated right away to detect it). But really, I always point out to good, long lived APIs that make compromises in their API for the sake of backwards compatibility (eg. we still live with "Referer" instead of "Referrer" in HTTP, 35 years later; and it is OK!).
    - lelanthran 10mo ago
      > Assuming there is an update every week, you can expect all teams to update within two weeks Hah; I wish :-) https://www.youtube.com/watch?v=FopyRHHlt3M https://www.youtube.com/watch?v=FopyRHHlt3M
- aezart 10mo ago
  We had this problem, 119 services that all got their dependencies from a shared domain. Individual services had to depend on the exact version of libraries provided by the domain. It made updates essentially impossible. Eventually we were forced by a licensing change to move to containers, which fixed that issue but substantially increased our research usage, we went from 16 GB of RAM for the entire domain to 1.5 GB per service, with similar increases on CPU.
3rodents 10mo ago
Yes, you’re describing a distributed monolith. Microservices are independent, with nothing shared. They define a public interface and that’s it, that’s the entire exposed surface area. You will need to do major version bumps sometimes, when there are backwards incompatible changes to make, but these are rare. The logical problem you’re running into is exactly why microservices are such a bad idea for most businesses. How many businesses can have entirely independent system components? Almost all “microservice” systems in production are distributed monoliths. Real microservices are incredibly rare. A mental model for true microservices is something akin to depending on the APIs of Netflix, Hulu, HBO Max and YouTube. They’ll have their own data models, their own versioning cycles and all that you consume is the public interface.
- wowohwow 10mo ago
  This type of elitist mentality is such a problem and such a drain for software development. "Real micro services are incredibly rare". I'll repeat myself from my other post, by this level of logic nothing is a micro service. Do you depend on a cloud provider? Not a microservice. Do you depend on an ISP for Internet? Not a microservice. Depend on humans to do something? Not a microservice. Textbook definitions and reality rarely coincide, rather than taking such a fundamentalist approach that leads nowhere, recognize that for all intents and purposes, what I described is a microservice, not a distributed monolith.
  - AndrewKemendo 10mo ago
    And if my grandmother had wheels she would be a bike There are categories and ontologies are real in the world. If you create one thing and call it something else that doesn’t mean the definition of “something else” should change By your definition it is impossible to create a state based on coherent specifications because most states don’t align to the specification. We know for a fact that’s wrong via functional programming, state machines, and formal verification
  - ollysb 10mo ago
    It's fine to have dependencies, the point is two services that need to be deployed at the same time are not independent microservices.

mjr00 10mo ago

> It's likely there's a single source of truth for where you pull libraries or shared resources from, when team A wants to update the pointer to library-latest to 2.0 but the current reference of library-latest is still 1.0, everyone needs to migrate off of it otherwise things will break due to backwards compatibility or whatever. No, this misses one of the biggest benefits of services; you explicitly don't need everyone to upgrade library-latest to 2.0 at the same time. If you do find yourself in a situation where you can't upgrade a core library like e.g. SQLAlchemy or Spring, or the underlying Python/Java/Go/etc runtime, without requiring updates to every service, you are back in the realm of a distributed monolith.

wowohwow 10mo ago
I disagree. Both can be true at the same time. A good design should not point to library-latest in a production setting, it should point to a stable known good version via direct reference, i.e library-1.0.0-stable. However, the world we live in, people choose pointing to latest, to avoid manual work and trust other teams did the right diligence when updating to the latest version. You can point to a stable version in the model I described and still be distributed and a micro service, while depending on a shared service or repository.
- vlovich123 10mo ago
  You can do that but you keep missing that you’re no longer a true microservice as originally defined and envisioned, which is that you can deploy the service independently under local control. Can you imagine if Google could only release a new API if all their customers simultaneously updated to that new API? You need loose coupling between services. OP is correct that you are indeed now in a weird hybrid monolith application where it’s deployed piecemeal but can’t really be deployed that way because of tightly coupled dependencies. Be ready for a blog post in ten years how they broke apart the monolith into loosely coupled components because it was too difficult to ship things with a large team and actually have it land in production without getting reverted to an unrelated issue.
  - dmoy 10mo ago
    > Can you imagine if Google could only release a new API if all their customers simultaneously updated to that new API? You need loose coupling between services. Internal Google services: *sweating profusely* (Mostly in jest, it's obviously a different ballgame internal to the monorepo on borg)
  - GeneralMayhem 10mo ago
    Internal and external have wildly different requirements. Google internally can't update a library unless the update is either backward-compatible for all current users or part of the same change that updates all those users, and that's enforced by the build/test harness. That was an explicit choice, and I think an excellent one, for that scenario: it's more important to be certain that you're done when you move forward, so that it's obvious when a feature no longer needs support, than it is to enable moving faster in "isolation" when you all work for the same company anyway. But also, you're conflating code and services. There's a huge difference between libraries that are deployed as part of various binaries and those that are used as remote APIs. If you want to update a utility library that's used by importing code, then you don't need simultaneous deployment, but you would like to update everywhere to get it done with - that's only really possible with a monorepo. If you want to update a remote API without downtime, then you need a multi-phase rollout where you introduce a backward-compatibility mode... but that's true whether you store the code in one place or two.
mekoka 10mo ago
You're both right, but talking past each other. You're right that shared dependencies create a problem, but it can be the problem without semantically redefining the services themselves as a shared monolith. Imagine someone came to you with a similar problem and you concluded "distributed monolith", which may lead them to believe that their services should be merged into a single monolith. What if they then told you that it's going to be tough because these were truly separate apps, but that used the same OS wide Python install, one ran on Django/Postgres, another on Flask/SQLite, and another was on Fastapi/Mongo, but they all relied on some of the same underlying libs that are frequently updated. The more accurate finger should point to bad dependency management and you'd tell them about virtualenv or docker.
deaddodo 10mo ago
The dependencies they're likely referring to aren't core libraries, they're shared interfaces. If you're using protobufs, for instance, and you share the interfaces in a repo. Updating Service A's interface(s) necessitates all services dependent on communicating with it to be updated as well (whether you utilize those changes or not). Generally for larger systems, but smaller/scrappier teams, a true dependency management tree for something like this is out of scope so they just redeploy everything in a domain.
- mjr00 10mo ago
  > If you're using protobufs, for instance, and you share the interfaces in a repo. Updating Service A's interface(s) necessitates all services dependent on communicating with it to be updated as well (whether you utilize those changes or not). This is not true! This is one of the core strengths of protobuf. Non-destructive protobuf changes, such as adding new API methods or new fields, do not require clients to update. On the server-side you do need to handle the case when clients don't send you the new data--plus deal with the annoying "was this int64 actually set to 0 or is it just using the default?" problem--but as a whole you can absolutely independently update a protobuf, implement it on the server, and existing clients can keep on calling and be totally fine. Now, that doesn't mean you can go crazy, as doing things like deleting fields, changing field numbering or renaming APIs will break clients, but this is just the reality of building distributed systems.
  - deaddodo 10mo ago
    I was oversimplifying for the sake of example, but yes you are correct. Properly managed protobufs don't require an update on strict interface expansion; so shouldn't always require a redeploy.
  - necovek 10mo ago
    What you are talking about is simply keeping the API (whether a library or a service) backwards-compatible. There are plenty strategies to achieve that, and it can be done with almost any interface layer (HTTP, protobuf, JSON, SQL, ...).
- 10mo ago
rbranson 10mo ago
This is explicitly called out in the blog post in the trade-offs section. I was one of the engineers who helped make the decisions around this migration. There is no one size fits all. We believed in that thinking originally, but after observing how things played out, decided to make different trade-offs.
- mjr00 10mo ago
  > There is no one size fits all. Totally agree. For what it's worth, based on the limited information in the article, I actually do think it was the right decision to pull all of the per-destination services back into one. The shared library problem can go both ways, after all: maybe the solution is to remove the library so your microservices are fully independent, or maybe they really should have never been independent in the first place and the solution is to put them back together. I don't think either extreme of "every line of code in the company is deployed as one service" or "every function is an independent FaaS" really works in practice, it's all about finding the right balance, which is domain-specific every time.
- wowohwow 10mo ago
  FWIW, I think it was a great write up. It's clear to me what the rationale was and had good justification. Based on the people responding to all of my comments, it is clear people didn't actually read it and are opining without appropriate context.
- nine_k 10mo ago
  To me it sounds like so: "We realized that we were not running microservice architecture, but rather a distributed monolith, so it made sense to make it a regular monolith". It's a decision I would wholeheartedly agree with.
  - necovek 10mo ago
    I don't think you read the post carefully enough: they were not running a distributed monolith, and every service was using different dependencies (versions of them). This meant that it was costly to maintain and caused a lot of confusion, especially with internal dependencies (shared libraries): this is the trade-off they did not like and wanted to move away from. They moved away from this in multiple steps, first one of those being making it a "distributed monolith" (as per your implied definition) by putting services in a monorepo and then making them use the same dependency versions (before finally making them a single service too).
philwelch 10mo ago
> If you do find yourself in a situation where you can't upgrade a core library like e.g. SQLAlchemy or Spring, or the underlying Python/Java/Go/etc runtime, without requiring updates to every service, you are back in the realm of a distributed monolith. Show me a language runtime or core library that will never have a CVE. Otherwise, by your definition, microservices don’t exist and all service oriented architectures are distributed monoliths.
lelanthran 10mo ago
Right, but theres a cost to having to support 12 different versions of a library in your system. Its a tradeoff
- imtringued 10mo ago
  No, that sounds like you're breaking backwards compatibility too often. Assuming there is an update every week, you can expect all teams to update within two weeks, which means if everything goes well only two versions are active.
  - necovek 10mo ago
    That's not how things play out in practice. Say you do backwards incompatible changes "infrequently", say every 4 months, or 3 times per year. In 5 years, that's 15 versions with backwards incompatible changes. Everybody is time pressured at least sometimes, and you miss one update, and now you've got multiple backwards-incompatible updates and you need to do it carefully the next time around, meaning more time needed, meaning it needs to be scheduled and planned along with other feature work now. And then you end up with something close to a normal distribution of versions across ~140 services: some have the very old versions (v1-v4), majority are on some middle, not ancient versions but still ~7 versions behind on average (v5-v10), and only some are on the latest few versions (v11-v15). "Patch" versions can become even crazier, and yes, there will be bugs in them making them inadvertently not backwards compatible either (as not everybody updated right away to detect it). But really, I always point out to good, long lived APIs that make compromises in their API for the sake of backwards compatibility (eg. we still live with "Referer" instead of "Referrer" in HTTP, 35 years later; and it is OK!).
  - lelanthran 10mo ago
    > Assuming there is an update every week, you can expect all teams to update within two weeks Hah; I wish :-) https://www.youtube.com/watch?v=FopyRHHlt3M https://www.youtube.com/watch?v=FopyRHHlt3M
aezart 10mo ago
We had this problem, 119 services that all got their dependencies from a shared domain. Individual services had to depend on the exact version of libraries provided by the domain. It made updates essentially impossible. Eventually we were forced by a licensing change to move to containers, which fixed that issue but substantially increased our research usage, we went from 16 GB of RAM for the entire domain to 1.5 GB per service, with similar increases on CPU.

3rodents 10mo ago

Yes, you’re describing a distributed monolith. Microservices are independent, with nothing shared. They define a public interface and that’s it, that’s the entire exposed surface area. You will need to do major version bumps sometimes, when there are backwards incompatible changes to make, but these are rare. The logical problem you’re running into is exactly why microservices are such a bad idea for most businesses. How many businesses can have entirely independent system components? Almost all “microservice” systems in production are distributed monoliths. Real microservices are incredibly rare. A mental model for true microservices is something akin to depending on the APIs of Netflix, Hulu, HBO Max and YouTube. They’ll have their own data models, their own versioning cycles and all that you consume is the public interface.

wowohwow 10mo ago
This type of elitist mentality is such a problem and such a drain for software development. "Real micro services are incredibly rare". I'll repeat myself from my other post, by this level of logic nothing is a micro service. Do you depend on a cloud provider? Not a microservice. Do you depend on an ISP for Internet? Not a microservice. Depend on humans to do something? Not a microservice. Textbook definitions and reality rarely coincide, rather than taking such a fundamentalist approach that leads nowhere, recognize that for all intents and purposes, what I described is a microservice, not a distributed monolith.
- AndrewKemendo 10mo ago
  And if my grandmother had wheels she would be a bike There are categories and ontologies are real in the world. If you create one thing and call it something else that doesn’t mean the definition of “something else” should change By your definition it is impossible to create a state based on coherent specifications because most states don’t align to the specification. We know for a fact that’s wrong via functional programming, state machines, and formal verification
- ollysb 10mo ago
  It's fine to have dependencies, the point is two services that need to be deployed at the same time are not independent microservices.