10 ms·

> Once the code for all destinations lived in a single repo, they could be merged into a single service. With every destination living in one service, our devel

by mjr00 10mo ago

> Once the code for all destinations lived in a single repo, they could be merged into a single service. With every destination living in one service, our developer productivity substantially improved. We no longer had to deploy 140+ services for a change to one of the shared libraries. One engineer can deploy the service in a matter of minutes. If you must to deploy every service because of a library change, you don't have services, you have a distributed monolith. The entire idea of a "shared library" which must be kept updated across your entire service fleet is antithetical to how you need to treat services.

wowohwow 10mo ago
I think your point while valid, it is probably a lot more nuanced. From the post it's more akin to an Amazon shared build and deployment system than "every library update needs to redeploy every time scenario". It's likely there's a single source of truth where you pull libraries or shared resources from, when team A wants to update the pointer to library-latest to 2.0 but the current reference of library-latest is still 1.0, everyone needs to migrate off of it otherwise things will break due to backwards compatibility or whatever. Likewise, if there's a -need- to remove a version for a vulnerability or what have you, then everyone needs to redeploy, sure, but the centralized benefit of this likely outweighs the security cost and complexity of tracking the patching and deployment process for each and every service. I would say those systems -are- and likely would be classified as micro services but from a cost and ease perspective operate within a shared services environment. I don't think it's fair to consider this style of design decision as a distributed monolith. By that level of logic, having a singular business entity vs 140 individual business entities for each service would mean it's a distributed monolith.
- mjr00 10mo ago
  > It's likely there's a single source of truth for where you pull libraries or shared resources from, when team A wants to update the pointer to library-latest to 2.0 but the current reference of library-latest is still 1.0, everyone needs to migrate off of it otherwise things will break due to backwards compatibility or whatever. No, this misses one of the biggest benefits of services; you explicitly don't need everyone to upgrade library-latest to 2.0 at the same time. If you do find yourself in a situation where you can't upgrade a core library like e.g. SQLAlchemy or Spring, or the underlying Python/Java/Go/etc runtime, without requiring updates to every service, you are back in the realm of a distributed monolith.
  - wowohwow 10mo ago
    I disagree. Both can be true at the same time. A good design should not point to library-latest in a production setting, it should point to a stable known good version via direct reference, i.e library-1.0.0-stable. However, the world we live in, people choose pointing to latest, to avoid manual work and trust other teams did the right diligence when updating to the latest version. You can point to a stable version in the model I described and still be distributed and a micro service, while depending on a shared service or repository.
    - vlovich123 10mo ago
      You can do that but you keep missing that you’re no longer a true microservice as originally defined and envisioned, which is that you can deploy the service independently under local control. Can you imagine if Google could only release a new API if all their customers simultaneously updated to that new API? You need loose coupling between services. OP is correct that you are indeed now in a weird hybrid monolith application where it’s deployed piecemeal but can’t really be deployed that way because of tightly coupled dependencies. Be ready for a blog post in ten years how they broke apart the monolith into loosely coupled components because it was too difficult to ship things with a large team and actually have it land in production without getting reverted to an unrelated issue.
      - dmoy 10mo ago
        > Can you imagine if Google could only release a new API if all their customers simultaneously updated to that new API? You need loose coupling between services. Internal Google services: *sweating profusely* (Mostly in jest, it's obviously a different ballgame internal to the monorepo on borg)
      - GeneralMayhem 10mo ago
        Internal and external have wildly different requirements. Google internally can't update a library unless the update is either backward-compatible for all current users or part of the same change that updates all those users, and that's enforced by the build/test harness. That was an explicit choice, and I think an excellent one, for that scenario: it's more important to be certain that you're done when you move forward, so that it's obvious when a feature no longer needs support, than it is to enable moving faster in "isolation" when you all work for the same company anyway. But also, you're conflating code and services. There's a huge difference between libraries that are deployed as part of various binaries and those that are used as remote APIs. If you want to update a utility library that's used by importing code, then you don't need simultaneous deployment, but you would like to update everywhere to get it done with - that's only really possible with a monorepo. If you want to update a remote API without downtime, then you need a multi-phase rollout where you introduce a backward-compatibility mode... but that's true whether you store the code in one place or two.
- 3rodents 10mo ago
  Yes, you’re describing a distributed monolith. Microservices are independent, with nothing shared. They define a public interface and that’s it, that’s the entire exposed surface area. You will need to do major version bumps sometimes, when there are backwards incompatible changes to make, but these are rare. The logical problem you’re running into is exactly why microservices are such a bad idea for most businesses. How many businesses can have entirely independent system components? Almost all “microservice” systems in production are distributed monoliths. Real microservices are incredibly rare. A mental model for true microservices is something akin to depending on the APIs of Netflix, Hulu, HBO Max and YouTube. They’ll have their own data models, their own versioning cycles and all that you consume is the public interface.
  - wowohwow 10mo ago
    This type of elitist mentality is such a problem and such a drain for software development. "Real micro services are incredibly rare". I'll repeat myself from my other post, by this level of logic nothing is a micro service. Do you depend on a cloud provider? Not a microservice. Do you depend on an ISP for Internet? Not a microservice. Depend on humans to do something? Not a microservice. Textbook definitions and reality rarely coincide, rather than taking such a fundamentalist approach that leads nowhere, recognize that for all intents and purposes, what I described is a microservice, not a distributed monolith.
    - AndrewKemendo 10mo ago
      And if my grandmother had wheels she would be a bike There are categories and ontologies are real in the world. If you create one thing and call it something else that doesn’t mean the definition of “something else” should change By your definition it is impossible to create a state based on coherent specifications because most states don’t align to the specification. We know for a fact that’s wrong via functional programming, state machines, and formal verification
    - ollysb 10mo ago
      It's fine to have dependencies, the point is two services that need to be deployed at the same time are not independent microservices.
andrewmutz 10mo ago
Needing to upgrade a library everywhere isn’t necessarily a sign of inappropriate coupling. For example, a library with a security vulnerability would need to be upgraded everywhere regardless of how well you’ve designed your system. In that example the monolith is much easier to work with.
- mettamage 10mo ago
  Example: log4j. That was an update fiasco everywhere.
  - smrtinsert 10mo ago
    1 line change and redeploy
    - jabroni_salad 10mo ago
      Works great if you are the product owner. We ended up having to fire and replace about a dozen 3rd party vendors over this.
- mjr00 10mo ago
  While you're right, I can only think of twice in my career where there was a "code red all services must update now", which were log4shell and spectre/meltdown (which were a bit different anyway). I just don't think this comes up enough in practice to be worth optimizing for.
  - wowohwow 10mo ago
    You have not been in the field very long than I presume? There's multiple per year that require all hands on deck depending on your tech stack. Just look at the recent NPM supply chain attacks.
    - mjr00 10mo ago
      You presume very incorrectly to say the least. The npm supply chain attacks were only an issue if you don't use lock files. In fact they were a great example of why you shouldn't blindly upgrade to the latest packages when they are available.
j45 10mo ago
Monorepos reasonably well designed and fleixble to grow with you can increase development speed quite a bit.
deleted 10mo ago
[deleted]
reactordev 10mo ago
I was coming here to say this. That the whole idea of a shared library couples all those services together. Sounds like someone wanted to be clever and then included their cleverness all over the platform. Dooming all services together. Decoupling is the first part of microservices. Pass messages. Use json. I shouldn’t need your code to function. Just your API. Then you can be clever and scale out and deploy on saturdays if you want to and it doesn’t disturb the rest of us.
- xienze 10mo ago
  > Pass messages. Use json. I shouldn’t need your code to function. Just your API. Yes, but there’s likely a lot of common code related to parsing those messages, interpreting them, calling out to other services etc. shared amongst all of them. That’s to be expected. The question is how that common code is structured if everything has to get updated at once if the common code changes.
  - c-fe 10mo ago
    one way is by using schemas to communicate between them that are backwards compatible. eg with avro its quite nice
    - gmueckl 10mo ago
      But the you're outsourcing the same shared code problem to a third party shared library. It fundamentally doesn't go away.
      - reactordev 10mo ago
        That 3rd party library rarely gets updated whereas Jon’s commit adds a field and now everyone has to update or the marshaling doesn’t work. Yes, there are scenarios where you have to deploy everything but when dealing with micro services, you should only be deploying the service you are changing. If updating a field in a domain affects everyone else, you have a distributed monolith and your architecture is questionable at best. The whole point is I can deploy my services without relying on yours, or touching yours, because it sounds like you might not know what you’re doing. That’s the beautiful effect of a good micro service architecture.
smrtinsert 10mo ago
100%. It's almost like they jumped into it not understanding what they were signing up for.
threethirtytwo 10mo ago
Then every microservice network in existence is a distributed monolith so long as they communicate with one another. If you communicate with one another you are serializing and deserializing a shared type. That shared type will break at the communication channels if you do not simultaneously deploy the two services. The irony is to prevent this you have to deploy simultaneously and treat it as a distributed monolith. This is the fundamental problem of micro services. Under a monorepo it is somewhat more mitigated because now you can have type checking and integration tests across multiple repos. Make no mistake the world isn’t just library dependencies. There are communication dependencies that flow through communication channels. A microservice architecture by definition has all its services depend on each other through this communication channels. The logical outcome of this is virtually identical to a distributed monolith. In fact shared libraries don’t do much damage at all if the versions are off. It is only shared types in the communication channels that break. There is no way around this unless you have a mechanism for simultaneous merging code and deploying code across different repos which breaks the definition of what it is to be a microservice. Microservices always and I mean always share dependencies with everything they communicate with. All the problems that come from shared libraries are intrinsic to microservices EVEN when you remove shared libraries. People debate me on this but it’s an invariant.
- wowohwow 10mo ago
  Bingo. Couldn't agree more. The other posters in this comment chain seem to view things from a dogmatic approach vs a pragmatic approach. It's important to do both, but individuals should call out when they are discussing something that is practiced vs preached.
  - threethirtytwo 10mo ago
    Agreed. What I’m describing here isn’t solely pragmatic it’s axiomatic as well. If you model this as a distributed system with graph all microservices by definition will always reach a state where the apis are broken. Most microservice companies either live with the fact or they have round about ways to deal with it including simultaneous deploys across multiple services and simultaneous merging, CI and type checking across different repos.
mlhpdx 10mo ago
Agreed. It sounds like they never made it to the distributed architecture they would have benefited from. That said, if the team thrives on a monolithic one they made the right choice.
Aeolun 10mo ago
Once all the code for the services lived in one repo there was nothing preventing them from deploying the thing 140 times. I’m not sure why they act like that wasn’t an option.
echelon 10mo ago
> If you must to deploy every service because of a library change Hello engineer. Jira ticket VULN-XXX had been assigned to you as your team's on call engineer. A critical vulnerability has been found in the netxyz library. Please deploy service $foo after SHA before 2025-12-14 at 12:00 UTC. Hello engineer. Jira ticket VULN-XXX had been assigned to you as your team's on call engineer. A critical vulnerability has been found in the netxyz library. Please deploy service $bar after SHA before 2025-12-14 at 12:00 UTC. ... It's never ending. You get a half dozen of these on each on call rotation.
- imtringued 10mo ago
  I don't know what a call rotation is, but I keep getting email flooded by half a dozen Linux vulnerabilities every day and it's getting old.
- sethammons 10mo ago
  My experience doesn't align with yours. I worked at SendGrid for over a decade and they were on the (micro) service train. I was on call for all dev teams on a rotation for a couple of years and later just for my team. I have seen like a dozen security updates like you describe.
  - echelon 10mo ago
    This was at a fintech and we took every single little vuln with the utmost priority. Triaged by severity of course, but everything had a ticking clock. We didn't just have multiple security teams, we had multiple security orgs. If you didn't stay in compliance with VULN SLAs, you'd get a talking to. We also had to frequently roll secrets. If the secrets didn't support auto-rotation, that was also a deployment (with other steps). We also had to deploy our apps if they were stale. It's dangerous not to deploy your app every month or two, because who knows if stale builds introduced some kind of brittleness? Perhaps a change to some net library you didn't deploy caused the app not to tolerate traffic spikes. And it's been six months and there are several such library changes.
ChuckMcM 10mo ago
While I think that's a bit harsh :-) the sentiment of "if you have these problems, perhaps you don't understand systems architecture" is kind of spot on. I have heard people scoff at a bunch of "dead legacy code" in the Windows APIs (as an example) without understanding the challenge of moving millions of machines, each at different places in the evolution timeline, through to the next step in the timeline. To use an example from the article, there was this statement: "The split to separate repos allowed us to isolate the destination test suites easily. This isolation allowed the development team to move quickly when maintaining destinations." This is architecture bleed through. The format produced by Twilio "should" be the canonical form, which is submitted the adapter which mangles it to the "destination" form. Great, that transformation is expressible semantically in a language that takes the canonical form and spits out the special form. Changes to the transformation expression should not "bleed through" to other destinations, and changes to the canonical form should be backwards compatible to prevent bleed through of changes in the source from impacting the destination. At all times, if something worked before, it should continue to work without touching it because the architecture boundaries are robust. Being able to work with a team that understood this was common "in the old days" when people were working on an operating system. The operating system would evolve (new features, new devices, new capabilities) but because there was a moat between the OS and applications, people understood that they had to architect things so that the OS changes would not cause applications that currently worked to stop working. I don't judge Twilio for not doing robust architecture, I was astonished when I went to work of Google how lazy everyone got when the entire system is under their control (like there are no third party apps running in the fleet). The was a persistent theme of some bright person "deciding" to completely change some interface and Wham! every other group at Google had to stop what they were doing and move their code to the new thing. There was a particularly poor 'mandate' on a new version of their RPC while I was there. As Twilio notes, that can make things untenable.
__abc 10mo ago
So you should re-write your logging code on each and every one of your 140+ services vs. leverage a shared module?
- xboxnolifes 10mo ago
  You can keep using an older version for a while. You shouldn't need to redeploy everything at once. If you can't keep using the older version, you did it wrong. And ideally, your logging library should rarely need to update. If you need unique integrations per service, use a plug-in architecture and keep the plug-ins local to each service.
  - __abc 10mo ago
    I wasn't taking into account velocity of fleet-wide rollout, as I agree, you can migrate over time however. however, I was focusing on the idea that anytime of fleet wide rollout for a specific change was somehow "bad."
philwelch 10mo ago
If there’s any shared library across all your services, even a third party library, if that library has a security patch you now need to update that shared library across your entire service fleet. Maybe you don’t have that; maybe each service is written in a completely different programming language, uses a different database, and reimplements monitoring in a totally different way. In that case you have completely different problems.
- sethammons 10mo ago
  Everyone needing to update due to a security thing happens infrequently. Otherwise, coding and deploying may be happening multiple times a day. We have had shared libraries. Teams updated to them when they next wanted to. When it was important, on call people made it happen asap. Zero issues.
necovek 10mo ago
Imagine your services were built on react-server-* components or used Log4J logging. This is simply dependency hell exploding with microservices.
andy_ppp 10mo ago
It’s easy to say things like this but also incredibly difficult to know if you’ll introduce subtle bugs or incompatibilities between services. It’s an example of people following the microservices pattern and then being given additional risk or problems deploying that are not immediately obvious when buying into this! So let’s say you have a shared money library that you have fixed a bug in… what would you do in the real world - redeploy all your services that use said library or something else?
- mjr00 10mo ago
  > It’s easy to say things like this but also incredibly difficult to know if you’ll introduce subtle bugs or incompatibilities between services. You are right: it is difficult. It is harder than building a monolith. No argument there. I just don't think proper microservices are as difficult as people think. It's just more of a mindshift. Plenty of projects and companies continue to release backwards compatible APIs: operating systems, Stripe/PayPal, cloud providers. Bugs come up, but in general people don't worry about ec2:DescribeInstances randomly breaking. These projects are still evolving internally while maintaining a stable external API. It's a skill, but something that can be learned. > So let’s say you have a shared money library that you have fixed a bug in… what would you do in the real world - redeploy all your services that use said library or something else? In the real world I would not have a shared "money library" to begin with. If there were money-related operations that needed to be used by multiple services, I would have a "money service" which exposed an API and could be deployed independently. A bug fix would then be a deploy to this service, and no other services would have to update or be aware of the fix. This isn't a theoretical, either, as a "payments service" that encapsulates access to payment processors is something I've commonly seen.
  - necovek 10mo ago
    But really, a shared "money library" is exactly the same thing as a shared "money service" if everyone is using the same, latest version (which is easier to enforce with a networked "service"). The difference is in what's easy and what's hard. With a library, it's easy for everyone to run a different version, and hard for everyone to run the same version. With a service, it's easy for everyone to use the same version, and harder to use a different one (eg. creating multiple environments, and especially ephemeral "pull request" environments where you can mix and match for best automated integration and e2e testing). But you can apply the same backwards-compatible API design patterns to a library that you would be applying to a service: no difference really. It's only about what's the time to detection when you break these patterns (with a library, someone finds out 2 years later when they update; with a service, they learn right away).
duxup 10mo ago
Yeah this seems very much not a microservices setup. I don't pretend proper microservices are a magic solution... but if you break the rules / system if microservices, that's not "microsercices" being bad, that's just creating problems for yourself.

by mjr00 10mo ago

wowohwow 10mo ago

I think your point while valid, it is probably a lot more nuanced. From the post it's more akin to an Amazon shared build and deployment system than "every library update needs to redeploy every time scenario". It's likely there's a single source of truth where you pull libraries or shared resources from, when team A wants to update the pointer to library-latest to 2.0 but the current reference of library-latest is still 1.0, everyone needs to migrate off of it otherwise things will break due to backwards compatibility or whatever. Likewise, if there's a -need- to remove a version for a vulnerability or what have you, then everyone needs to redeploy, sure, but the centralized benefit of this likely outweighs the security cost and complexity of tracking the patching and deployment process for each and every service. I would say those systems -are- and likely would be classified as micro services but from a cost and ease perspective operate within a shared services environment. I don't think it's fair to consider this style of design decision as a distributed monolith. By that level of logic, having a singular business entity vs 140 individual business entities for each service would mean it's a distributed monolith.

mjr00 10mo ago
> It's likely there's a single source of truth for where you pull libraries or shared resources from, when team A wants to update the pointer to library-latest to 2.0 but the current reference of library-latest is still 1.0, everyone needs to migrate off of it otherwise things will break due to backwards compatibility or whatever. No, this misses one of the biggest benefits of services; you explicitly don't need everyone to upgrade library-latest to 2.0 at the same time. If you do find yourself in a situation where you can't upgrade a core library like e.g. SQLAlchemy or Spring, or the underlying Python/Java/Go/etc runtime, without requiring updates to every service, you are back in the realm of a distributed monolith.
- wowohwow 10mo ago
  I disagree. Both can be true at the same time. A good design should not point to library-latest in a production setting, it should point to a stable known good version via direct reference, i.e library-1.0.0-stable. However, the world we live in, people choose pointing to latest, to avoid manual work and trust other teams did the right diligence when updating to the latest version. You can point to a stable version in the model I described and still be distributed and a micro service, while depending on a shared service or repository.
  - vlovich123 10mo ago
    You can do that but you keep missing that you’re no longer a true microservice as originally defined and envisioned, which is that you can deploy the service independently under local control. Can you imagine if Google could only release a new API if all their customers simultaneously updated to that new API? You need loose coupling between services. OP is correct that you are indeed now in a weird hybrid monolith application where it’s deployed piecemeal but can’t really be deployed that way because of tightly coupled dependencies. Be ready for a blog post in ten years how they broke apart the monolith into loosely coupled components because it was too difficult to ship things with a large team and actually have it land in production without getting reverted to an unrelated issue.
    - dmoy 10mo ago
      > Can you imagine if Google could only release a new API if all their customers simultaneously updated to that new API? You need loose coupling between services. Internal Google services: *sweating profusely* (Mostly in jest, it's obviously a different ballgame internal to the monorepo on borg)
    - GeneralMayhem 10mo ago
      Internal and external have wildly different requirements. Google internally can't update a library unless the update is either backward-compatible for all current users or part of the same change that updates all those users, and that's enforced by the build/test harness. That was an explicit choice, and I think an excellent one, for that scenario: it's more important to be certain that you're done when you move forward, so that it's obvious when a feature no longer needs support, than it is to enable moving faster in "isolation" when you all work for the same company anyway. But also, you're conflating code and services. There's a huge difference between libraries that are deployed as part of various binaries and those that are used as remote APIs. If you want to update a utility library that's used by importing code, then you don't need simultaneous deployment, but you would like to update everywhere to get it done with - that's only really possible with a monorepo. If you want to update a remote API without downtime, then you need a multi-phase rollout where you introduce a backward-compatibility mode... but that's true whether you store the code in one place or two.
3rodents 10mo ago
Yes, you’re describing a distributed monolith. Microservices are independent, with nothing shared. They define a public interface and that’s it, that’s the entire exposed surface area. You will need to do major version bumps sometimes, when there are backwards incompatible changes to make, but these are rare. The logical problem you’re running into is exactly why microservices are such a bad idea for most businesses. How many businesses can have entirely independent system components? Almost all “microservice” systems in production are distributed monoliths. Real microservices are incredibly rare. A mental model for true microservices is something akin to depending on the APIs of Netflix, Hulu, HBO Max and YouTube. They’ll have their own data models, their own versioning cycles and all that you consume is the public interface.
- wowohwow 10mo ago
  This type of elitist mentality is such a problem and such a drain for software development. "Real micro services are incredibly rare". I'll repeat myself from my other post, by this level of logic nothing is a micro service. Do you depend on a cloud provider? Not a microservice. Do you depend on an ISP for Internet? Not a microservice. Depend on humans to do something? Not a microservice. Textbook definitions and reality rarely coincide, rather than taking such a fundamentalist approach that leads nowhere, recognize that for all intents and purposes, what I described is a microservice, not a distributed monolith.
  - AndrewKemendo 10mo ago
    And if my grandmother had wheels she would be a bike There are categories and ontologies are real in the world. If you create one thing and call it something else that doesn’t mean the definition of “something else” should change By your definition it is impossible to create a state based on coherent specifications because most states don’t align to the specification. We know for a fact that’s wrong via functional programming, state machines, and formal verification
  - ollysb 10mo ago
    It's fine to have dependencies, the point is two services that need to be deployed at the same time are not independent microservices.

andrewmutz 10mo ago

Needing to upgrade a library everywhere isn’t necessarily a sign of inappropriate coupling. For example, a library with a security vulnerability would need to be upgraded everywhere regardless of how well you’ve designed your system. In that example the monolith is much easier to work with.

mettamage 10mo ago
Example: log4j. That was an update fiasco everywhere.
- smrtinsert 10mo ago
  1 line change and redeploy
  - jabroni_salad 10mo ago
    Works great if you are the product owner. We ended up having to fire and replace about a dozen 3rd party vendors over this.
mjr00 10mo ago
While you're right, I can only think of twice in my career where there was a "code red all services must update now", which were log4shell and spectre/meltdown (which were a bit different anyway). I just don't think this comes up enough in practice to be worth optimizing for.
- wowohwow 10mo ago
  You have not been in the field very long than I presume? There's multiple per year that require all hands on deck depending on your tech stack. Just look at the recent NPM supply chain attacks.
  - mjr00 10mo ago
    You presume very incorrectly to say the least. The npm supply chain attacks were only an issue if you don't use lock files. In fact they were a great example of why you shouldn't blindly upgrade to the latest packages when they are available.

j45 10mo ago

Monorepos reasonably well designed and fleixble to grow with you can increase development speed quite a bit.

deleted 10mo ago

[deleted]

reactordev 10mo ago

I was coming here to say this. That the whole idea of a shared library couples all those services together. Sounds like someone wanted to be clever and then included their cleverness all over the platform. Dooming all services together. Decoupling is the first part of microservices. Pass messages. Use json. I shouldn’t need your code to function. Just your API. Then you can be clever and scale out and deploy on saturdays if you want to and it doesn’t disturb the rest of us.

xienze 10mo ago
> Pass messages. Use json. I shouldn’t need your code to function. Just your API. Yes, but there’s likely a lot of common code related to parsing those messages, interpreting them, calling out to other services etc. shared amongst all of them. That’s to be expected. The question is how that common code is structured if everything has to get updated at once if the common code changes.
- c-fe 10mo ago
  one way is by using schemas to communicate between them that are backwards compatible. eg with avro its quite nice
  - gmueckl 10mo ago
    But the you're outsourcing the same shared code problem to a third party shared library. It fundamentally doesn't go away.
    - reactordev 10mo ago
      That 3rd party library rarely gets updated whereas Jon’s commit adds a field and now everyone has to update or the marshaling doesn’t work. Yes, there are scenarios where you have to deploy everything but when dealing with micro services, you should only be deploying the service you are changing. If updating a field in a domain affects everyone else, you have a distributed monolith and your architecture is questionable at best. The whole point is I can deploy my services without relying on yours, or touching yours, because it sounds like you might not know what you’re doing. That’s the beautiful effect of a good micro service architecture.

smrtinsert 10mo ago

100%. It's almost like they jumped into it not understanding what they were signing up for.

threethirtytwo 10mo ago

Then every microservice network in existence is a distributed monolith so long as they communicate with one another. If you communicate with one another you are serializing and deserializing a shared type. That shared type will break at the communication channels if you do not simultaneously deploy the two services. The irony is to prevent this you have to deploy simultaneously and treat it as a distributed monolith. This is the fundamental problem of micro services. Under a monorepo it is somewhat more mitigated because now you can have type checking and integration tests across multiple repos. Make no mistake the world isn’t just library dependencies. There are communication dependencies that flow through communication channels. A microservice architecture by definition has all its services depend on each other through this communication channels. The logical outcome of this is virtually identical to a distributed monolith. In fact shared libraries don’t do much damage at all if the versions are off. It is only shared types in the communication channels that break. There is no way around this unless you have a mechanism for simultaneous merging code and deploying code across different repos which breaks the definition of what it is to be a microservice. Microservices always and I mean always share dependencies with everything they communicate with. All the problems that come from shared libraries are intrinsic to microservices EVEN when you remove shared libraries. People debate me on this but it’s an invariant.

wowohwow 10mo ago
Bingo. Couldn't agree more. The other posters in this comment chain seem to view things from a dogmatic approach vs a pragmatic approach. It's important to do both, but individuals should call out when they are discussing something that is practiced vs preached.
- threethirtytwo 10mo ago
  Agreed. What I’m describing here isn’t solely pragmatic it’s axiomatic as well. If you model this as a distributed system with graph all microservices by definition will always reach a state where the apis are broken. Most microservice companies either live with the fact or they have round about ways to deal with it including simultaneous deploys across multiple services and simultaneous merging, CI and type checking across different repos.

mlhpdx 10mo ago

Agreed. It sounds like they never made it to the distributed architecture they would have benefited from. That said, if the team thrives on a monolithic one they made the right choice.

Aeolun 10mo ago

Once all the code for the services lived in one repo there was nothing preventing them from deploying the thing 140 times. I’m not sure why they act like that wasn’t an option.

echelon 10mo ago

> If you must to deploy every service because of a library change Hello engineer. Jira ticket VULN-XXX had been assigned to you as your team's on call engineer. A critical vulnerability has been found in the netxyz library. Please deploy service $foo after SHA before 2025-12-14 at 12:00 UTC. Hello engineer. Jira ticket VULN-XXX had been assigned to you as your team's on call engineer. A critical vulnerability has been found in the netxyz library. Please deploy service $bar after SHA before 2025-12-14 at 12:00 UTC. ... It's never ending. You get a half dozen of these on each on call rotation.

imtringued 10mo ago
I don't know what a call rotation is, but I keep getting email flooded by half a dozen Linux vulnerabilities every day and it's getting old.
sethammons 10mo ago
My experience doesn't align with yours. I worked at SendGrid for over a decade and they were on the (micro) service train. I was on call for all dev teams on a rotation for a couple of years and later just for my team. I have seen like a dozen security updates like you describe.
- echelon 10mo ago
  This was at a fintech and we took every single little vuln with the utmost priority. Triaged by severity of course, but everything had a ticking clock. We didn't just have multiple security teams, we had multiple security orgs. If you didn't stay in compliance with VULN SLAs, you'd get a talking to. We also had to frequently roll secrets. If the secrets didn't support auto-rotation, that was also a deployment (with other steps). We also had to deploy our apps if they were stale. It's dangerous not to deploy your app every month or two, because who knows if stale builds introduced some kind of brittleness? Perhaps a change to some net library you didn't deploy caused the app not to tolerate traffic spikes. And it's been six months and there are several such library changes.

ChuckMcM 10mo ago

While I think that's a bit harsh :-) the sentiment of "if you have these problems, perhaps you don't understand systems architecture" is kind of spot on. I have heard people scoff at a bunch of "dead legacy code" in the Windows APIs (as an example) without understanding the challenge of moving millions of machines, each at different places in the evolution timeline, through to the next step in the timeline. To use an example from the article, there was this statement: "The split to separate repos allowed us to isolate the destination test suites easily. This isolation allowed the development team to move quickly when maintaining destinations." This is architecture bleed through. The format produced by Twilio "should" be the canonical form, which is submitted the adapter which mangles it to the "destination" form. Great, that transformation is expressible semantically in a language that takes the canonical form and spits out the special form. Changes to the transformation expression should not "bleed through" to other destinations, and changes to the canonical form should be backwards compatible to prevent bleed through of changes in the source from impacting the destination. At all times, if something worked before, it should continue to work without touching it because the architecture boundaries are robust. Being able to work with a team that understood this was common "in the old days" when people were working on an operating system. The operating system would evolve (new features, new devices, new capabilities) but because there was a moat between the OS and applications, people understood that they had to architect things so that the OS changes would not cause applications that currently worked to stop working. I don't judge Twilio for not doing robust architecture, I was astonished when I went to work of Google how lazy everyone got when the entire system is under their control (like there are no third party apps running in the fleet). The was a persistent theme of some bright person "deciding" to completely change some interface and Wham! every other group at Google had to stop what they were doing and move their code to the new thing. There was a particularly poor 'mandate' on a new version of their RPC while I was there. As Twilio notes, that can make things untenable.

__abc 10mo ago

So you should re-write your logging code on each and every one of your 140+ services vs. leverage a shared module?

xboxnolifes 10mo ago
You can keep using an older version for a while. You shouldn't need to redeploy everything at once. If you can't keep using the older version, you did it wrong. And ideally, your logging library should rarely need to update. If you need unique integrations per service, use a plug-in architecture and keep the plug-ins local to each service.
- __abc 10mo ago
  I wasn't taking into account velocity of fleet-wide rollout, as I agree, you can migrate over time however. however, I was focusing on the idea that anytime of fleet wide rollout for a specific change was somehow "bad."

philwelch 10mo ago

If there’s any shared library across all your services, even a third party library, if that library has a security patch you now need to update that shared library across your entire service fleet. Maybe you don’t have that; maybe each service is written in a completely different programming language, uses a different database, and reimplements monitoring in a totally different way. In that case you have completely different problems.

sethammons 10mo ago
Everyone needing to update due to a security thing happens infrequently. Otherwise, coding and deploying may be happening multiple times a day. We have had shared libraries. Teams updated to them when they next wanted to. When it was important, on call people made it happen asap. Zero issues.

necovek 10mo ago

Imagine your services were built on react-server-* components or used Log4J logging. This is simply dependency hell exploding with microservices.

andy_ppp 10mo ago

It’s easy to say things like this but also incredibly difficult to know if you’ll introduce subtle bugs or incompatibilities between services. It’s an example of people following the microservices pattern and then being given additional risk or problems deploying that are not immediately obvious when buying into this! So let’s say you have a shared money library that you have fixed a bug in… what would you do in the real world - redeploy all your services that use said library or something else?

mjr00 10mo ago
> It’s easy to say things like this but also incredibly difficult to know if you’ll introduce subtle bugs or incompatibilities between services. You are right: it is difficult. It is harder than building a monolith. No argument there. I just don't think proper microservices are as difficult as people think. It's just more of a mindshift. Plenty of projects and companies continue to release backwards compatible APIs: operating systems, Stripe/PayPal, cloud providers. Bugs come up, but in general people don't worry about ec2:DescribeInstances randomly breaking. These projects are still evolving internally while maintaining a stable external API. It's a skill, but something that can be learned. > So let’s say you have a shared money library that you have fixed a bug in… what would you do in the real world - redeploy all your services that use said library or something else? In the real world I would not have a shared "money library" to begin with. If there were money-related operations that needed to be used by multiple services, I would have a "money service" which exposed an API and could be deployed independently. A bug fix would then be a deploy to this service, and no other services would have to update or be aware of the fix. This isn't a theoretical, either, as a "payments service" that encapsulates access to payment processors is something I've commonly seen.
- necovek 10mo ago
  But really, a shared "money library" is exactly the same thing as a shared "money service" if everyone is using the same, latest version (which is easier to enforce with a networked "service"). The difference is in what's easy and what's hard. With a library, it's easy for everyone to run a different version, and hard for everyone to run the same version. With a service, it's easy for everyone to use the same version, and harder to use a different one (eg. creating multiple environments, and especially ephemeral "pull request" environments where you can mix and match for best automated integration and e2e testing). But you can apply the same backwards-compatible API design patterns to a library that you would be applying to a service: no difference really. It's only about what's the time to detection when you break these patterns (with a library, someone finds out 2 years later when they update; with a service, they learn right away).

duxup 10mo ago

Yeah this seems very much not a microservices setup. I don't pretend proper microservices are a magic solution... but if you break the rules / system if microservices, that's not "microsercices" being bad, that's just creating problems for yourself.