15 ms·

How to Improve Your Monolith Before Transitioning to Microservices

235 points by ahamez 4y ago

Akronymus 4y ago
A lot of those steps just seem like good engineering. (I personally prefer modular monoliths over microservices though, in all but very few cases.)
- tleasure 4y ago
  "I personally prefer modular monoliths over microservices though, in all but very few cases." Couldn't agree more. Often times folks are using microservices to achieve good modularity at a cost.
  - paskozdilar 4y ago
    > Often times folks are using microservices to achieve good modularity at a cost. And even more often, folks use microservices, but make them so coupled that you can't really run/test/hack one without running all the others... Basically creating a distributed monolith.
    - deleted 4y ago
      [deleted]
    - delecti 4y ago
      Strong agree. I worked on a "distributed monolith" once, and now I loudly make it a requirement that all my team's microservices can easily be run locally. Running one stack locally required starting up 12 different microservices, all in a particular order, before you could do anything with any of them. Insanity.
  - Akronymus 4y ago
    Kinda reminds me of how you "need to have a horizontally scaling database setup because 1 million rows a day are impossible to handle via a normal database server" people really underestimate the power that vertical scaling can achieve, along with the long tail that microservices can bring. (The more calls between services you need to handle x, the more likely it is that you get into a case where one call takes exceptionally long) https://www.youtube.com/watch?v=SjC9bFjeR1k https://www.youtube.com/watch?v=SjC9bFjeR1k
    - iamflimflam1 4y ago
      I’ve faced this a number of times “we think we’ll have scaling issues” when they are running on the lowest possible database tier on offer. I think people just don’t understand how much power they actually have at their fingertips without needing anything esoteric.
  - paskozdilar 4y ago
    (second comment - after I had some time to think about it) Actually I think microservices serve as a tool for enforcing modularity. When pressure is high, corners are cut, and when unrelated code is easy to reach (as in case of a monolithic codebase), it's easy to fall into temptation - a small dirty hack is faster than refactoring. And when you consider different maintainers, it's easy to lose track of the original monolith architecture idea in all the dirty hacks. Microservices enforce some kind of strict separation, so in theory nobody can do anything that they're not supposed to. In practice, a lot of coupling can happen at the microservices level - the most common symptom being some weird isolated APIs whose only purpose is to do that one thing that another service needs for some reason. Those kind of ad-hoc dependencies tend to make services implementation-specific and non-interchangeable, therefore breaking modularity. So, in conclusion, some approaches are easier to keep modular than others, but there seems to be no silver bullet for replacing care and effort.
- xbar 4y ago
  Right. I feel like it's a better article without any reference to microservices.
- davidkuennen 4y ago
  Having a modular monolith myself I couldn't agree more.
- spaetzleesser 4y ago
  "A lot of those steps just seem like good engineering. " Agreed. I always wonder why people think that their inability to write libraries with good modularization will be solved by introducing microservices.
  - yrgulation 4y ago
    It takes experience and guts to know when to use what and most people just go with the the latest and fanciest. Well tested, focused and self contained libraries are good architecture even when micro-services are a must.
greatpostman 4y ago
People forget, micro services commonly serve massive tech companies. With 100 developers working on the same product, you need it broken up and separated. If you’re in a small company, the value proposition is not as great. It’s contextual, and a tech stack that solves organizational problems, not always technical ones
- adra 4y ago
  Agreed, but I'll go a step further and say microservices are really valuable when you have products that are powered through a reasonable DevOps and CI/CD support within and organization. If you're a company that only releases quarterly, a monolith probably makes sense. If you're a company releasing changes daily / hourly, monoliths make progressively less sense and become progressively harder to work with. When we release software (a lot) our downtime SLO is generally zero minutes. If you're a well out together outfit with strict discipline, this can be achieved with microservices. Inversely, monoliths almost never have to deal with multiple components at different release levels, so they don't do a particularly good job to support it, which is why you often see hours long upgrade windows for monoliths. Shut everything down, deploy updates, start everything back up and hope the house is still working plus the changes.
  - codethief 4y ago
    > If you're a company releasing changes daily / hourly, monoliths make progressively less sense and become progressively harder to work with. Counter argument: Your overall CI/CD infrastructure landscape will be much simpler and muss less error-prone. As will be your release and test pipelines. For instance, if you have a few dozen microservices, each being released every other hour, how do you run E2E tests (across all services) before you release anything? Let's say you take your microservice A in version v1.2.3 (soon to be released) and test it against microservice B's current prod version v2.0.1. Meanwhile, team B is also working on releasing the new version v2.1.0 of microservice B and testing it against A's current prod version v1.2.2. Both test runs work fine. But no one ever bothered to test A v1.2.3 and B v2.1.0 against each other…
    - 4y ago
whiddershins 4y ago
Does each microservice really need its own database? I have recently proposed my team initially not do this, and I'm wondering if I am creating a huge problem.
- lapser 4y ago
  Yes, you need it. Imagine having to make a change to the DB for one service. You'll have to coordinate between all microservices using that DB.
  - iamflimflam1 4y ago
    Agree and disagree - it really depends on why you are going to micro services. Is it because you have too many people trying to work on the same thing and you’re architecture is just a reflection of your organisation. Or is it to decouple some services that need to scale in different ways but still need to sit on top of the same data. Or is it some other reason? I think the dogmatic “you always need a separate database for each micro service” ignores a lot of subtleties - and cost…
    - bcrosby95 4y ago
      > Or is it to decouple some services that need to scale in different ways This is really over sold. You could allocate another instance to a specific service to provide more CPU to it, or you can allocate another instance to your whole monolith to provide more CPU. Maybe if the services use disproportionately different types of resources - such as GPU vs CPU vs memory vs disk. But if your resources are fungible across services, it generally doesn't matter if you can independently scale them. Compute for most projects is the easiest thing to scale out. The database is the hard part.
      - jiggawatts 4y ago
        > Compute for most projects is the easiest thing to scale out. The database is the hard part. If you did things the "right" way to begin with. You have to keep in mind that many people in the industry need to solve problems of their own making, and this then doesn't translate or make sense to other people. Scaling is a great example. It's common to see web applications written in horrendously inefficient front-end languages. Developers often forget to turn "debug" builds off, or spend 90% of the CPU cycles logging to text files. Single-threaded web servers were actually fairly common until recently. Then of course the web tier will have performance issues, which developers can paper over by scaling out to rack after rack of hardware. Single threaded web server? No worries! Just spin up 500 tiny virtual machines behind a load balancer. Meanwhile, the database engine itself is probably some CotS product. It was probably written in C++, is likely well-optimised, scalable, etc... So in that scenario, the database is the "easy" part that developers can just ignore, and scaling the web servers is the "hard" part. Meanwhile, if you write your front-end properly, then its CPU usage relative to the database engine will be roughly one-to-one. Then, scaling means scaling both the front-end and database tiers.
codethief 4y ago
Here's a quote from https://grugbrain.dev/ https://grugbrain.dev/ (discussed here on HN a while ago) which seems very appropriate: > Microservices: grug wonder why big brain take hardest problem, factoring system correctly, and introduce network call too. seem very confusing to grug
- ducharmdev 4y ago
  grug is the peak of SWE wisdom to me
- nostrebored 4y ago
  Seems like grug only works on tiny teams and on undifferentiated problems
  - forgetfulness 4y ago
    Well yeah, there's cases where it makes sense. I was once on a team dedicated to orchestrating machine learning models and their publishing in a prediction service. The training-to-publishing process was best kept an atomic one, and its Scala codebases separate and distinct from the Java monolith that called it, despite being the only caller and not benefiting from asynchronicity in the call, this design, implementation and operation of a very profitable part of the business was way more convenient than having tried to shove the prediction logic inside the caller. But there are teams that will be splitting every damned endpoint of a service into their own process, that's just unnecessary overhead from whichever way you look at it; network, deployment, infrastructure, monitoring, testing, and deployment again because it bears repeating that it complicates rollouts of new features.
  - roflyear 4y ago
    You miss the point. Fucking up your infrastructure isn't a solution to organizational problems.
    - benreesman 4y ago
      Bingo. In the long tradition of COM/CORBA, XML World, capital-A Agile, micro-services are hard to argue against because people fucking freak out if you push hard, because it’s a career path.
muh_gradle 4y ago
Enjoyed the article, and found #11 and #12 to be almost a requirement for most teams. I've worked for an org that adopted microservices when it's small size didn't justify it but eventually found its footing with good containerization, K8s from a growing DevOps team. Individual teams were able to eventually deploy and release to various environments independently, test on QA environments easily. And I now work for an org with a monorepo with 100+ developers that probably should have been broken up into microservices a while ago. Everything just feels broken and we're constantly running around wondering who broke what build when. We have like 6 SREs for a team of 100+ devs? I think a lot depends on how well CI/CD is developed and the DevOps/SRE team.
- aliswe 4y ago
  > And I now work for an org with a monorepo with 100+ developers that probably should have been broken up into microservices a while ago. If its a monolith, then I think its not a monorepo?
ianpurton 4y ago
You can actually use microservices to reduce the complexity of a monolith. One example which I built just for this purpose is Barricade https://github.com/purton-tech/barricade https://github.com/purton-tech/barricade which extracts authentication out of your app and into a container.
- bvrmn 4y ago
  Authentication is a very simple part and doesn't add complexity. Every platform has a plethora of libraries to deal with anything. Authorization is a real deal.
  - ianpurton 4y ago
    There are libraries but you still have to integrate them into a front end whilst being aware of any security issues. Those libraries generally come with lots of dependencies so splitting out the front end and back end auth code into a container can reduce the complexity of your main app and is a nice separation of concerns in my opinion.
    - bvrmn 4y ago
      But you solution requires architectural decisions. Like fixed DB. For example main app uses Mongo. And now it must be integrated with PG. I choose orthogonal set of primitives as a library any day. Frameworks and more so 3d-party services are too rigid and can contain hidden maintenance cost regarding flexibility. For example, it seems barricade doesn't support TOTP or SMS. What should I do as a backend developer?
      - ianpurton 4y ago
        Agreed Barricade is a Postgres solution so may not fit if you are already using Mongo. Barricade supports OTP via email. TOTP is on the road map.
  - hamandcheese 4y ago
    And best of luck to anyone attempting to extract authorization to a microservice in a way that doesn't create new problems.
bvrmn 4y ago
Looks like microservices became a self-goal. However author can be praised buy giving a context (implicit) of a huge team which should be split.
bvrmn 4y ago
Why microservices called "micro"? When it's almost implied it requires a dedicated team to develop and maintain? Looks it's like SOA but with JSON and k8s.
- LeonM 4y ago
  In the context of microservices, 'micro' means that the service itself is small. Thus, each service performs a 'micro' (and preferably independent) task. It is the opposite of a monolith, which you could call a 'macro' service. The system as a whole (whether it being monolith or microservices) still requires a dedicated team to maintain. Switching to microservices will not magically remove the team requirement. In fact, splitting a monolith into smaller services creates overhead, so you'll probably end up with a larger team then what you'd have to maintain a monolith.
  - yrgulation 4y ago
    In reality many of the micro services end up as tightly coupled macro services. I’ve rarely seen teams with the discipline or need for creating truly self contained separate services.
- mirekrusin 4y ago
  Maybe because there are very little real world cases when they're better than alternatives?
chrsig 4y ago
Some thoughts: - #12 add observability needs to be #1. if you can't observe your service, for all you know it's not even running. Less hyperbolically, good instrumentation will make every other step go faster by lowering the time to resolution for any issues that come up (and there will be some) - #11 is incredibly oversimplified, and potentially dangerous. how to do double writes like that and not create temporal soup is...complicated. very complicated. it's important to remember that the database the app is sitting on top of is (probably) taking care of a great many synchronization needs. if you can do a one way replication, that drastically simplifies things. otherwise either do it in the monolith before breaking up into services, or do it after you've broken up services, and have the services share the database layer in the interim. (I'm not debating that it needs to be done -- just advocating for sane approaches) - #10 - I've had great results with the strangler pattern. Intercepting data at I/O gives a lot of tools for being able to gradually change internal interfaces while keeping public/external interfaces constant. - #5 - as you introduce more processes, integration and end to end testing becomes more and more vital. it becomes harder and harder to run the service locally, and it becomes harder to tell which where a problem is occurring. cross service debugging can be a nightmare. in general it's just important to keep an eye on what the system is doing from an outside perspective and if any contracts have inadvertently changed behaviors.
- jjav 4y ago
  > add observability needs to be #1 Very much this. The importance of this is overlooked so often and then when there is a problem it's far more difficult to solve than it should've been.
- tomfern 4y ago
  Author here. I agree that observability is really high priority. I've never meant the numbers to represent steps or say that one point has a higher priority than others (maybe I didn't make that clear enough in the article). In theory, you would be following all/most of the points while you prepare the monolith for migration.
sackerhews 4y ago
I was once working with a guy that was hell bent on microservices. When our banking application became incompatible with logging, due to his microservice design, he really argued, fought and sulked that logging wasn't important and we didn't need it.
- choward 4y ago
  I'm skeptical of over using microservices but I don't quite understand how they make an app "incompatible with logging".
  - aliswe 4y ago
    Microservices are incompatible with monlithic text based logging - maybe they meant that?
- why-el 4y ago
  I suppose tracing becomes more important in that new architecture, assuming of course that each service is logging the unique identifier for the trace (for instance the request ID or some system-initiated event ID), but of course that presupposes logging to begin with, so I am not sure what "incompatible with logging" means.
dinvlad 4y ago
Is there a guide on doing the opposite?
jacquesm 4y ago
Microservices have one big advantage over monoliths: when you have a very large number of employees developing software it means that you can keep your teams out of each others' hair. If you don't have a very large team (as in 50+ developers) you are probably better off with a monolith, or at best a couple of much larger services that can be developed, tested and released independently of each other. That will get you very far, further than most start-ups will ever go.
- mbrodersen 4y ago
  Not my experience at all. It takes the same skills to manage a monolith as it does managing microservices. However microservices are inherently more complex than monoliths. So they are harder to keep clean/maintainable. All complex problems/systems are solved by breaking them down into smaller problems/systems that can be solved. Using microservices doesn’t make that easier. It actually makes it harder because you now also have to deal with the complexity of a distributed system.
  - jacquesm 4y ago
    The advantages of services and the complexity of distributed systems are obviously at odds with each other and proper architecture is going to be a requirement to be able to use a services oriented setup to its advantage. If your team was bad at systems architecture before attempting such a project then they will do much worse than before. But this is akin to saying that cooks should not use sharp knives because they can cut themselves. Yes they can. But expert cooks would much rather use sharp knives than blunt ones even if they run the risk of cutting themselves. A team of also-rans that go for a services oriented solution in a cargo cult fashion is going to get bogged down by complexity. That's no change from them getting bogged down by using a monolith that turned into a bowl of spaghetti. I've seen microservices applied in ways that made my hair stand on end, every shitty little function was its own service and had its own API. Utter madness. But I've also seen a monolith with several major functional blocks cut up on sensible lines and then dealt with by a much larger number of teams than before with success. Like everything else: if you are doing it wrong no amount of magic is going to help you, you first need to know what it is that you are doing. My best advice if this sounds familiar is that you should up the hiring bar, considerably so. But that will also require paying more per person and companies that have a problem with the former usually also have a problem with the latter.
hinkley 4y ago
> 10 Modularize the monolith A couple paragraphs on a couple of tools for the middle to late stages of such an effort is tantamount to "and then draw the rest of the fucking owl". Decomposition is hard. It's doubly hard when you have coworkers who are addicted to the vast namespace of possibilities in the code and are reaching halfway across the codebase to grab data to do something and now those things can never be separated. One of the best tools I know for breaking up a monolith is to start writing little command line debugging tools for these concerns. This exposes the decision making process, the dependency tree, and how they complicate each other. CLIs are much easier to run in a debugger than trying to spin up the application and step through some integration tests. If you can't get feature parity between the running system and the CLI, then you aren't going to be able to reach feature parity running it in a microservice either. It's an easier milestone to reach, and it has applications to more immediate problems like trying to figure out what change just broke preprod. I have things that will never be microservices that benefit from this. I have things that used to be microservices that didn't need to be, especially once you had a straightforward way to run the code without it.
SergeAx 4y ago
I have a strong feeling that the author never in their career transitioned from monolith to microservices. Even to the point "we are getting somewhere", not to mention "we are successful at this". Text reads like self- complacency.
iasay 4y ago
Advice. Don’t bother if you have a monolith. Just keep trucking on with it and partition your stuff vertically with shared auth and marketing site. Fuck microservices unless you have a tiny little product with a few million users. Which is almost none of us.
yawnxyz 4y ago
hold on, if we have half a dozen web apps that serve different needs, but all share similar "guts" — a monolith doesn't really make sense because App B will have all the code of App A but only use like 10% of it... So if I split App A from App B (and App C, D, etc.), but create a single "back-end monolith" for both apps, that makes sense right? Is that still microservices? And if half of them need authentication, but I want to keep my auth data separate from my other backend data, shouldn't I create a "regular db backend" and an "auth backend", for my many web apps? Essentially this is what I've done. I don't start with "microservices" but at the same time I don't want to glom ALL my random sites and apps' front-end and back-ends into the same code base... that makes upkeep WAY too difficult. Is this a monolith, a microservices, or none of those? I'm not really a "real" engineer and haven't drank any koolaid so I honestly don't know if this is the right way to do it. Most of my sites and web apps are Sveltekit / Vercel, with CF Workers, Supabase, Deta, and Railway doing some other stuff Vercel can't do. (I don't have a hosted machine or anything lol)
- majormajor 4y ago
  This sounds like "multiple frontend apps talking to a single API service." Frontend apps are a different beast here, the general "monolith -> microservice transition" like discussed here is talking just about the backend services. For standalone app code, that "I only share 10% of the stuff" problem can be fairly easily solved with libraries. For backend services, libraries most likely aren't what you want, because you don't want 5 different services all using the same library to talk directly to the same SQL database or whatever. You'd rather have a single service responsible for that database communication. In a microservices world, you tend towards "one logical domain object in the system? one service." In a more monolithic world, you have much larger groupings, where a single service might be related to, say, user accounts and user text content and references to photo libraries, or whatever. It's probably not actually everything (some stuff scales at wildly different rates than others) but you aren't nearly as aggressive at looking for ways to split stuff up, and your default for new functionality tends to be to find the service to add it to.
- jjtheblunt 4y ago
  If different apps share code, why is that code not in a linked-against library, like has been normal for 40 years or more?
wly_cdgr 4y ago
90%+ of transitions to microservices occur because the developers involved need to give the impression they are doing something in the absence of being allowed to build something new like they would prefer, and want to put "microservices" on their resume for when they try to leave their current shithole of a company for a slightly better paying, trendier shithole
- gscott 4y ago
  Biggest mistake of my life was to say lets keep maintaining and building on the old thing rather then transitioning to the newest greatest (language, micro-service, whatever). The old thing on your resume does nothing for you. Whatever is the new thing, jump on it hard and fast like your life depends on it.
mbrodersen 4y ago
If you don’t have the skills to write modular maintainable monoliths then you most definitely don’t have the skills to implement modular maintainable microservices. It’s the same skills required to do both: the ability to break down a complex system/problem into a number of simpler systems/solutions/modules/libraries/services/… so that the complex system/problem can be solved by composing those simpler systems/solutions/modules/libraries/… into a single system/monolith/…
deepsquirrelnet 4y ago
I sympathize with both positions — however, spend any time working somewhere like a bank, or hardware/electronics company founded before 1980, and you’ll develop a distinct distaste for monoliths. I’ve seen monoliths create teams of career software bureaucrats who exist to choke out companies to their last breath. Companies will go underserved for decades while underfunded projects fail to replace a monolith one after another. All crossroads lead to the monolith that was written in some language that died 30 years ago, performs like it did 30 years ago and is maintained by people hired 30 years ago — creating imminent crises with their looming retirements. I think a lot of people here read this article as writing monoliths vs microservices — not so. There are many very old, inherited monoliths that are actively creating maintenance crises at many businesses. These need to be modernized in chunks, because full rewrites take years to fail (and often do, eg. the mythical data warehouse). Microservices offer some incremental path out of the quagmire.
- thfuran 4y ago
  >teams of career software bureaucrats who exist to choke out companies to their last breath. Companies will go underserved for decades while underfunded projects fail to replace a monolith one after another. All crossroads lead to the monolith that was written in some language that died 30 years ago, performs like it did 30 years ago and is maintained by people hired 30 years ago — creating imminent crises with their looming retirements. None of this seems like a monolith problem, it seems like a very-legacy codebase at a non-tech company problem.
  - lolinder 4y ago
    I think their point is that trying to continue to work inside such a monolith is soul-crushing, but attempts to replace the monolith in one step fail because it's too big. I do think there's some confusion, though, because microservices are not the only solution to this problem. You can solve it with two services: the big legacy service hosts all the logic you haven't cleaned up yet, the small-but-growing modern service has the logic that you've cleaned up and made livable. This way you don't have to replace the old monolith in one step, but you also don't need to go to microservices—the target is still one well-factored monolith. (Edit: there's a risk here, of course, that your new code becomes another legacy ball of mud that everyone hates and the cycle repeats, but this time with two services...)

235 points by ahamez 4y ago

A lot of those steps just seem like good engineering. (I personally prefer modular monoliths over microservices though, in all but very few cases.)

tleasure 4y ago
"I personally prefer modular monoliths over microservices though, in all but very few cases." Couldn't agree more. Often times folks are using microservices to achieve good modularity at a cost.
- paskozdilar 4y ago
  > Often times folks are using microservices to achieve good modularity at a cost. And even more often, folks use microservices, but make them so coupled that you can't really run/test/hack one without running all the others... Basically creating a distributed monolith.
  - deleted 4y ago
    [deleted]
  - delecti 4y ago
    Strong agree. I worked on a "distributed monolith" once, and now I loudly make it a requirement that all my team's microservices can easily be run locally. Running one stack locally required starting up 12 different microservices, all in a particular order, before you could do anything with any of them. Insanity.
- Akronymus 4y ago
  Kinda reminds me of how you "need to have a horizontally scaling database setup because 1 million rows a day are impossible to handle via a normal database server" people really underestimate the power that vertical scaling can achieve, along with the long tail that microservices can bring. (The more calls between services you need to handle x, the more likely it is that you get into a case where one call takes exceptionally long) https://www.youtube.com/watch?v=SjC9bFjeR1k https://www.youtube.com/watch?v=SjC9bFjeR1k
  - iamflimflam1 4y ago
    I’ve faced this a number of times “we think we’ll have scaling issues” when they are running on the lowest possible database tier on offer. I think people just don’t understand how much power they actually have at their fingertips without needing anything esoteric.
- paskozdilar 4y ago
  (second comment - after I had some time to think about it) Actually I think microservices serve as a tool for enforcing modularity. When pressure is high, corners are cut, and when unrelated code is easy to reach (as in case of a monolithic codebase), it's easy to fall into temptation - a small dirty hack is faster than refactoring. And when you consider different maintainers, it's easy to lose track of the original monolith architecture idea in all the dirty hacks. Microservices enforce some kind of strict separation, so in theory nobody can do anything that they're not supposed to. In practice, a lot of coupling can happen at the microservices level - the most common symptom being some weird isolated APIs whose only purpose is to do that one thing that another service needs for some reason. Those kind of ad-hoc dependencies tend to make services implementation-specific and non-interchangeable, therefore breaking modularity. So, in conclusion, some approaches are easier to keep modular than others, but there seems to be no silver bullet for replacing care and effort.
xbar 4y ago
Right. I feel like it's a better article without any reference to microservices.
davidkuennen 4y ago
Having a modular monolith myself I couldn't agree more.
spaetzleesser 4y ago
"A lot of those steps just seem like good engineering. " Agreed. I always wonder why people think that their inability to write libraries with good modularization will be solved by introducing microservices.
- yrgulation 4y ago
  It takes experience and guts to know when to use what and most people just go with the the latest and fanciest. Well tested, focused and self contained libraries are good architecture even when micro-services are a must.

greatpostman 4y ago

People forget, micro services commonly serve massive tech companies. With 100 developers working on the same product, you need it broken up and separated. If you’re in a small company, the value proposition is not as great. It’s contextual, and a tech stack that solves organizational problems, not always technical ones

adra 4y ago
Agreed, but I'll go a step further and say microservices are really valuable when you have products that are powered through a reasonable DevOps and CI/CD support within and organization. If you're a company that only releases quarterly, a monolith probably makes sense. If you're a company releasing changes daily / hourly, monoliths make progressively less sense and become progressively harder to work with. When we release software (a lot) our downtime SLO is generally zero minutes. If you're a well out together outfit with strict discipline, this can be achieved with microservices. Inversely, monoliths almost never have to deal with multiple components at different release levels, so they don't do a particularly good job to support it, which is why you often see hours long upgrade windows for monoliths. Shut everything down, deploy updates, start everything back up and hope the house is still working plus the changes.
- codethief 4y ago
  > If you're a company releasing changes daily / hourly, monoliths make progressively less sense and become progressively harder to work with. Counter argument: Your overall CI/CD infrastructure landscape will be much simpler and muss less error-prone. As will be your release and test pipelines. For instance, if you have a few dozen microservices, each being released every other hour, how do you run E2E tests (across all services) before you release anything? Let's say you take your microservice A in version v1.2.3 (soon to be released) and test it against microservice B's current prod version v2.0.1. Meanwhile, team B is also working on releasing the new version v2.1.0 of microservice B and testing it against A's current prod version v1.2.2. Both test runs work fine. But no one ever bothered to test A v1.2.3 and B v2.1.0 against each other…
  - 4y ago

whiddershins 4y ago

Does each microservice really need its own database? I have recently proposed my team initially not do this, and I'm wondering if I am creating a huge problem.

lapser 4y ago
Yes, you need it. Imagine having to make a change to the DB for one service. You'll have to coordinate between all microservices using that DB.
- iamflimflam1 4y ago
  Agree and disagree - it really depends on why you are going to micro services. Is it because you have too many people trying to work on the same thing and you’re architecture is just a reflection of your organisation. Or is it to decouple some services that need to scale in different ways but still need to sit on top of the same data. Or is it some other reason? I think the dogmatic “you always need a separate database for each micro service” ignores a lot of subtleties - and cost…
  - bcrosby95 4y ago
    > Or is it to decouple some services that need to scale in different ways This is really over sold. You could allocate another instance to a specific service to provide more CPU to it, or you can allocate another instance to your whole monolith to provide more CPU. Maybe if the services use disproportionately different types of resources - such as GPU vs CPU vs memory vs disk. But if your resources are fungible across services, it generally doesn't matter if you can independently scale them. Compute for most projects is the easiest thing to scale out. The database is the hard part.
    - jiggawatts 4y ago
      > Compute for most projects is the easiest thing to scale out. The database is the hard part. If you did things the "right" way to begin with. You have to keep in mind that many people in the industry need to solve problems of their own making, and this then doesn't translate or make sense to other people. Scaling is a great example. It's common to see web applications written in horrendously inefficient front-end languages. Developers often forget to turn "debug" builds off, or spend 90% of the CPU cycles logging to text files. Single-threaded web servers were actually fairly common until recently. Then of course the web tier will have performance issues, which developers can paper over by scaling out to rack after rack of hardware. Single threaded web server? No worries! Just spin up 500 tiny virtual machines behind a load balancer. Meanwhile, the database engine itself is probably some CotS product. It was probably written in C++, is likely well-optimised, scalable, etc... So in that scenario, the database is the "easy" part that developers can just ignore, and scaling the web servers is the "hard" part. Meanwhile, if you write your front-end properly, then its CPU usage relative to the database engine will be roughly one-to-one. Then, scaling means scaling both the front-end and database tiers.

codethief 4y ago

Here's a quote from https://grugbrain.dev/ https://grugbrain.dev/ (discussed here on HN a while ago) which seems very appropriate: > Microservices: grug wonder why big brain take hardest problem, factoring system correctly, and introduce network call too. seem very confusing to grug

ducharmdev 4y ago
grug is the peak of SWE wisdom to me
nostrebored 4y ago
Seems like grug only works on tiny teams and on undifferentiated problems
- forgetfulness 4y ago
  Well yeah, there's cases where it makes sense. I was once on a team dedicated to orchestrating machine learning models and their publishing in a prediction service. The training-to-publishing process was best kept an atomic one, and its Scala codebases separate and distinct from the Java monolith that called it, despite being the only caller and not benefiting from asynchronicity in the call, this design, implementation and operation of a very profitable part of the business was way more convenient than having tried to shove the prediction logic inside the caller. But there are teams that will be splitting every damned endpoint of a service into their own process, that's just unnecessary overhead from whichever way you look at it; network, deployment, infrastructure, monitoring, testing, and deployment again because it bears repeating that it complicates rollouts of new features.
- roflyear 4y ago
  You miss the point. Fucking up your infrastructure isn't a solution to organizational problems.
  - benreesman 4y ago
    Bingo. In the long tradition of COM/CORBA, XML World, capital-A Agile, micro-services are hard to argue against because people fucking freak out if you push hard, because it’s a career path.

muh_gradle 4y ago

Enjoyed the article, and found #11 and #12 to be almost a requirement for most teams. I've worked for an org that adopted microservices when it's small size didn't justify it but eventually found its footing with good containerization, K8s from a growing DevOps team. Individual teams were able to eventually deploy and release to various environments independently, test on QA environments easily. And I now work for an org with a monorepo with 100+ developers that probably should have been broken up into microservices a while ago. Everything just feels broken and we're constantly running around wondering who broke what build when. We have like 6 SREs for a team of 100+ devs? I think a lot depends on how well CI/CD is developed and the DevOps/SRE team.

aliswe 4y ago
> And I now work for an org with a monorepo with 100+ developers that probably should have been broken up into microservices a while ago. If its a monolith, then I think its not a monorepo?

ianpurton 4y ago

You can actually use microservices to reduce the complexity of a monolith. One example which I built just for this purpose is Barricade https://github.com/purton-tech/barricade https://github.com/purton-tech/barricade which extracts authentication out of your app and into a container.

bvrmn 4y ago
Authentication is a very simple part and doesn't add complexity. Every platform has a plethora of libraries to deal with anything. Authorization is a real deal.
- ianpurton 4y ago
  There are libraries but you still have to integrate them into a front end whilst being aware of any security issues. Those libraries generally come with lots of dependencies so splitting out the front end and back end auth code into a container can reduce the complexity of your main app and is a nice separation of concerns in my opinion.
  - bvrmn 4y ago
    But you solution requires architectural decisions. Like fixed DB. For example main app uses Mongo. And now it must be integrated with PG. I choose orthogonal set of primitives as a library any day. Frameworks and more so 3d-party services are too rigid and can contain hidden maintenance cost regarding flexibility. For example, it seems barricade doesn't support TOTP or SMS. What should I do as a backend developer?
    - ianpurton 4y ago
      Agreed Barricade is a Postgres solution so may not fit if you are already using Mongo. Barricade supports OTP via email. TOTP is on the road map.
- hamandcheese 4y ago
  And best of luck to anyone attempting to extract authorization to a microservice in a way that doesn't create new problems.

bvrmn 4y ago

Looks like microservices became a self-goal. However author can be praised buy giving a context (implicit) of a huge team which should be split.

bvrmn 4y ago

Why microservices called "micro"? When it's almost implied it requires a dedicated team to develop and maintain? Looks it's like SOA but with JSON and k8s.

LeonM 4y ago
In the context of microservices, 'micro' means that the service itself is small. Thus, each service performs a 'micro' (and preferably independent) task. It is the opposite of a monolith, which you could call a 'macro' service. The system as a whole (whether it being monolith or microservices) still requires a dedicated team to maintain. Switching to microservices will not magically remove the team requirement. In fact, splitting a monolith into smaller services creates overhead, so you'll probably end up with a larger team then what you'd have to maintain a monolith.
- yrgulation 4y ago
  In reality many of the micro services end up as tightly coupled macro services. I’ve rarely seen teams with the discipline or need for creating truly self contained separate services.
mirekrusin 4y ago
Maybe because there are very little real world cases when they're better than alternatives?

chrsig 4y ago

Some thoughts: - #12 add observability needs to be #1. if you can't observe your service, for all you know it's not even running. Less hyperbolically, good instrumentation will make every other step go faster by lowering the time to resolution for any issues that come up (and there will be some) - #11 is incredibly oversimplified, and potentially dangerous. how to do double writes like that and not create temporal soup is...complicated. very complicated. it's important to remember that the database the app is sitting on top of is (probably) taking care of a great many synchronization needs. if you can do a one way replication, that drastically simplifies things. otherwise either do it in the monolith before breaking up into services, or do it after you've broken up services, and have the services share the database layer in the interim. (I'm not debating that it needs to be done -- just advocating for sane approaches) - #10 - I've had great results with the strangler pattern. Intercepting data at I/O gives a lot of tools for being able to gradually change internal interfaces while keeping public/external interfaces constant. - #5 - as you introduce more processes, integration and end to end testing becomes more and more vital. it becomes harder and harder to run the service locally, and it becomes harder to tell which where a problem is occurring. cross service debugging can be a nightmare. in general it's just important to keep an eye on what the system is doing from an outside perspective and if any contracts have inadvertently changed behaviors.

jjav 4y ago
> add observability needs to be #1 Very much this. The importance of this is overlooked so often and then when there is a problem it's far more difficult to solve than it should've been.
tomfern 4y ago
Author here. I agree that observability is really high priority. I've never meant the numbers to represent steps or say that one point has a higher priority than others (maybe I didn't make that clear enough in the article). In theory, you would be following all/most of the points while you prepare the monolith for migration.

sackerhews 4y ago

I was once working with a guy that was hell bent on microservices. When our banking application became incompatible with logging, due to his microservice design, he really argued, fought and sulked that logging wasn't important and we didn't need it.

choward 4y ago
I'm skeptical of over using microservices but I don't quite understand how they make an app "incompatible with logging".
- aliswe 4y ago
  Microservices are incompatible with monlithic text based logging - maybe they meant that?
why-el 4y ago
I suppose tracing becomes more important in that new architecture, assuming of course that each service is logging the unique identifier for the trace (for instance the request ID or some system-initiated event ID), but of course that presupposes logging to begin with, so I am not sure what "incompatible with logging" means.

dinvlad 4y ago

Is there a guide on doing the opposite?

jacquesm 4y ago

Microservices have one big advantage over monoliths: when you have a very large number of employees developing software it means that you can keep your teams out of each others' hair. If you don't have a very large team (as in 50+ developers) you are probably better off with a monolith, or at best a couple of much larger services that can be developed, tested and released independently of each other. That will get you very far, further than most start-ups will ever go.

mbrodersen 4y ago
Not my experience at all. It takes the same skills to manage a monolith as it does managing microservices. However microservices are inherently more complex than monoliths. So they are harder to keep clean/maintainable. All complex problems/systems are solved by breaking them down into smaller problems/systems that can be solved. Using microservices doesn’t make that easier. It actually makes it harder because you now also have to deal with the complexity of a distributed system.
- jacquesm 4y ago
  The advantages of services and the complexity of distributed systems are obviously at odds with each other and proper architecture is going to be a requirement to be able to use a services oriented setup to its advantage. If your team was bad at systems architecture before attempting such a project then they will do much worse than before. But this is akin to saying that cooks should not use sharp knives because they can cut themselves. Yes they can. But expert cooks would much rather use sharp knives than blunt ones even if they run the risk of cutting themselves. A team of also-rans that go for a services oriented solution in a cargo cult fashion is going to get bogged down by complexity. That's no change from them getting bogged down by using a monolith that turned into a bowl of spaghetti. I've seen microservices applied in ways that made my hair stand on end, every shitty little function was its own service and had its own API. Utter madness. But I've also seen a monolith with several major functional blocks cut up on sensible lines and then dealt with by a much larger number of teams than before with success. Like everything else: if you are doing it wrong no amount of magic is going to help you, you first need to know what it is that you are doing. My best advice if this sounds familiar is that you should up the hiring bar, considerably so. But that will also require paying more per person and companies that have a problem with the former usually also have a problem with the latter.

hinkley 4y ago

> 10 Modularize the monolith A couple paragraphs on a couple of tools for the middle to late stages of such an effort is tantamount to "and then draw the rest of the fucking owl". Decomposition is hard. It's doubly hard when you have coworkers who are addicted to the vast namespace of possibilities in the code and are reaching halfway across the codebase to grab data to do something and now those things can never be separated. One of the best tools I know for breaking up a monolith is to start writing little command line debugging tools for these concerns. This exposes the decision making process, the dependency tree, and how they complicate each other. CLIs are much easier to run in a debugger than trying to spin up the application and step through some integration tests. If you can't get feature parity between the running system and the CLI, then you aren't going to be able to reach feature parity running it in a microservice either. It's an easier milestone to reach, and it has applications to more immediate problems like trying to figure out what change just broke preprod. I have things that will never be microservices that benefit from this. I have things that used to be microservices that didn't need to be, especially once you had a straightforward way to run the code without it.

SergeAx 4y ago

I have a strong feeling that the author never in their career transitioned from monolith to microservices. Even to the point "we are getting somewhere", not to mention "we are successful at this". Text reads like self- complacency.

iasay 4y ago

Advice. Don’t bother if you have a monolith. Just keep trucking on with it and partition your stuff vertically with shared auth and marketing site. Fuck microservices unless you have a tiny little product with a few million users. Which is almost none of us.

yawnxyz 4y ago

hold on, if we have half a dozen web apps that serve different needs, but all share similar "guts" — a monolith doesn't really make sense because App B will have all the code of App A but only use like 10% of it... So if I split App A from App B (and App C, D, etc.), but create a single "back-end monolith" for both apps, that makes sense right? Is that still microservices? And if half of them need authentication, but I want to keep my auth data separate from my other backend data, shouldn't I create a "regular db backend" and an "auth backend", for my many web apps? Essentially this is what I've done. I don't start with "microservices" but at the same time I don't want to glom ALL my random sites and apps' front-end and back-ends into the same code base... that makes upkeep WAY too difficult. Is this a monolith, a microservices, or none of those? I'm not really a "real" engineer and haven't drank any koolaid so I honestly don't know if this is the right way to do it. Most of my sites and web apps are Sveltekit / Vercel, with CF Workers, Supabase, Deta, and Railway doing some other stuff Vercel can't do. (I don't have a hosted machine or anything lol)

majormajor 4y ago
This sounds like "multiple frontend apps talking to a single API service." Frontend apps are a different beast here, the general "monolith -> microservice transition" like discussed here is talking just about the backend services. For standalone app code, that "I only share 10% of the stuff" problem can be fairly easily solved with libraries. For backend services, libraries most likely aren't what you want, because you don't want 5 different services all using the same library to talk directly to the same SQL database or whatever. You'd rather have a single service responsible for that database communication. In a microservices world, you tend towards "one logical domain object in the system? one service." In a more monolithic world, you have much larger groupings, where a single service might be related to, say, user accounts and user text content and references to photo libraries, or whatever. It's probably not actually everything (some stuff scales at wildly different rates than others) but you aren't nearly as aggressive at looking for ways to split stuff up, and your default for new functionality tends to be to find the service to add it to.
jjtheblunt 4y ago
If different apps share code, why is that code not in a linked-against library, like has been normal for 40 years or more?

wly_cdgr 4y ago

90%+ of transitions to microservices occur because the developers involved need to give the impression they are doing something in the absence of being allowed to build something new like they would prefer, and want to put "microservices" on their resume for when they try to leave their current shithole of a company for a slightly better paying, trendier shithole

gscott 4y ago
Biggest mistake of my life was to say lets keep maintaining and building on the old thing rather then transitioning to the newest greatest (language, micro-service, whatever). The old thing on your resume does nothing for you. Whatever is the new thing, jump on it hard and fast like your life depends on it.

mbrodersen 4y ago

If you don’t have the skills to write modular maintainable monoliths then you most definitely don’t have the skills to implement modular maintainable microservices. It’s the same skills required to do both: the ability to break down a complex system/problem into a number of simpler systems/solutions/modules/libraries/services/… so that the complex system/problem can be solved by composing those simpler systems/solutions/modules/libraries/… into a single system/monolith/…

deepsquirrelnet 4y ago

I sympathize with both positions — however, spend any time working somewhere like a bank, or hardware/electronics company founded before 1980, and you’ll develop a distinct distaste for monoliths. I’ve seen monoliths create teams of career software bureaucrats who exist to choke out companies to their last breath. Companies will go underserved for decades while underfunded projects fail to replace a monolith one after another. All crossroads lead to the monolith that was written in some language that died 30 years ago, performs like it did 30 years ago and is maintained by people hired 30 years ago — creating imminent crises with their looming retirements. I think a lot of people here read this article as writing monoliths vs microservices — not so. There are many very old, inherited monoliths that are actively creating maintenance crises at many businesses. These need to be modernized in chunks, because full rewrites take years to fail (and often do, eg. the mythical data warehouse). Microservices offer some incremental path out of the quagmire.

thfuran 4y ago
>teams of career software bureaucrats who exist to choke out companies to their last breath. Companies will go underserved for decades while underfunded projects fail to replace a monolith one after another. All crossroads lead to the monolith that was written in some language that died 30 years ago, performs like it did 30 years ago and is maintained by people hired 30 years ago — creating imminent crises with their looming retirements. None of this seems like a monolith problem, it seems like a very-legacy codebase at a non-tech company problem.
- lolinder 4y ago
  I think their point is that trying to continue to work inside such a monolith is soul-crushing, but attempts to replace the monolith in one step fail because it's too big. I do think there's some confusion, though, because microservices are not the only solution to this problem. You can solve it with two services: the big legacy service hosts all the logic you haven't cleaned up yet, the small-but-growing modern service has the logic that you've cleaned up and made livable. This way you don't have to replace the old monolith in one step, but you also don't need to go to microservices—the target is still one well-factored monolith. (Edit: there's a risk here, of course, that your new code becomes another legacy ball of mud that everyone hates and the cycle repeats, but this time with two services...)