9 ms·

We turned our monolith into a bunch of micro services almost 6 years ago to the day. For a long time I was very happy with the new pattern but over the years th

We turned our monolith into a bunch of micro services almost 6 years ago to the day. For a long time I was very happy with the new pattern but over the years the weight of keeping everything updated along with the inevitable corners that fall behind and have...questionable..security due to how long they sit neglected has really left me wondering if I am happy with it after all. I would love hear some thoughts from others that made the move, especially anyone that decided to move back to a monolith repo.

tuckerconnelly 7y ago
I moved back to monolith and am very happy. I think of the monolith now as a collection of modules. The rule is now, one should be able to drag any of the modules to the top-level of our monorepo and create a new microservice pretty easily when the time comes. I think the microservices book (that came from that Uber engineer...?) suggests a rule of 5 engineers per service.
- virmundi 7y ago
  How are you preventing transaction couplings? For example, module A and B are called by C. C starts a transaction that wraps A and B. If you move B up as a network feature, you lose the transactionalty.
  - m0zg 7y ago
    Usually the answer to this is "we pray to god it doesn't fail".
  - deleted 7y ago
    [deleted]
  - tuckerconnelly 7y ago
    Good question, and this rule is meant to be bent in those scenarios. I try to avoid these dependencies if at all possible, but if not possible, the "writes" for those modules all belong to a single service, and any other service, depending on how "pure" I need to be in the project, will make network calls to the other service, or just grab that data directly from the database. For a more concrete example, I recently built a service ("scraper") that scraped data and upserted a large tree of structured data to postgres in a transaction. Writes were only allowed from scraper, but "api" could SELECT data for reporting to the frontend "web" as much as it wanted. In the future, "api" might make be refactored to make internal HTTP request to "scraper," so they could have totally separate databases.
osdev 7y ago
Like many things in life, there are no absolutes. So rather than going to opposite ends of the spectrum, check out modular monoliths. This approach is a nice balance (IMHO), since you start off with monolith but it is broken down into cleanly separated modules. Each module can potentially become its own micro-service if or when the time comes. In terms of implementation, this can be easily done, for example we do it JVM/Kotlin where each microservice is its own project that produces a binary/jar. All the projects are part of a multi-project build. Lastly we have a common project for shared code, utils, types, enums, interfaces, etc and an application project that loads/sets up all the microservices from each project. Works great so far. And when you do have break up 1 project into its own service, the effort is fairly manageable.
- rraghur 7y ago
  Do you also break up the data for each service up front or do you pull that out later?
dashwav 7y ago
I think that the fact you have much more visibility over which of the services are behind and have lower security practices one of the things I love about our microservices - we recently (a year ago) broke up our monolithic codebase into a service oriented architecture (I would hesitate to call our services micro personally) and I was astounded at all of the hidden security issues and random code in the far reaching corners of the monolith that hadn't been touched or thought about in years. It is much easier (imo) to pop into a repo of one of our services and look through the code in it's enitrety and see when things were last touched and where the issues are. I would make the argument that the "inevitable corners that fall behind and have questionable security" is something that is inevitable in any codebase that grows to a certain complexity, and microservices (or SOA in general) make it much easier to see those things as they are decomposed.
- marcosdumay 7y ago
  Well, I can only agree with all that... But the move to services adds a lot of surface that needs it's own security and architecture maintenance. The more you break down your code (the smaller the size of the services), the more maintenance need is created from the division, and the easier it is to fall behind on it.
dehrmann 7y ago
> weight of keeping everything updated...monolith repo You need CD or this is an accident waiting to happen.
zo1 7y ago
A recent project I was exposed to has been struggling with Microservices and a multi-repo setup. Even with CI/CD and a lot of good tooling around their setup. The overhead introduced with having such a setup in a corporate environment that has not-so-well-though-out requirements and design is ridiculous. Keeping track of dependencies, arcane knowledge of inter-service dependency quirks being siloed and hidden, keeping individual services up to date, dealing with older services and their interaction with newer services till they "migrate" to newer tooling/common code, etc. Everyone then skirts around the fact that the problem could potentially be Microservices or a micro-repo setup. Instead, they throw process, sign-off, complicated promotion pipelines and just plain warm-bodies at the problem in an attempt to mitigate it. But the damage is done, velocity has slowed to a crawl and everyone is miserable, especially when having to explain the whole thing to newcomers.
- pm90 7y ago
  The best ideas can be implemented poorly. Software design principle that works for nimble Silicon Valley startups doesn’t work in your big corporate environment? Big surprise. When I worked at a pretty large software co, a team there were always adopting the latest techniques and tools but their deployment pipeline was an over architected disaster that nobody could reliably deploy. Reason? Their CI/CD system consisted of a person manually clicking around to build Jenkins Jobs. There will be other such silly nonsense (hopefully less extreme) in other corporate environments too.
- deleted 7y ago
  [deleted]
bakul 7y ago
IMHO a microservice should follow the unix philosophy of doing one thing well. But interfacing with other services is not as simple as unix pipes. In particular it is more of a request/response communication. Consequently, interfacing needs to be thought through carefully and made as simple as possible. They should evolve much more slowly than individual service code. While you may use (g)rpc or http at a lower level, your specific protocols will have many more constraints. Ideally you have codified assertions about their behavior in tests early on. Note that they are not all going to start/stop at exactly the same time and during development they may even crash so each microservice should survive such transitions. You may have two different versions of service code or even completely different implementations or two different version of the API in use at the same time. This can happen as different services evolve at a different rate. And you may want to transition to a new version in a piecemeal manner so as to not bring the whole service down. All these considerations complicate things. So ideally you have factored out these common tasks in shared library/packages. And ideally you write your code such that if necessary more than one service can be compiled into the same binary for performance reasons. In a monolith some things become easier since everything dies at once! But somethings become more complicated - such as supporting evolving code. And monolitha require more discipline to keep things modular. Over time this gets harder and harder. Lack of modularity means you have to understand a lot more code and when you evolve things, more code will have to change and there may be unforeseen side-effects. And scaling can become harder.
dangoor 7y ago
Our approach to services at Khan Academy is likely a bit different from most. We're sticking with a monorepo (the code for all services lives in one repository). We have a single go.mod file at the top of the repo, so all services use the _same versions_ of dependencies. We're still building out our deployment system to better support multiple services, but we're planning to redeploy all of the services when library code changes (which is something we're trying to minimize). All of this ensures that we don't have trouble with services lagging behind on critical updates.
- solidasparagus 7y ago
  I don't really understand moving to microservices if you aren't going to give the service teams the autonomy to make their own decisions and move at their own pace. Microservices always seemed to me to be more of an organizational strategy than a technical one - if you have services, but they can't operate independently, it feels a bit like you are just creating a distributed monolith.
  - gtirloni 7y ago
    Choosing the dependencies' version is not all there's to decide. Teams have a lot of control of basically everything else.
    - andyroid 7y ago
      So, no choice of language and no choice of the libraries used. What’s “everything else” exactly? Sounds like missing out on the more interesting parts of a micro service architecture.
      - sangnoir 7y ago
        Yeah, I'll never work for a company[1] where service teams are free to choose any language; 2 or 3 options at most is fine, but more than that is a hard no. I'll have to read/work on that code sooner or later, and I have no time to be dealing with a hodgepodge of languages 1. In the 10-5000 employee range: tech giants are a different beast when it comes to team accountability.
axaxs 7y ago
Same. I tried microservices before and while it cleaned up the code base, I didn't quite like the results down the road. Some things go stale, you multiply ops * number of microservices, a change in one can mean a change in multiple others. I'm not against services in general, but not a fan of so called microservices.
kodablah 7y ago
I've found there's a happy middleground. You need medium-sized-services that still share code libraries. For many companies, this is often 7 or 8. The key is to combine like business units/features, not necessarily fragmenting at every visible code boundary. A deployed "service" can really just be several HTTP paths and/or gRPC services in one repo. You still get to keep decent separation of work, deployment, versioning, dev focus, etc with these medium sized services without sacrificing the benefits of larger, more centralized/shared reuse.
- stult 7y ago
  I’ve accidentally landed on an architecture like this and I’m actually pretty happy about it. It was driven by a desire to kill a large monolith slowly, by extracting key features into separate services. Sold to the customer as microservices because sexy fad of the moment. Our real motivation was we had way too much trouble recruiting in the monolith stack (.NET) and had a surplus of embedded C++, python, and JS engineers. Anyway, turns out our teams naturally self-organize around four or five domain+language clusters that effectively form separate services that are too large to really be micro, but too dissimilar to play nicely together in a monolith. Eg, python data science module wrapped in a Flask API providing physics calcs, legacy C# service providing simple data models via REST, JS react front end served from a separate node service, a weird C++/python hybrid used for an embedded device sim service, etc. It’s not what I planned as the tech lead/architect, but I think we organically reached what is really the best approach for our team. Definitely an element of Conway’s law in action, but in a good way. We are making the most of our organizational structure rather than fighting against it. Would this scale to google levels? No, probably not. But we don’t need it to, and it’s incredibly unlikely we ever would given our specific business.
bborud 7y ago
Start with a monolith that has clear internal APIs that are designed so they can later be made into network APIs. This gives you the development speed of a monolith while maintaining an options for the future. When you do break things out into separate services: try to make as few of them as possible and maintain the ability to build as a monolith. Forget everything you have heard about micro services. Most of it is bullshit from people who don’t actually think for themselves.
- bcrosby95 7y ago
  This. If you can't design a well segmented monolith, you can't design a well segmented system of microservices either. The microservices will just be buggier and much harder to fix after the fact.
  - bborud 7y ago
    Design is best evolved. If you can get a lot of that done while you still are able to run a well structured system as a monolith you can save a lot of time. It is cheaper to change an interface in Java/Golang than a REST API.
- stult 7y ago
  Could not agree more. I recently led a refactor of a monolithic .NET MVC app and took this exact approach. We made all of the controllers thin, with almost no logic at all beyond specifying the route and dependency injection. Then redirected the request to a “service”, which originally was just a reworked combination of the old controller/model logic hidden behind a common service interface. Then, slowly we replaced the C# services with microservices. So we went from ball of spaghetti to monolithic service oriented architecture lite to actual microservices with the monolith converted into an API gateway. If we didn’t have independent motives for going to microservices, sticking to the clean and well organized internal APIs of the refactored monolith would have been totally fine.
twblalock 7y ago
In my experience, the biggest benefit of microservices is decoupling teams. Developer productivity is very hard to maintain in a monolithic app as the number of developers increases and the legacy code piles up. Breaking up services and giving each dev team control over their own codebases enables them to develop their own products at their own pace. If you only have one dev team, microservices are a lot less attractive. However, there are still some benefits, such as being able to refactor parts of your codebase in isolation (including perhaps rewriting them in different languages), and the ability to individually adjust the runtime scale of different parts of your codebase.
- whatsmyusername 7y ago
  This. The problem with decoupling services is usually there end up being a couple services that are critical but not sexy. No one wants to touch them so they sit around unmaintained until an unrelated change or unpatched security issue comes around. Suddenly you've got a big problem with a mystery codebase.
  - viraptor 7y ago
    That sounds very familiar, but I'm not sure this is something that can be blamed on decoupling itself. An unpopular module is going to need as much attention as a micro service from code point of view. For upgrades / patching, there would be a company-wide process around it that doesn't care that much how the code is organised.
    - whatsmyusername 7y ago
      'company wide' process is either, 'squeaky wheel gets the grease,' or, 'no one even knows this exists,' 100% of the time in my experience. This is from going from 10k+ to 150 to 5k+ to 50 people.
      - viraptor 7y ago
        I wrote company-wide, but it's not the case everywhere. At some scale you'll want department, or even project-wide process. But the policy should be fairly common - who owns it, what's the response time, how to escalate urgent things, etc.
scottrogowski 7y ago
A company I am affiliated with made a decision to rewrite their code in microservices-oriented architecture thinking it would only take one year. Now we're 7 years into the transition and starting to come up against some hard deadlines that threaten revenue streams. It seems obvious to everyone except the leadership and the architects that this has been an unmitigated disaster. Other comments on this thread seem to indicate that many have had similar experiences. For those who are curious, here is a classic article on why rewriting code from scratch is a bad idea: https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/ https://www.joelonsoftware.com/2000/04/06/things-you-should-.... For a more in-depth analysis on the unforeseen challenges of microservices in particular, I would encourage a lot of careful research into how other companies have tried and failed at this. In particular, I might look at Uber's ongoing difficulties. All I have to say to the Khan Academy engineers is to buckle up because frankly, moving from Python 2->3 is not that hard and you have no idea what you are getting yourself into.
- tmpfile 7y ago
  this. 100% this.
- andy_ppp 7y ago
  Yes, the micro services + Golang vanity project because you think you’re Google. I really don’t think people understand error states in distributed systems very well, putting possible network partitions everywhere is not a great idea. I would strongly suggest trying a Golang monolith first and seeing if there are one or two heavily used services that need splitting off. Also monorepo. Always.
  - StavrosK 7y ago
    Microservices have always seemed dangerous to me. "Bugs thrive in the seams between libraries, so let's put more and deeper seams in!" Microservices have an immense cost, and you have to make sure they're worth it. Many teams years ago found it a nice pattern and implemented it because why not, and now we're at the "oops this isn't actually amazing" part of the cycle.
deleted 7y ago
[deleted]
hendry 7y ago
I way prefer monolith compared to dabbling in microservices here: https://natalian.org/2019/05/16/Microservices_pitfalls/ https://natalian.org/2019/05/16/Microservices_pitfalls/
jordanbeiber 7y ago
I’ve been through a transition with about 60 devs that went DDD plus microservices. A few monotliths ended up as a couple of hundred services, and looking back I feel we got basically all positives. What other people say about scaling teams is true, but I have a few other points as well: - personally, I spent 6 months writing tooling for service lifecycle management and setting strict conventions. This was before the microservices decision and I was recruited to do “devops” which gave me a lot of head room. :p - when talks about microservices surfaced I had to fight for several months to get the lead devs on-board with tooling and conventions. From the infra/cm/systems management side we’re used to manage many thousands of “configuration items” - devs are usually not, and many underestimate the value of proper automated lifecycle management. - once everyone was on-board we all used a common language. Big win. - I could form a “devops” team to help develop the tooling further and the infra platform as well. - almost all teams worked in mobs - amongst other things it really helped with the ownership part, something that’s absolutely crucial. Well defined domains and accountable mob teams, just awesome! - quality rose by a mile. Small commits and somewhat robust tooling under the eyes of a mob. - graph the pains. If outdated versions are a problem - put it on a graph, green, yellow, red. Show services in relation to other services - the application is the sum of all connected services. I could go on, but the post is getting long! :)
- gtirloni 7y ago
  Thanks for sharing, that's pretty exciting. I think most people don't realize the work they have to put in to make this work (and reap the benefits).
  - jordanbeiber 7y ago
    Thanks for reading. The more i write and talk about it, the more I realize that it is about ”externalities”, so to speak. A microservice is just code - one small piece that does one, tightly defined thing. We’ve been doing code always and smaller pieces of is easier to deal with and reason about. The structure around keeping 100s or 1000s of moving pieces in concert is where a lot of the work is shifted. It takes teamwork as well as a common vision and language. The above sentence tangents “culture”.
josteink 7y ago
> I would love hear some thoughts from others that made the move, especially anyone that decided to move back to a monolith repo. We had a big monolith where I work. We’ve been slowly, but surely isolating parts of the monolith as separate deliverables, extracted into their own repos. But only when appropriate, and not as a forced exercise. The remaining “monolith” is still pretty big, but it does (mostly) represent one logical deliverable, so effort to split it up has some what stalled. There’s been small points of friction, but nothing near as painful as we used to have it. No way we’re going back. So everything in moderation. Micro services architecture is a tool. Use it when it’s the right one.

tuckerconnelly 7y ago

I moved back to monolith and am very happy. I think of the monolith now as a collection of modules. The rule is now, one should be able to drag any of the modules to the top-level of our monorepo and create a new microservice pretty easily when the time comes. I think the microservices book (that came from that Uber engineer...?) suggests a rule of 5 engineers per service.

virmundi 7y ago
How are you preventing transaction couplings? For example, module A and B are called by C. C starts a transaction that wraps A and B. If you move B up as a network feature, you lose the transactionalty.
- m0zg 7y ago
  Usually the answer to this is "we pray to god it doesn't fail".
- deleted 7y ago
  [deleted]
- tuckerconnelly 7y ago
  Good question, and this rule is meant to be bent in those scenarios. I try to avoid these dependencies if at all possible, but if not possible, the "writes" for those modules all belong to a single service, and any other service, depending on how "pure" I need to be in the project, will make network calls to the other service, or just grab that data directly from the database. For a more concrete example, I recently built a service ("scraper") that scraped data and upserted a large tree of structured data to postgres in a transaction. Writes were only allowed from scraper, but "api" could SELECT data for reporting to the frontend "web" as much as it wanted. In the future, "api" might make be refactored to make internal HTTP request to "scraper," so they could have totally separate databases.

osdev 7y ago

Like many things in life, there are no absolutes. So rather than going to opposite ends of the spectrum, check out modular monoliths. This approach is a nice balance (IMHO), since you start off with monolith but it is broken down into cleanly separated modules. Each module can potentially become its own micro-service if or when the time comes. In terms of implementation, this can be easily done, for example we do it JVM/Kotlin where each microservice is its own project that produces a binary/jar. All the projects are part of a multi-project build. Lastly we have a common project for shared code, utils, types, enums, interfaces, etc and an application project that loads/sets up all the microservices from each project. Works great so far. And when you do have break up 1 project into its own service, the effort is fairly manageable.

rraghur 7y ago
Do you also break up the data for each service up front or do you pull that out later?

dashwav 7y ago

I think that the fact you have much more visibility over which of the services are behind and have lower security practices one of the things I love about our microservices - we recently (a year ago) broke up our monolithic codebase into a service oriented architecture (I would hesitate to call our services micro personally) and I was astounded at all of the hidden security issues and random code in the far reaching corners of the monolith that hadn't been touched or thought about in years. It is much easier (imo) to pop into a repo of one of our services and look through the code in it's enitrety and see when things were last touched and where the issues are. I would make the argument that the "inevitable corners that fall behind and have questionable security" is something that is inevitable in any codebase that grows to a certain complexity, and microservices (or SOA in general) make it much easier to see those things as they are decomposed.

marcosdumay 7y ago
Well, I can only agree with all that... But the move to services adds a lot of surface that needs it's own security and architecture maintenance. The more you break down your code (the smaller the size of the services), the more maintenance need is created from the division, and the easier it is to fall behind on it.

dehrmann 7y ago

> weight of keeping everything updated...monolith repo You need CD or this is an accident waiting to happen.

zo1 7y ago

A recent project I was exposed to has been struggling with Microservices and a multi-repo setup. Even with CI/CD and a lot of good tooling around their setup. The overhead introduced with having such a setup in a corporate environment that has not-so-well-though-out requirements and design is ridiculous. Keeping track of dependencies, arcane knowledge of inter-service dependency quirks being siloed and hidden, keeping individual services up to date, dealing with older services and their interaction with newer services till they "migrate" to newer tooling/common code, etc. Everyone then skirts around the fact that the problem could potentially be Microservices or a micro-repo setup. Instead, they throw process, sign-off, complicated promotion pipelines and just plain warm-bodies at the problem in an attempt to mitigate it. But the damage is done, velocity has slowed to a crawl and everyone is miserable, especially when having to explain the whole thing to newcomers.

pm90 7y ago
The best ideas can be implemented poorly. Software design principle that works for nimble Silicon Valley startups doesn’t work in your big corporate environment? Big surprise. When I worked at a pretty large software co, a team there were always adopting the latest techniques and tools but their deployment pipeline was an over architected disaster that nobody could reliably deploy. Reason? Their CI/CD system consisted of a person manually clicking around to build Jenkins Jobs. There will be other such silly nonsense (hopefully less extreme) in other corporate environments too.
deleted 7y ago
[deleted]

bakul 7y ago

IMHO a microservice should follow the unix philosophy of doing one thing well. But interfacing with other services is not as simple as unix pipes. In particular it is more of a request/response communication. Consequently, interfacing needs to be thought through carefully and made as simple as possible. They should evolve much more slowly than individual service code. While you may use (g)rpc or http at a lower level, your specific protocols will have many more constraints. Ideally you have codified assertions about their behavior in tests early on. Note that they are not all going to start/stop at exactly the same time and during development they may even crash so each microservice should survive such transitions. You may have two different versions of service code or even completely different implementations or two different version of the API in use at the same time. This can happen as different services evolve at a different rate. And you may want to transition to a new version in a piecemeal manner so as to not bring the whole service down. All these considerations complicate things. So ideally you have factored out these common tasks in shared library/packages. And ideally you write your code such that if necessary more than one service can be compiled into the same binary for performance reasons. In a monolith some things become easier since everything dies at once! But somethings become more complicated - such as supporting evolving code. And monolitha require more discipline to keep things modular. Over time this gets harder and harder. Lack of modularity means you have to understand a lot more code and when you evolve things, more code will have to change and there may be unforeseen side-effects. And scaling can become harder.

dangoor 7y ago

Our approach to services at Khan Academy is likely a bit different from most. We're sticking with a monorepo (the code for all services lives in one repository). We have a single go.mod file at the top of the repo, so all services use the _same versions_ of dependencies. We're still building out our deployment system to better support multiple services, but we're planning to redeploy all of the services when library code changes (which is something we're trying to minimize). All of this ensures that we don't have trouble with services lagging behind on critical updates.

solidasparagus 7y ago
I don't really understand moving to microservices if you aren't going to give the service teams the autonomy to make their own decisions and move at their own pace. Microservices always seemed to me to be more of an organizational strategy than a technical one - if you have services, but they can't operate independently, it feels a bit like you are just creating a distributed monolith.
- gtirloni 7y ago
  Choosing the dependencies' version is not all there's to decide. Teams have a lot of control of basically everything else.
  - andyroid 7y ago
    So, no choice of language and no choice of the libraries used. What’s “everything else” exactly? Sounds like missing out on the more interesting parts of a micro service architecture.
    - sangnoir 7y ago
      Yeah, I'll never work for a company[1] where service teams are free to choose any language; 2 or 3 options at most is fine, but more than that is a hard no. I'll have to read/work on that code sooner or later, and I have no time to be dealing with a hodgepodge of languages 1. In the 10-5000 employee range: tech giants are a different beast when it comes to team accountability.

axaxs 7y ago

Same. I tried microservices before and while it cleaned up the code base, I didn't quite like the results down the road. Some things go stale, you multiply ops * number of microservices, a change in one can mean a change in multiple others. I'm not against services in general, but not a fan of so called microservices.

kodablah 7y ago

I've found there's a happy middleground. You need medium-sized-services that still share code libraries. For many companies, this is often 7 or 8. The key is to combine like business units/features, not necessarily fragmenting at every visible code boundary. A deployed "service" can really just be several HTTP paths and/or gRPC services in one repo. You still get to keep decent separation of work, deployment, versioning, dev focus, etc with these medium sized services without sacrificing the benefits of larger, more centralized/shared reuse.

stult 7y ago
I’ve accidentally landed on an architecture like this and I’m actually pretty happy about it. It was driven by a desire to kill a large monolith slowly, by extracting key features into separate services. Sold to the customer as microservices because sexy fad of the moment. Our real motivation was we had way too much trouble recruiting in the monolith stack (.NET) and had a surplus of embedded C++, python, and JS engineers. Anyway, turns out our teams naturally self-organize around four or five domain+language clusters that effectively form separate services that are too large to really be micro, but too dissimilar to play nicely together in a monolith. Eg, python data science module wrapped in a Flask API providing physics calcs, legacy C# service providing simple data models via REST, JS react front end served from a separate node service, a weird C++/python hybrid used for an embedded device sim service, etc. It’s not what I planned as the tech lead/architect, but I think we organically reached what is really the best approach for our team. Definitely an element of Conway’s law in action, but in a good way. We are making the most of our organizational structure rather than fighting against it. Would this scale to google levels? No, probably not. But we don’t need it to, and it’s incredibly unlikely we ever would given our specific business.

bborud 7y ago

Start with a monolith that has clear internal APIs that are designed so they can later be made into network APIs. This gives you the development speed of a monolith while maintaining an options for the future. When you do break things out into separate services: try to make as few of them as possible and maintain the ability to build as a monolith. Forget everything you have heard about micro services. Most of it is bullshit from people who don’t actually think for themselves.

bcrosby95 7y ago
This. If you can't design a well segmented monolith, you can't design a well segmented system of microservices either. The microservices will just be buggier and much harder to fix after the fact.
- bborud 7y ago
  Design is best evolved. If you can get a lot of that done while you still are able to run a well structured system as a monolith you can save a lot of time. It is cheaper to change an interface in Java/Golang than a REST API.
stult 7y ago
Could not agree more. I recently led a refactor of a monolithic .NET MVC app and took this exact approach. We made all of the controllers thin, with almost no logic at all beyond specifying the route and dependency injection. Then redirected the request to a “service”, which originally was just a reworked combination of the old controller/model logic hidden behind a common service interface. Then, slowly we replaced the C# services with microservices. So we went from ball of spaghetti to monolithic service oriented architecture lite to actual microservices with the monolith converted into an API gateway. If we didn’t have independent motives for going to microservices, sticking to the clean and well organized internal APIs of the refactored monolith would have been totally fine.

twblalock 7y ago

In my experience, the biggest benefit of microservices is decoupling teams. Developer productivity is very hard to maintain in a monolithic app as the number of developers increases and the legacy code piles up. Breaking up services and giving each dev team control over their own codebases enables them to develop their own products at their own pace. If you only have one dev team, microservices are a lot less attractive. However, there are still some benefits, such as being able to refactor parts of your codebase in isolation (including perhaps rewriting them in different languages), and the ability to individually adjust the runtime scale of different parts of your codebase.

whatsmyusername 7y ago
This. The problem with decoupling services is usually there end up being a couple services that are critical but not sexy. No one wants to touch them so they sit around unmaintained until an unrelated change or unpatched security issue comes around. Suddenly you've got a big problem with a mystery codebase.
- viraptor 7y ago
  That sounds very familiar, but I'm not sure this is something that can be blamed on decoupling itself. An unpopular module is going to need as much attention as a micro service from code point of view. For upgrades / patching, there would be a company-wide process around it that doesn't care that much how the code is organised.
  - whatsmyusername 7y ago
    'company wide' process is either, 'squeaky wheel gets the grease,' or, 'no one even knows this exists,' 100% of the time in my experience. This is from going from 10k+ to 150 to 5k+ to 50 people.
    - viraptor 7y ago
      I wrote company-wide, but it's not the case everywhere. At some scale you'll want department, or even project-wide process. But the policy should be fairly common - who owns it, what's the response time, how to escalate urgent things, etc.

scottrogowski 7y ago

A company I am affiliated with made a decision to rewrite their code in microservices-oriented architecture thinking it would only take one year. Now we're 7 years into the transition and starting to come up against some hard deadlines that threaten revenue streams. It seems obvious to everyone except the leadership and the architects that this has been an unmitigated disaster. Other comments on this thread seem to indicate that many have had similar experiences. For those who are curious, here is a classic article on why rewriting code from scratch is a bad idea: https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/ https://www.joelonsoftware.com/2000/04/06/things-you-should-.... For a more in-depth analysis on the unforeseen challenges of microservices in particular, I would encourage a lot of careful research into how other companies have tried and failed at this. In particular, I might look at Uber's ongoing difficulties. All I have to say to the Khan Academy engineers is to buckle up because frankly, moving from Python 2->3 is not that hard and you have no idea what you are getting yourself into.

tmpfile 7y ago
this. 100% this.
andy_ppp 7y ago
Yes, the micro services + Golang vanity project because you think you’re Google. I really don’t think people understand error states in distributed systems very well, putting possible network partitions everywhere is not a great idea. I would strongly suggest trying a Golang monolith first and seeing if there are one or two heavily used services that need splitting off. Also monorepo. Always.
- StavrosK 7y ago
  Microservices have always seemed dangerous to me. "Bugs thrive in the seams between libraries, so let's put more and deeper seams in!" Microservices have an immense cost, and you have to make sure they're worth it. Many teams years ago found it a nice pattern and implemented it because why not, and now we're at the "oops this isn't actually amazing" part of the cycle.

deleted 7y ago

[deleted]

hendry 7y ago

I way prefer monolith compared to dabbling in microservices here: https://natalian.org/2019/05/16/Microservices_pitfalls/ https://natalian.org/2019/05/16/Microservices_pitfalls/

jordanbeiber 7y ago

I’ve been through a transition with about 60 devs that went DDD plus microservices. A few monotliths ended up as a couple of hundred services, and looking back I feel we got basically all positives. What other people say about scaling teams is true, but I have a few other points as well: - personally, I spent 6 months writing tooling for service lifecycle management and setting strict conventions. This was before the microservices decision and I was recruited to do “devops” which gave me a lot of head room. :p - when talks about microservices surfaced I had to fight for several months to get the lead devs on-board with tooling and conventions. From the infra/cm/systems management side we’re used to manage many thousands of “configuration items” - devs are usually not, and many underestimate the value of proper automated lifecycle management. - once everyone was on-board we all used a common language. Big win. - I could form a “devops” team to help develop the tooling further and the infra platform as well. - almost all teams worked in mobs - amongst other things it really helped with the ownership part, something that’s absolutely crucial. Well defined domains and accountable mob teams, just awesome! - quality rose by a mile. Small commits and somewhat robust tooling under the eyes of a mob. - graph the pains. If outdated versions are a problem - put it on a graph, green, yellow, red. Show services in relation to other services - the application is the sum of all connected services. I could go on, but the post is getting long! :)

gtirloni 7y ago
Thanks for sharing, that's pretty exciting. I think most people don't realize the work they have to put in to make this work (and reap the benefits).
- jordanbeiber 7y ago
  Thanks for reading. The more i write and talk about it, the more I realize that it is about ”externalities”, so to speak. A microservice is just code - one small piece that does one, tightly defined thing. We’ve been doing code always and smaller pieces of is easier to deal with and reason about. The structure around keeping 100s or 1000s of moving pieces in concert is where a lot of the work is shifted. It takes teamwork as well as a common vision and language. The above sentence tangents “culture”.

josteink 7y ago

> I would love hear some thoughts from others that made the move, especially anyone that decided to move back to a monolith repo. We had a big monolith where I work. We’ve been slowly, but surely isolating parts of the monolith as separate deliverables, extracted into their own repos. But only when appropriate, and not as a forced exercise. The remaining “monolith” is still pretty big, but it does (mostly) represent one logical deliverable, so effort to split it up has some what stalled. There’s been small points of friction, but nothing near as painful as we used to have it. No way we’re going back. So everything in moderation. Micro services architecture is a tool. Use it when it’s the right one.