4 ms·

Microservices can solve a lot of organizational problems: 1. Fewer merge conflicts, easier branching/merging strategy. If you have 200 people checking into a m

Microservices can solve a lot of organizational problems: 1. Fewer merge conflicts, easier branching/merging strategy. If you have 200 people checking into a monolith then it is easy to start stomping all over each other. 2. Large changes(say upgrading to a new major version of a library for security reasons) can result in the code base being in a state of conflict hell, or just frozen, while work gets done. With microservices you can upgrade them one at a time. 3. The build/test/deploy pipeline for any given team is shorter. Large code bases take a long time to build and run through tests. If instead everyone has to have well defined interfaces to their microservices then it is possible to test the interface points of a given microservice that is being committed to. 4. Serious screw-ups in one part of the code don't impact code in other places. If someone starts leaking memory in one microservice, then one part of the system has degraded functionality. Worst case it is a critical microservice and your system goes down. Best case it isn't on the golden path for users and it isn't a major outage. 5. It makes you think about deployment and configuration up front. I have seen major systems (billions of dollars, systems you've used in the past) where only tribal knowledge of how to "get it running" existed. There was no formal deployment plan, and deployments could take days. Microservices force organizations to solve deployment up front.

bszupnick 6y ago
I've worked on a large-scale micro-service architecture and I had a great time! I don't "pick a side" of this almost idealistic argument, but I do have some pushback for you: 1) You're right that there aren't merge conflicts, but you swap out merge conflicts with interface changes. Meaning let's say programmer A is working on Microservice B that consumes from Microservice C and she changes the "what I expect from Microservice C" interface. Who changes Microservice C? In THEORY it could be whoever "owns" it, or Programmer A, but in practice sometimes it falls between the cracks. 2/4) We ended up actually writing our own Python packages that did end up being shared amongst some of the microservices. If you want to abide by DRY (not juts as dogma but you just don't want to implement the same thing twice), you do need some shared code. Maybe that "breaks" the microservice "model" and we didn't do it "right"? Maybe. But with what we did, we still did have "internal Python package" updates intended for one Microservice that effected another This is all from my personal experience so it could be that we just didn't follow best practices or fell into some traps.
- jeffbee 6y ago
  You don't change interfaces. It is that simple. There's not a paradox here. To change your RPC protocol in a distributed system, you may add optional fields to the request but you must not add required fields nor change the semantic meaning of any field. You can add new services and methods but you must not disable one that anyone calls.
  - marcus_holmes 6y ago
    Which is surely an argument for "never start a project using microservices"? Who knows exactly what their interfaces are going to be, across their entire project, on day 1, and gets that right every time?
    - com2kid 6y ago
      > Who knows exactly what their interfaces are going to be, across their entire project, on day 1, and gets that right every time? Same problem exists for binary interfaces, or source code interfaces. Or for a client system that talks to a cloud service. Or for a cloud service that exposes APIs for other cloud services to call. Technologies exist to version APIs. It is sad that JSON is the standard interchange format, since it is pretty miserable at this entire problem.
      - marcus_holmes 6y ago
        I agree completely, but all this is exacerbated if you have to make your interfaces immutable from the first version
        jeffbee 6y ago
        They aren't immutable. You can safely add fields and you can add methods or entire services. It is easy to change microservice interfaces, you just do it in a particular way.
    - jeffbee 6y ago
marcus_holmes 6y ago
Sorry, but none of this is a feature of microservices: 1. You just move your conflicts outside your git repo (or other VCS). The reason you get conflicts is because multiple people are changing the same bit of code. If that's happening over multiple repos/services, it still happens, but now it's harder to detect and deal with. 2. Again, if your change causes an interface change, then that's going to be easier to deal with in a monolith than in a microservice. 3. This is only true if you're not running integration tests across multiple microservices. Which you need to be. 4. This is plain untrue. If one of your microservices is crashing all the time, then the other services that depend on it are also fubar'd. The only different is that you only detect this at run-time. 5. This is a management problem, not an architecture problem. And the fact that several successful monolith deployments have scaled to the point where I'd have used them without needing formal deployment plans is a positive point for monoliths, surely?
- com2kid 6y ago
  > 1. You just move your conflicts outside your git repo (or other VCS). The reason you get conflicts is because multiple people are changing the same bit of code. If that's happening over multiple repos/services, it still happens, but now it's harder to detect and deal with. Integration tests cover this. > 2. Again, if your change causes an interface change, then that's going to be easier to deal with in a monolith than in a microservice. This point was more about upgrading dependencies. A lot of code bases never upgrade dependencies because it is too much work. (I've seen some really old versions of OpenSSL in code repos...). In a monolith the task is "we have to freeze the entire code base and upgrade this library in hundreds of places". In a microservice based system, the work is "farm out the work to a dozen teams and add upgrading the dependency to their sprint backlog." That is a huge difference. The first time I saw this in action I was blown away by how incredibly different this was vs what I'd experienced before. It means keeping up to date with security changes, or just not letting dependencies get too stale, is no longer a huge burden on a code base. > 3. This is only true if you're not running integration tests across multiple microservices. Which you need to be. Yup, of course you need to be! But monoliths also need tests, and ideally the monolith is broken up into black box modules that don't know each other's insides and the code is tested accordingly. Microservices force that test approach, but ideally there is not much difference. (Placing the entire system under test may be easier/harder, depending on business logic being tested) Also I have experience where build/test pipelines can take hours. With microservices you can only run the integration tests that directly impact that microservice. In theory with a monolith you can use tools that intelligently detect what code was changed and only run a subset of tests accordingly, but I've not seen those tools used, only discussed on HN. > 4. This is plain untrue. If one of your microservices is crashing all the time, then the other services that depend on it are also fubar'd. The only different is that you only detect this at run-time. It is an edge case benefit, but I'd argue it is still a benefit. Degraded performance in one use case is better than an entire service going down. There are differences between "Everything Xbox is down", "Xbox Live match making is down" and "You can't redeem gift cards". (No comment on actual Xbox Live functionality or stability!) > 5. This is a management problem, not an architecture problem. And the fact that several successful monolith deployments have scaled to the point where I'd have used them without needing formal deployment plans is a positive point for monoliths, surely? Sure, but most things I mentioned can be solved by management! Better test strategies, giving enough time to planning to avoid brittle interfaces, etc etc. To be clear, outside of running my own startup (which was mostly serverless), the backends I've seen or worked on have been part of large systems. For a team of 5? Microservices are probably the wrong solution.

bszupnick 6y ago

I've worked on a large-scale micro-service architecture and I had a great time! I don't "pick a side" of this almost idealistic argument, but I do have some pushback for you: 1) You're right that there aren't merge conflicts, but you swap out merge conflicts with interface changes. Meaning let's say programmer A is working on Microservice B that consumes from Microservice C and she changes the "what I expect from Microservice C" interface. Who changes Microservice C? In THEORY it could be whoever "owns" it, or Programmer A, but in practice sometimes it falls between the cracks. 2/4) We ended up actually writing our own Python packages that did end up being shared amongst some of the microservices. If you want to abide by DRY (not juts as dogma but you just don't want to implement the same thing twice), you do need some shared code. Maybe that "breaks" the microservice "model" and we didn't do it "right"? Maybe. But with what we did, we still did have "internal Python package" updates intended for one Microservice that effected another This is all from my personal experience so it could be that we just didn't follow best practices or fell into some traps.

jeffbee 6y ago
You don't change interfaces. It is that simple. There's not a paradox here. To change your RPC protocol in a distributed system, you may add optional fields to the request but you must not add required fields nor change the semantic meaning of any field. You can add new services and methods but you must not disable one that anyone calls.
- marcus_holmes 6y ago
  Which is surely an argument for "never start a project using microservices"? Who knows exactly what their interfaces are going to be, across their entire project, on day 1, and gets that right every time?
  - com2kid 6y ago
    > Who knows exactly what their interfaces are going to be, across their entire project, on day 1, and gets that right every time? Same problem exists for binary interfaces, or source code interfaces. Or for a client system that talks to a cloud service. Or for a cloud service that exposes APIs for other cloud services to call. Technologies exist to version APIs. It is sad that JSON is the standard interchange format, since it is pretty miserable at this entire problem.
    - marcus_holmes 6y ago
      I agree completely, but all this is exacerbated if you have to make your interfaces immutable from the first version
      - jeffbee 6y ago
        They aren't immutable. You can safely add fields and you can add methods or entire services. It is easy to change microservice interfaces, you just do it in a particular way.
  - jeffbee 6y ago

marcus_holmes 6y ago

Sorry, but none of this is a feature of microservices: 1. You just move your conflicts outside your git repo (or other VCS). The reason you get conflicts is because multiple people are changing the same bit of code. If that's happening over multiple repos/services, it still happens, but now it's harder to detect and deal with. 2. Again, if your change causes an interface change, then that's going to be easier to deal with in a monolith than in a microservice. 3. This is only true if you're not running integration tests across multiple microservices. Which you need to be. 4. This is plain untrue. If one of your microservices is crashing all the time, then the other services that depend on it are also fubar'd. The only different is that you only detect this at run-time. 5. This is a management problem, not an architecture problem. And the fact that several successful monolith deployments have scaled to the point where I'd have used them without needing formal deployment plans is a positive point for monoliths, surely?

com2kid 6y ago
> 1. You just move your conflicts outside your git repo (or other VCS). The reason you get conflicts is because multiple people are changing the same bit of code. If that's happening over multiple repos/services, it still happens, but now it's harder to detect and deal with. Integration tests cover this. > 2. Again, if your change causes an interface change, then that's going to be easier to deal with in a monolith than in a microservice. This point was more about upgrading dependencies. A lot of code bases never upgrade dependencies because it is too much work. (I've seen some really old versions of OpenSSL in code repos...). In a monolith the task is "we have to freeze the entire code base and upgrade this library in hundreds of places". In a microservice based system, the work is "farm out the work to a dozen teams and add upgrading the dependency to their sprint backlog." That is a huge difference. The first time I saw this in action I was blown away by how incredibly different this was vs what I'd experienced before. It means keeping up to date with security changes, or just not letting dependencies get too stale, is no longer a huge burden on a code base. > 3. This is only true if you're not running integration tests across multiple microservices. Which you need to be. Yup, of course you need to be! But monoliths also need tests, and ideally the monolith is broken up into black box modules that don't know each other's insides and the code is tested accordingly. Microservices force that test approach, but ideally there is not much difference. (Placing the entire system under test may be easier/harder, depending on business logic being tested) Also I have experience where build/test pipelines can take hours. With microservices you can only run the integration tests that directly impact that microservice. In theory with a monolith you can use tools that intelligently detect what code was changed and only run a subset of tests accordingly, but I've not seen those tools used, only discussed on HN. > 4. This is plain untrue. If one of your microservices is crashing all the time, then the other services that depend on it are also fubar'd. The only different is that you only detect this at run-time. It is an edge case benefit, but I'd argue it is still a benefit. Degraded performance in one use case is better than an entire service going down. There are differences between "Everything Xbox is down", "Xbox Live match making is down" and "You can't redeem gift cards". (No comment on actual Xbox Live functionality or stability!) > 5. This is a management problem, not an architecture problem. And the fact that several successful monolith deployments have scaled to the point where I'd have used them without needing formal deployment plans is a positive point for monoliths, surely? Sure, but most things I mentioned can be solved by management! Better test strategies, giving enough time to planning to avoid brittle interfaces, etc etc. To be clear, outside of running my own startup (which was mostly serverless), the backends I've seen or worked on have been part of large systems. For a team of 5? Microservices are probably the wrong solution.