11 ms·

We’ve done exactly this - turned a team of 15 engineers from managing one giant monolith to two teams managing about 10 or so microservices (docker + kubernetes

We’ve done exactly this - turned a team of 15 engineers from managing one giant monolith to two teams managing about 10 or so microservices (docker + kubernetes, OpenAPI + light4j framework). Even though we are in the early stages of redesign, I’m already seeing some drawbacks and challenges that just didn’t exist before: - Performance. Each of the services talks to the other service via well-defined JSON interface (OpenAPI/Swagger yaml definitions). This sounds good in theory, but parsing JSON and then serializing it N times has a real performance cost. In a giant “monolith” (in the Java world) EJB talked to each other, which despite being java-only (in practice), was relatively fast, and could work across web app containers. In hindsight, it was probably a bad decision to JSON-ize all the things (maybe another protocol?) - Management of 10-ish repositories and build jobs. We have Jenkins for our semi-automatic CI. We also have our microservices in a hierarchy, all depending on a common parent microservice. So naturally, branching, building and testing across all these different microservices is difficult. Imagine having to roll back a commit, then having to find the equivalent commit in the two other parent services, then rolling back the horizontal services to the equivalent commit, some with different commit hooks tied to different JIRA boards. Not fun. - Authentication/Authorization also becomes challenging since every microservice needs to be auth-aware. As I said we are still early in this, so it is hard to say if we reduced our footprint/increased productivity in a measurable way, but at least I can identify the pitfalls at this point.

merb 8y ago
> So naturally, branching, building and testing across all these different microservices is difficult. Imagine having to roll back a commit, then having to find the equivalent commit in the two other parent services, then rolling back the horizontal services to the equivalent commit that should not happen. if it does you don't have a microservice architecture, you have a spaghetti service architecture.
- tabtab 8y ago
  How does one know if X is the wrong solution or if X is the right solution but the shop is "doing X wrong"? This also applies to monoliths: maybe they can scale, but one is doing them wrong. Changing from doing monoliths wrong to doing microservices wrong is obviously not progress. The same issue appeared when OOP was fairly new: people started using it heavily and ended up making messes. They were then told that they were "doing it wrong". OOP was initially sold as magic reusable Lego blocks that automatically create nice modularity. There was even an OOP magazine cover showing just that: a coder using magic Legos that farted kaleidoscopic glitter. Microservices is making similar promises. It took a while to learn where and how to use OOP and also when not to: it sucks at some things.
  - freedomben 8y ago
    If you want to learn "the right way" I highly recommend "Building Microservices: Designing Fine-Grained Systems" by Sam Newman[1] [1] https://smile.amazon.com/Building-Microservices-Designing-Fine-Grained-Systems/dp/1491950358/ref=sr_1_3 https://smile.amazon.com/Building-Microservices-Designing-Fi...
  - notyourday 8y ago
    I have posted about this before. OP is describing nano-services, not micro-services. Nano-service is micro-service that provides leftpad via JSON api. You have an auth.yourapp.com and api.yourapp.com and maybe tracer.yourapp.com and those three things are not a single app that behaves like auth, api or tracer depending on setting of a NODE_ENV variable? If so, you have micro services.
  - taneq 8y ago
    Simple: If X is a technology I don't like, and it's not working for you, then it's the wrong solution. If X is a technology I don't like, and it is working for you, then you simply haven't scaled enough to understand its limitations. If X is a technology I like, but it's not working for you, then your shop is "doing X wrong". If X is a technology I like, and it's working for you, then it's the right solution and we're both very clever.
    - staticassertion 8y ago
      edit: See below
      - taneq 8y ago
        Uh, my post was snark towards engineers' general tendency to champion and defend their own favourite architecture over competing approaches rather than focusing on the most suitable architecture for the circumstance, in response to this question: > How does one know if X is the wrong solution or if X is the right solution but the shop is "doing X wrong"? Edit: And a civil conversation ensued. :)
- mirkules 8y ago
  Heh, I like that “Spaghetti Microservices”. You are right. It should not happen. It is difficult to see these pitfalls when unwinding an unwieldy monolith, and, as an organization all you’ve ever done are unwieldy monoliths, that have a gazillion dependencies, interfaces and factories. We learned from it, and we move on - hopefully, it serves as a warning to others.
  - tcbasche 8y ago
    I've heard the term 'Distributed Monolith'
  - james_s_tayler 8y ago
    MicroPasta
    - cestith 8y ago
      Tangled angelhair.
  - c3534l 8y ago
    I think we should call it angel-hair pasta code. You know, because it's very small spaghetti.
  - smnplk 8y ago
    No, it's ravioli oriented architecture
- dragonwriter 8y ago
  > that should not happen. if it does you don't have a microservice architecture, you have a spaghetti service architecture. A "service" is not defined principally by a code repository or communications-channel boundary (though it should have the latter and may have the former), but by a coupling boundary. OTOH, maintaining a coupling boundary can have non-obvious pitfalls; e.g., supported message format versions can become a source of coupling--if you roll back a service to the version that can't send message format v3, but a consumer of the messages requires v3 and no longer supports consuming v2, then you have a problem.
  - merb 8y ago
    you should never actually remove an api without a deprecation process. basically a message format is part of your api. actually in a microservice world your interfaces/api should be as stable as possible. if something gets changed, it needs to go trough a deprecation, that means not removing it until a certain amount of time.
    - NicoJuicy 8y ago
      The accept header could fix this easily if no third parties are involved
    - NicoJuicy 8y ago
      The accept header could fix this easily if no third parties are involved and if it's a rest like API protocol
      - dragonwriter 8y ago
        The accept header is a mechanism for communicating formats you support; it doesn't do anything to address the problem of managing change to supported versions, which is a dev process issue, not a technical issue.
        geezerjay 8y ago
        > The accept header is a mechanism for communicating formats you support; it doesn't do anything to address the problem of managing change to supported versions That assertion is not true. Media type API versioning is a well established technique.
- sequoia 8y ago
  It sounds like you're paraphrasing the great Steven Jobs: "You're holding it wrong" Be that as it may, I believe mirkules's issue is not an uncommon one. Perhaps saying "building a microservice architecture 'the right way' is a complex and subtle challenge" would capture a bit of what both of you are saying. Something being complex and therefor easy to mess up does not mean it's a great system and the users are dumb, especially if there are other (less complicated, less easy to mess up) ways to complete the task.
  - geezerjay 8y ago
    > Perhaps saying "building a microservice architecture 'the right way' is a complex and subtle challenge" would capture a bit of what both of you are saying. Supporting API versioning is not a complex or subtle challenge. It's actually a very basic requirement of a microservice architecture. It's like having to test null pointers: it requires additional work but it's still a very basic requirement if you don't want your app to blow up in your face.
- jeffasinger 8y ago
  I think a bigger thing here, is that each deploy should have a REALLY good reason to not be backwards compatible for some pretty long time period. If that requirement is painful for you, then you probably have two pretty tightly coupled services.
- balp 8y ago
  To handle deploys and especially rollbacks you need a working CI, or rather CD chain. Where eveything is automated. In there are dependencied in your architecture, all of that needs to be part of the same deploy so you can redo the deploy before that one. With a monolith, you get a simple deploy of all dependencies, as they are baked into one thing. There are down sides of everything baked into one thing. Having all your "micoro" servies deplyed as one package, would make the dependency you have act as before the monolith change. Seeing this kinds of dependency imho even in a monolith are an architechture problem, that in the monolith shows up at code changed and fixes can't be handles by localized fixes, but changed have to go into large parts of the code base.
jrockway 8y ago
Protocol buffers exist to solve two problems you list, parsing overhead and backwards compatibility.
- jcims 8y ago
  I always wondered why ASN.1 never really seemed to take off. Tooling maybe?
  - acdha 8y ago
    Yes - both the protocol and tooling are expensive to support, even by 90s standards. Unless you have to use it for compatibility with a service you can’t fix it’d be much better to start with a better implementation of the concept.
  - throwaway2048 8y ago
    ASN.1 has a number of problems with equivalent parsings (very bad in a security context, has been the source of a number of TLS vulnerabilities), as well as the fact that even discovering if two ASN.1 parsers will give the same result is undecidable.
eweise 8y ago
We use grpc instead of rest for internal synchronous communication but we've also found that by using event pub/sub between services, that there are not many uses cases where we have direct calls between services. We used to have a parent maven pom and common libraries but got rid of most of that because it caused too much coupling. Now we create smaller more focused common libraries and favor copy/paste code over reuse to reduce coupling. We also moved a lot of the cross cutting concerns into Envoy so that the services can focus on business functionality.
- mirkules 8y ago
  gRPC and Envoy are exactly the things we are exploring now, although copy/paste would never fly in our org.
  - praneshp 8y ago
    I know copy paste is looked down upon, but it's even suggested as a good practice (a little bit) in golang. A little copying is better than a little dependency. https://go-proverbs.github.io/ https://go-proverbs.github.io/
- lucidone 8y ago
  How do you handle cases where your client is awaiting a response with a decoupled pub/sub backend? E.g a user creates an account and the client needs to know their user id. Would that user object be the responsibility of one service, or written to many tables in the system under different services, or...?
  - RhodesianHunter 8y ago
    For one, you could use something like snowflake IDs so that whatever server receives the user data first can generate and return an id for that user before tossing the data on a queue to be processed.
    - lucidone 8y ago
gregorygoc 8y ago
You must be doing something wrong. In a company I work for, we clearly have separated microservices by bounded context, thus making them completely decoupled.
- jasonpeacock 8y ago
  This is the key - decoupled. When you have to rollback commits across multiple service, they are not decoupled, and you're doing something wrong. Each service should be fully independent, able to be be deployed & rolled-back w/o other services changing. If you're making API changes, then you have to start talking about API versioning and supporting multiple versions of an API while clients migrate, etc.
  - vogre 8y ago
    >then you have to start talking about API versioning and supporting multiple versions of an API while clients migrate, etc. Which adds some more complexity that just does not exist in monolythic architecture
    - icebraining 8y ago
      Sure it does. Once you have a large monolith, the same coordination problems hit. Just ask the Linux kernel developers: https://lwn.net/Articles/769365/ https://lwn.net/Articles/769365/
nevir 8y ago
Consider larger services than a typical microservice. For the same reason monoliths tend to split when the organization grows, it is often more manageable to have a small number of services per team (ideally 1, or less). It's ok if a service owns more than one type of entity. It's less good if a service owns more than one part of your businesses' domain, however
- gav 8y ago
  > Consider larger services than a typical microservice. People seem to forget that there’s a continuum between monolith and microservices, it’s not one or the other. Multiple monoliths, “medium”-services, monolith plus microservices, and so on are perfectly workable options that can help transition to microservices (if you ever need to get there at all).
- jeffasinger 8y ago
  That's fine, but reorganizations happen, teams can grow, and there is an advantage to having things be separate services in cases like this. Definitely don't just stuff unrelated stuff into a service since a team that normally deals with that service is now working on unrelated stuff. If the unrelated stuff takes off, you now have two teams untangling your monolithic service. That said, I'm a big fan of medium sized services, the kind of thing that might handle 10 or 20 different entities.
jimbokun 8y ago
"turned a team of 15 engineers from managing one giant monolith to two teams managing about 10 or so microservices" Knowing absolutely nothing about your product, this sounds like a bad way to split up your monolith. Probably something like 5 teams of 3 each managing 1 microservice would be a better way to split things up. That way each team is responsible for defining their service's API, and testing and validating the API works correctly including performance requirements. This structure makes it much less likely services will change in a tightly coupled way. Also, each service team must make sure new functionality does not break existing APIs. Which all make it less likely to have to roll back multiple commits across multiple projects. The performance issues you cite, also seem to indicate you have too many services, because you are crossing service boundaries often, with the associated serialization and deserialization costs. So each service probably should be doing more work per call. "all depending on a common parent microservice" This makes you microservices more like a monolith all over again, because a change in the parent can break something in the child, or prevent the child from changing independently of the parent. Shared libraries I think are a better approach. "Authentication/Authorization also becomes challenging since every microservice needs to be auth-aware." Yes, this is a pain. Because security concerns are so important, it is going to add significant overhead to every service to make sure you get it right, no matter what approach you use.
- james_s_tayler 8y ago
  Probably more like do your best to reason about the boundaries of each bounded context and pick an appropriate number of services based off that analysis? Surely splitting up your application along arbitrary lines based on advice of an internet stranger whose never seen the application and doesn't know the product/business domain just isn't sound way of approaching the problem.
  - toolslive 8y ago
    in the end, your architecture will reflect your organigram. (IIRC it's Conway's law?)
    - 8y ago
DelightOne 8y ago
> Management of 10-ish repositories and build jobs. Does each micro service have to live in it's own repository? Especially with a common library everyone uses?
- statictype 8y ago
  I think having a common library that does anything specific to your product is something of an anti-pattern in micro services. Its not really a micro service - its a distributed monolith
  - TheCoelacanth 8y ago
    There are always going to be cross-cutting concerns. You don't want to have every microservice implement it's own authentication, auditing, etc. from the ground up.
xvector 8y ago
Perhaps a few ideas: - Performance: use gRPC/protobuf instead of HTTP/OpenAPI, really not much of a reason to use HTTP/OpenAPI for internal endpoints these days - Repo Management: No one is stopping you from using a monorepo but yourselves :)
- mirkules 8y ago
  Even just defining what “internal communication” means is difficult. We definitely suffer from the what-if syndrome- “what if some day we want to expose this service to a client?” Our product is a collection of large systems used by many customers with very different requirements - and so we often fall into this configurability trap: “make everything super configurable so that we don’t have to rebuild, and let integration teams customize it”
  - xvector 8y ago
    Ah. In that case, you can expose your gRPC endpoints as traditional JSON/HTTP ones with gRPC-Gateway, which supports generating OpenAPI documentation too! Best of both worlds.
  - tchaffee 8y ago
    Sounds like you're suffering from not using YNGNI enough: you're not gonna need it. Build what you need now. When that changes, you can change what you built. That was the original intention of Agile methodologies and BDD or TDD. When the tests pass, you're done.
zebraflask 8y ago
These are excellent points. I was just having a conversation yesterday about system design and how there seems to be a tipping point after which the transactional/organizational costs of segmenting services outweigh the benefits. My unscientific impression is that some of the organizational costs - just keeping the teams coordinated and on the same page - can become even more "expensive" than the technical costs.
clon 8y ago
Don't forget: - Logging. All messages pertaining to a request (or task) should have a unique ID across the entire fleet of services in order to follow the trail while debugging.
- tyingq 8y ago
  "correlation id" is probably the best thing to poke into Google for guidance on it.
  - dustindiamond 8y ago
    I would recommend investigating APM, opentracing and Uber’s jaeger project.
flukus 8y ago
> This sounds good in theory, but parsing JSON and then serializing it N times has a real performance cost. It's not just the serialization cost but latency (https://gist.github.com/jboner/2841832 https://gist.github.com/jboner/2841832) as well, every step of the process adds latency, from accessing the object graph, serializing it, sending it to another process and/or over the network, then building up the object graph again. The fashion in .net apps used to be to separate service layers from web front ends and slap an SOA (the previous name for micro-services) label on it. I experimented with moving the service to in process and got an instant 3x wall clock improvement on every single page load, we were pissing away 2/3rds of our performance and getting nothing of value from it. And this was int the best case scenario, a reasonably optimized app with binary serialization and only a single boundary crossing per user web request. Other worse apps I've worked on since had the same anti-pattern but would cross the service boundary dozens/hundreds/thousands of times and very simple pages would take several seconds to load. It's enterprise scale n+1. If you want to share code like this then make a dll and install it on every machine necessary, you've got to define a strict API either way.
Rapzid 8y ago
JSON parsing IS expensive. Way more expensive than many people realize. There is actually an almost-underground JSON parser SCENE in the .Net ecosystem where people develop new parser and try to squeeze the maximum performance using all sorts of tricks: https://github.com/neuecc/Utf8Json https://github.com/neuecc/Utf8Json . Here there is discussion of needing JSON support in .Net core that's faster than Json.Net: https://github.com/dotnet/announcements/issues/90 https://github.com/dotnet/announcements/issues/90 . And people say web applications are never CPU bound :)
- NicoJuicy 8y ago
  gRCP is a little bit faster, Google created protobuff which should be easier to migrate to than gRPC, but the protocol is unreadable ( binary)... JSON has its advantages, I prefer a feature flag when I need performance ( protobuff vs Json), http headers do the rest
  - stouset 8y ago
    gRPC is built on top of protobuf. It’s literally just RPC with protobufs.
- 0815test 8y ago
  > JSON parsing IS expensive Well, maybe https://news.ycombinator.com/item?id=19214387 https://news.ycombinator.com/item?id=19214387
- tekknik 8y ago
  one could argue that if serialization/deserialization is the majority of what your api is spending time on then you’ve got distributed monoliths and shouldn’t be making the call in the first place. also ensuring you have the fastest json parser before having a real problem is premature optimization
tanilama 8y ago
Why so many microservices? Monolith has huge advantage when maybe your code is like 100k lines or below: 1. Easy cross module unit testing/integration testing, thus sharing components is just easier. 2. Single deployment process 3. CR visibility automatically promotes to all parties of interests, assuming the CR process is working as desired. 4. Also, just a personal preference, easier IDE code suggestion. If you went through json serializing/de-serializing across module boundary, type inference/cohesion is just out-of-reach. And it is not like monolith doesn't have separation of concern at all. After all, monolith can have modules, and submodules. Start abstracting using file system API, but grouping relevant stuff into folders, before put them into different packages. After all, once diverging, it is really hard to go back. Unless you have a giant team and more than enough engineers to spare for devops. Micrservices can be considered as a organizational premature optimization.
badloginagain 8y ago
> We also have our microservices in a hierarchy, all depending on a common parent microservice. Can you explain this a bit more? I thought the point was to have each service be as atomic as possible, so that a change to one service does not significantly impact other services in terms of rollbacks/etc. If I'm wrong here let me know, our company is still early days of figuring out how to get out of the problems presented by monolith (or in our case, mega-monolith).
mcguire 8y ago
I'm going to go out on a limb here and suggest that parsing (and serializing) JSON is unlikely to be the actual problem, performance-wise. (Although "OpenAPI/Swagger" doesn't fill me with enthusiasm.) More likely, I suspect, is that either you are shipping way too much data around, you have too much synchrony, or some other problem is being being hidden in the distribution. (I once dealt with an ESB service that took 2.5 seconds to convert an auth token from one format to another. I parallelized the requests, and the time to load a page went from 10 sec to <3; then I yanked the service's code into our app and that dropped to milliseconds.) Performance problems in large distributed systems are a pain to diagnose and the tools are horrible.
- therealdrag0 8y ago
  I've been using NewRelic and it's been wondrous around illuminating performance problems.
philwelch 8y ago
I don't think it's necessarily the best idea to immediately have almost as many services as you have engineers. There are usually more gradual and logical ways to split things up.
bunderbunder 8y ago
The first thing I start trying to convince people to ditch on any internal service is JSON. There are only two times when JSON is a particularly good choice: when it's important for the messages themselves to be human-readable, or when it's important to be able to consume it from JavaScript without using a library. Any other time, something like protocol buffers is going to give you lower latency, lower bandwidth requirements, lower CPU costs, lower development effort, less need for maintaining documentation, and better standardization. If you ditch the HTTP stuff while you're at it, you can also handily circumvent all the ambiguities and inter-developer holy wars that are all but inherent to the process of taking your service's semantics, whatever they are, and trying to shoehorn them into a 30-year-old protocol that was really only meant to be used for transferring hypermedia documents. Instead you get to design your own protocol that meets your own needs. Which, if you're already building on top of something like protobuf, will probably end up being a much simpler and easier-to-use protocol than HTTP.
- CoolGuySteve 8y ago
  Man if you think protobuf is fast, just wait until you try C pod structs with type-length headers. They are literally infinitely faster to encode/decode than protobuf. They even have the same obnoxious append-only extensibility of protobuf if that’s what really gets your jimmies firing.
  - netmonk 8y ago
    asn1 for the win
    - carapace 8y ago
      It's a shame you're catching downvotes for mentioning ASN1. It's so much better than all these newfangled things that it makes them look like toys.
      - rkeene2 8y ago
        I'm not sure why ASN.1 isn't more popular for a zero-copy encoding system
ElBarto 8y ago
That's really a list of how not to develop microservices, or even software... Thoughts must obviously be given to protocols. Json is an obvious bad choice for this use case... The point of microservices is loose coupling, including in the code. Having a code hierarchy negates this and arguably is bad practice in general.
jon-wood 8y ago
I agree with all those drawbacks, but auth is something that can be handled with a bit of one time engineering effort. Where I am all traffic to microservices comes through an API gateway which is responsible for routing traffic to the correct service, and more importantly authorising access to those endpoints. Once the gateway has completed auth it places a signed JWT in the Authorisation header, at which point the microservice's responsbility goes from handling the entire auth process to checking the signature can be verified.
salamander014 8y ago
The whole point of doing microservices is so that you can split up processing responsibility boundaries reasonably, and each team is responsible for being an "expert" in the service it's responsible for. This also means that each service should have no other services as dependencies, and if they do, you have to many separate services and you should probably look into why hey aren't wrapped up together. Using a stream from a different service is one thing: You should have clearly defined interfaces for inter-service communication. But if updating a service means you also need to fix an upstream service, your doing it wrong and are actually causing more work than just using a monolith. EDIT: and because you have clearly defined interfaces, these issues with updating one service and affecting another service literally cannot exist if you've done the rest correctly.

merb 8y ago

> So naturally, branching, building and testing across all these different microservices is difficult. Imagine having to roll back a commit, then having to find the equivalent commit in the two other parent services, then rolling back the horizontal services to the equivalent commit that should not happen. if it does you don't have a microservice architecture, you have a spaghetti service architecture.

tabtab 8y ago
How does one know if X is the wrong solution or if X is the right solution but the shop is "doing X wrong"? This also applies to monoliths: maybe they can scale, but one is doing them wrong. Changing from doing monoliths wrong to doing microservices wrong is obviously not progress. The same issue appeared when OOP was fairly new: people started using it heavily and ended up making messes. They were then told that they were "doing it wrong". OOP was initially sold as magic reusable Lego blocks that automatically create nice modularity. There was even an OOP magazine cover showing just that: a coder using magic Legos that farted kaleidoscopic glitter. Microservices is making similar promises. It took a while to learn where and how to use OOP and also when not to: it sucks at some things.
- freedomben 8y ago
  If you want to learn "the right way" I highly recommend "Building Microservices: Designing Fine-Grained Systems" by Sam Newman[1] [1] https://smile.amazon.com/Building-Microservices-Designing-Fine-Grained-Systems/dp/1491950358/ref=sr_1_3 https://smile.amazon.com/Building-Microservices-Designing-Fi...
- notyourday 8y ago
  I have posted about this before. OP is describing nano-services, not micro-services. Nano-service is micro-service that provides leftpad via JSON api. You have an auth.yourapp.com and api.yourapp.com and maybe tracer.yourapp.com and those three things are not a single app that behaves like auth, api or tracer depending on setting of a NODE_ENV variable? If so, you have micro services.
- taneq 8y ago
  Simple: If X is a technology I don't like, and it's not working for you, then it's the wrong solution. If X is a technology I don't like, and it is working for you, then you simply haven't scaled enough to understand its limitations. If X is a technology I like, but it's not working for you, then your shop is "doing X wrong". If X is a technology I like, and it's working for you, then it's the right solution and we're both very clever.
  - staticassertion 8y ago
    edit: See below
    - taneq 8y ago
      Uh, my post was snark towards engineers' general tendency to champion and defend their own favourite architecture over competing approaches rather than focusing on the most suitable architecture for the circumstance, in response to this question: > How does one know if X is the wrong solution or if X is the right solution but the shop is "doing X wrong"? Edit: And a civil conversation ensued. :)
mirkules 8y ago
Heh, I like that “Spaghetti Microservices”. You are right. It should not happen. It is difficult to see these pitfalls when unwinding an unwieldy monolith, and, as an organization all you’ve ever done are unwieldy monoliths, that have a gazillion dependencies, interfaces and factories. We learned from it, and we move on - hopefully, it serves as a warning to others.
- tcbasche 8y ago
  I've heard the term 'Distributed Monolith'
- james_s_tayler 8y ago
  MicroPasta
  - cestith 8y ago
    Tangled angelhair.
- c3534l 8y ago
  I think we should call it angel-hair pasta code. You know, because it's very small spaghetti.
- smnplk 8y ago
  No, it's ravioli oriented architecture
dragonwriter 8y ago
> that should not happen. if it does you don't have a microservice architecture, you have a spaghetti service architecture. A "service" is not defined principally by a code repository or communications-channel boundary (though it should have the latter and may have the former), but by a coupling boundary. OTOH, maintaining a coupling boundary can have non-obvious pitfalls; e.g., supported message format versions can become a source of coupling--if you roll back a service to the version that can't send message format v3, but a consumer of the messages requires v3 and no longer supports consuming v2, then you have a problem.
- merb 8y ago
  you should never actually remove an api without a deprecation process. basically a message format is part of your api. actually in a microservice world your interfaces/api should be as stable as possible. if something gets changed, it needs to go trough a deprecation, that means not removing it until a certain amount of time.
  - NicoJuicy 8y ago
    The accept header could fix this easily if no third parties are involved
  - NicoJuicy 8y ago
    The accept header could fix this easily if no third parties are involved and if it's a rest like API protocol
    - dragonwriter 8y ago
      The accept header is a mechanism for communicating formats you support; it doesn't do anything to address the problem of managing change to supported versions, which is a dev process issue, not a technical issue.
      - geezerjay 8y ago
        > The accept header is a mechanism for communicating formats you support; it doesn't do anything to address the problem of managing change to supported versions That assertion is not true. Media type API versioning is a well established technique.
sequoia 8y ago
It sounds like you're paraphrasing the great Steven Jobs: "You're holding it wrong" Be that as it may, I believe mirkules's issue is not an uncommon one. Perhaps saying "building a microservice architecture 'the right way' is a complex and subtle challenge" would capture a bit of what both of you are saying. Something being complex and therefor easy to mess up does not mean it's a great system and the users are dumb, especially if there are other (less complicated, less easy to mess up) ways to complete the task.
- geezerjay 8y ago
  > Perhaps saying "building a microservice architecture 'the right way' is a complex and subtle challenge" would capture a bit of what both of you are saying. Supporting API versioning is not a complex or subtle challenge. It's actually a very basic requirement of a microservice architecture. It's like having to test null pointers: it requires additional work but it's still a very basic requirement if you don't want your app to blow up in your face.
jeffasinger 8y ago
I think a bigger thing here, is that each deploy should have a REALLY good reason to not be backwards compatible for some pretty long time period. If that requirement is painful for you, then you probably have two pretty tightly coupled services.
balp 8y ago
To handle deploys and especially rollbacks you need a working CI, or rather CD chain. Where eveything is automated. In there are dependencied in your architecture, all of that needs to be part of the same deploy so you can redo the deploy before that one. With a monolith, you get a simple deploy of all dependencies, as they are baked into one thing. There are down sides of everything baked into one thing. Having all your "micoro" servies deplyed as one package, would make the dependency you have act as before the monolith change. Seeing this kinds of dependency imho even in a monolith are an architechture problem, that in the monolith shows up at code changed and fixes can't be handles by localized fixes, but changed have to go into large parts of the code base.

jrockway 8y ago

Protocol buffers exist to solve two problems you list, parsing overhead and backwards compatibility.

jcims 8y ago
I always wondered why ASN.1 never really seemed to take off. Tooling maybe?
- acdha 8y ago
  Yes - both the protocol and tooling are expensive to support, even by 90s standards. Unless you have to use it for compatibility with a service you can’t fix it’d be much better to start with a better implementation of the concept.
- throwaway2048 8y ago
  ASN.1 has a number of problems with equivalent parsings (very bad in a security context, has been the source of a number of TLS vulnerabilities), as well as the fact that even discovering if two ASN.1 parsers will give the same result is undecidable.

eweise 8y ago

We use grpc instead of rest for internal synchronous communication but we've also found that by using event pub/sub between services, that there are not many uses cases where we have direct calls between services. We used to have a parent maven pom and common libraries but got rid of most of that because it caused too much coupling. Now we create smaller more focused common libraries and favor copy/paste code over reuse to reduce coupling. We also moved a lot of the cross cutting concerns into Envoy so that the services can focus on business functionality.

mirkules 8y ago
gRPC and Envoy are exactly the things we are exploring now, although copy/paste would never fly in our org.
- praneshp 8y ago
  I know copy paste is looked down upon, but it's even suggested as a good practice (a little bit) in golang. A little copying is better than a little dependency. https://go-proverbs.github.io/ https://go-proverbs.github.io/
lucidone 8y ago
How do you handle cases where your client is awaiting a response with a decoupled pub/sub backend? E.g a user creates an account and the client needs to know their user id. Would that user object be the responsibility of one service, or written to many tables in the system under different services, or...?
- RhodesianHunter 8y ago
  For one, you could use something like snowflake IDs so that whatever server receives the user data first can generate and return an id for that user before tossing the data on a queue to be processed.
  - lucidone 8y ago

gregorygoc 8y ago

You must be doing something wrong. In a company I work for, we clearly have separated microservices by bounded context, thus making them completely decoupled.

jasonpeacock 8y ago
This is the key - decoupled. When you have to rollback commits across multiple service, they are not decoupled, and you're doing something wrong. Each service should be fully independent, able to be be deployed & rolled-back w/o other services changing. If you're making API changes, then you have to start talking about API versioning and supporting multiple versions of an API while clients migrate, etc.
- vogre 8y ago
  >then you have to start talking about API versioning and supporting multiple versions of an API while clients migrate, etc. Which adds some more complexity that just does not exist in monolythic architecture
  - icebraining 8y ago
    Sure it does. Once you have a large monolith, the same coordination problems hit. Just ask the Linux kernel developers: https://lwn.net/Articles/769365/ https://lwn.net/Articles/769365/

nevir 8y ago

Consider larger services than a typical microservice. For the same reason monoliths tend to split when the organization grows, it is often more manageable to have a small number of services per team (ideally 1, or less). It's ok if a service owns more than one type of entity. It's less good if a service owns more than one part of your businesses' domain, however

gav 8y ago
> Consider larger services than a typical microservice. People seem to forget that there’s a continuum between monolith and microservices, it’s not one or the other. Multiple monoliths, “medium”-services, monolith plus microservices, and so on are perfectly workable options that can help transition to microservices (if you ever need to get there at all).
jeffasinger 8y ago
That's fine, but reorganizations happen, teams can grow, and there is an advantage to having things be separate services in cases like this. Definitely don't just stuff unrelated stuff into a service since a team that normally deals with that service is now working on unrelated stuff. If the unrelated stuff takes off, you now have two teams untangling your monolithic service. That said, I'm a big fan of medium sized services, the kind of thing that might handle 10 or 20 different entities.

jimbokun 8y ago

"turned a team of 15 engineers from managing one giant monolith to two teams managing about 10 or so microservices" Knowing absolutely nothing about your product, this sounds like a bad way to split up your monolith. Probably something like 5 teams of 3 each managing 1 microservice would be a better way to split things up. That way each team is responsible for defining their service's API, and testing and validating the API works correctly including performance requirements. This structure makes it much less likely services will change in a tightly coupled way. Also, each service team must make sure new functionality does not break existing APIs. Which all make it less likely to have to roll back multiple commits across multiple projects. The performance issues you cite, also seem to indicate you have too many services, because you are crossing service boundaries often, with the associated serialization and deserialization costs. So each service probably should be doing more work per call. "all depending on a common parent microservice" This makes you microservices more like a monolith all over again, because a change in the parent can break something in the child, or prevent the child from changing independently of the parent. Shared libraries I think are a better approach. "Authentication/Authorization also becomes challenging since every microservice needs to be auth-aware." Yes, this is a pain. Because security concerns are so important, it is going to add significant overhead to every service to make sure you get it right, no matter what approach you use.

james_s_tayler 8y ago
Probably more like do your best to reason about the boundaries of each bounded context and pick an appropriate number of services based off that analysis? Surely splitting up your application along arbitrary lines based on advice of an internet stranger whose never seen the application and doesn't know the product/business domain just isn't sound way of approaching the problem.
- toolslive 8y ago
  in the end, your architecture will reflect your organigram. (IIRC it's Conway's law?)
  - 8y ago

DelightOne 8y ago

> Management of 10-ish repositories and build jobs. Does each micro service have to live in it's own repository? Especially with a common library everyone uses?

statictype 8y ago
I think having a common library that does anything specific to your product is something of an anti-pattern in micro services. Its not really a micro service - its a distributed monolith
- TheCoelacanth 8y ago
  There are always going to be cross-cutting concerns. You don't want to have every microservice implement it's own authentication, auditing, etc. from the ground up.

xvector 8y ago

Perhaps a few ideas: - Performance: use gRPC/protobuf instead of HTTP/OpenAPI, really not much of a reason to use HTTP/OpenAPI for internal endpoints these days - Repo Management: No one is stopping you from using a monorepo but yourselves :)

mirkules 8y ago
Even just defining what “internal communication” means is difficult. We definitely suffer from the what-if syndrome- “what if some day we want to expose this service to a client?” Our product is a collection of large systems used by many customers with very different requirements - and so we often fall into this configurability trap: “make everything super configurable so that we don’t have to rebuild, and let integration teams customize it”
- xvector 8y ago
  Ah. In that case, you can expose your gRPC endpoints as traditional JSON/HTTP ones with gRPC-Gateway, which supports generating OpenAPI documentation too! Best of both worlds.
- tchaffee 8y ago
  Sounds like you're suffering from not using YNGNI enough: you're not gonna need it. Build what you need now. When that changes, you can change what you built. That was the original intention of Agile methodologies and BDD or TDD. When the tests pass, you're done.

zebraflask 8y ago

These are excellent points. I was just having a conversation yesterday about system design and how there seems to be a tipping point after which the transactional/organizational costs of segmenting services outweigh the benefits. My unscientific impression is that some of the organizational costs - just keeping the teams coordinated and on the same page - can become even more "expensive" than the technical costs.

clon 8y ago

Don't forget: - Logging. All messages pertaining to a request (or task) should have a unique ID across the entire fleet of services in order to follow the trail while debugging.

tyingq 8y ago
"correlation id" is probably the best thing to poke into Google for guidance on it.
- dustindiamond 8y ago
  I would recommend investigating APM, opentracing and Uber’s jaeger project.

flukus 8y ago

> This sounds good in theory, but parsing JSON and then serializing it N times has a real performance cost. It's not just the serialization cost but latency (https://gist.github.com/jboner/2841832 https://gist.github.com/jboner/2841832) as well, every step of the process adds latency, from accessing the object graph, serializing it, sending it to another process and/or over the network, then building up the object graph again. The fashion in .net apps used to be to separate service layers from web front ends and slap an SOA (the previous name for micro-services) label on it. I experimented with moving the service to in process and got an instant 3x wall clock improvement on every single page load, we were pissing away 2/3rds of our performance and getting nothing of value from it. And this was int the best case scenario, a reasonably optimized app with binary serialization and only a single boundary crossing per user web request. Other worse apps I've worked on since had the same anti-pattern but would cross the service boundary dozens/hundreds/thousands of times and very simple pages would take several seconds to load. It's enterprise scale n+1. If you want to share code like this then make a dll and install it on every machine necessary, you've got to define a strict API either way.

Rapzid 8y ago

JSON parsing IS expensive. Way more expensive than many people realize. There is actually an almost-underground JSON parser SCENE in the .Net ecosystem where people develop new parser and try to squeeze the maximum performance using all sorts of tricks: https://github.com/neuecc/Utf8Json https://github.com/neuecc/Utf8Json . Here there is discussion of needing JSON support in .Net core that's faster than Json.Net: https://github.com/dotnet/announcements/issues/90 https://github.com/dotnet/announcements/issues/90 . And people say web applications are never CPU bound :)

NicoJuicy 8y ago
gRCP is a little bit faster, Google created protobuff which should be easier to migrate to than gRPC, but the protocol is unreadable ( binary)... JSON has its advantages, I prefer a feature flag when I need performance ( protobuff vs Json), http headers do the rest
- stouset 8y ago
  gRPC is built on top of protobuf. It’s literally just RPC with protobufs.
0815test 8y ago
> JSON parsing IS expensive Well, maybe https://news.ycombinator.com/item?id=19214387 https://news.ycombinator.com/item?id=19214387
tekknik 8y ago
one could argue that if serialization/deserialization is the majority of what your api is spending time on then you’ve got distributed monoliths and shouldn’t be making the call in the first place. also ensuring you have the fastest json parser before having a real problem is premature optimization

tanilama 8y ago

Why so many microservices? Monolith has huge advantage when maybe your code is like 100k lines or below: 1. Easy cross module unit testing/integration testing, thus sharing components is just easier. 2. Single deployment process 3. CR visibility automatically promotes to all parties of interests, assuming the CR process is working as desired. 4. Also, just a personal preference, easier IDE code suggestion. If you went through json serializing/de-serializing across module boundary, type inference/cohesion is just out-of-reach. And it is not like monolith doesn't have separation of concern at all. After all, monolith can have modules, and submodules. Start abstracting using file system API, but grouping relevant stuff into folders, before put them into different packages. After all, once diverging, it is really hard to go back. Unless you have a giant team and more than enough engineers to spare for devops. Micrservices can be considered as a organizational premature optimization.

badloginagain 8y ago

> We also have our microservices in a hierarchy, all depending on a common parent microservice. Can you explain this a bit more? I thought the point was to have each service be as atomic as possible, so that a change to one service does not significantly impact other services in terms of rollbacks/etc. If I'm wrong here let me know, our company is still early days of figuring out how to get out of the problems presented by monolith (or in our case, mega-monolith).

mcguire 8y ago

I'm going to go out on a limb here and suggest that parsing (and serializing) JSON is unlikely to be the actual problem, performance-wise. (Although "OpenAPI/Swagger" doesn't fill me with enthusiasm.) More likely, I suspect, is that either you are shipping way too much data around, you have too much synchrony, or some other problem is being being hidden in the distribution. (I once dealt with an ESB service that took 2.5 seconds to convert an auth token from one format to another. I parallelized the requests, and the time to load a page went from 10 sec to <3; then I yanked the service's code into our app and that dropped to milliseconds.) Performance problems in large distributed systems are a pain to diagnose and the tools are horrible.

therealdrag0 8y ago
I've been using NewRelic and it's been wondrous around illuminating performance problems.

philwelch 8y ago

I don't think it's necessarily the best idea to immediately have almost as many services as you have engineers. There are usually more gradual and logical ways to split things up.

bunderbunder 8y ago

The first thing I start trying to convince people to ditch on any internal service is JSON. There are only two times when JSON is a particularly good choice: when it's important for the messages themselves to be human-readable, or when it's important to be able to consume it from JavaScript without using a library. Any other time, something like protocol buffers is going to give you lower latency, lower bandwidth requirements, lower CPU costs, lower development effort, less need for maintaining documentation, and better standardization. If you ditch the HTTP stuff while you're at it, you can also handily circumvent all the ambiguities and inter-developer holy wars that are all but inherent to the process of taking your service's semantics, whatever they are, and trying to shoehorn them into a 30-year-old protocol that was really only meant to be used for transferring hypermedia documents. Instead you get to design your own protocol that meets your own needs. Which, if you're already building on top of something like protobuf, will probably end up being a much simpler and easier-to-use protocol than HTTP.

CoolGuySteve 8y ago
Man if you think protobuf is fast, just wait until you try C pod structs with type-length headers. They are literally infinitely faster to encode/decode than protobuf. They even have the same obnoxious append-only extensibility of protobuf if that’s what really gets your jimmies firing.
- netmonk 8y ago
  asn1 for the win
  - carapace 8y ago
    It's a shame you're catching downvotes for mentioning ASN1. It's so much better than all these newfangled things that it makes them look like toys.
    - rkeene2 8y ago
      I'm not sure why ASN.1 isn't more popular for a zero-copy encoding system

ElBarto 8y ago

That's really a list of how not to develop microservices, or even software... Thoughts must obviously be given to protocols. Json is an obvious bad choice for this use case... The point of microservices is loose coupling, including in the code. Having a code hierarchy negates this and arguably is bad practice in general.

jon-wood 8y ago

I agree with all those drawbacks, but auth is something that can be handled with a bit of one time engineering effort. Where I am all traffic to microservices comes through an API gateway which is responsible for routing traffic to the correct service, and more importantly authorising access to those endpoints. Once the gateway has completed auth it places a signed JWT in the Authorisation header, at which point the microservice's responsbility goes from handling the entire auth process to checking the signature can be verified.

salamander014 8y ago

The whole point of doing microservices is so that you can split up processing responsibility boundaries reasonably, and each team is responsible for being an "expert" in the service it's responsible for. This also means that each service should have no other services as dependencies, and if they do, you have to many separate services and you should probably look into why hey aren't wrapped up together. Using a stream from a different service is one thing: You should have clearly defined interfaces for inter-service communication. But if updating a service means you also need to fix an upstream service, your doing it wrong and are actually causing more work than just using a monolith. EDIT: and because you have clearly defined interfaces, these issues with updating one service and affecting another service literally cannot exist if you've done the rest correctly.