9 ms·

Untangling microservices, or balancing complexity in distributed systems

156 points by ablekh 6y ago

0xDEEPFAC 6y ago
Relevant comedy: https://www.youtube.com/watch?v=y8OnoxKotPQ https://www.youtube.com/watch?v=y8OnoxKotPQ
- Frost1x 6y ago
  This made my day, thank you.
- ablekh 6y ago
  Hilarious video (and most likely very true). Thank you for sharing.
  - hinkley 6y ago
    Do you think he ever found love?
    - ablekh 6y ago
      I was referring to technology aspect only. Regarding love, you'd have to ask him. :-)
- ecnahc515 6y ago
  I knew this would be in the comments before even opening the page. This thing has been getting shared a ton lately since it was just uploaded 2 weeks ago. I think it kind of goes to show how many people empathize with the video.
- tcgv 6y ago
  Great satire! haha There should be one of these videos for reactive programming as well. Misuse of observable pattern can quite quickly turn an app into a mess, making code navigation / minor change requests a nightmare.
- aledalgrande 6y ago
  Great video, also in a couple of clicks I got to this, which is equally hilarious: https://www.youtube.com/watch?v=5WL_jkFS2gw https://www.youtube.com/watch?v=5WL_jkFS2gw
- hn_throwaway_99 6y ago
  Hah, I have seen this before but this was the first time I actually notice the PRODUCT SPEC at the beginning: GOAL: Surprise and delight users by displaying their birthday on the setting page. DELIVERABLES: Display User Birthday on Settings Page "Surprise and delight" has definitely become one of my new bullshit bingo favorites.
wdb 6y ago
The backlog example in the article is crazy! Do they really do services like that in the big world? Backlog service which encapsulates all the services I can follow (partly) but all these separate ones
jennyyang 6y ago
Uber is not reorganizing their microservices into "macroservices". The original tweeter refuted it, but it's still being propagated as fact. Granted the original tweet was poorly written but in the thread itself, he refutes it later on.
- antoncohen 6y ago
  Thank you. This is the core of it[1], IMO: > Back in the day, we’d spin up a microservice that did one, small thing just like that. We had a bunch of small services built and maintained by one person. This was great for autonomy, iteration speed, learning and making devops a no-brainer. You could spin up a service anytime: but you’d be oncall for it. > Now, as my area is maturing and it’s easier to look ahead, as we create new platforms, we’re doing far more thoughtful planning on new services. These services don’t just do one thing: they serve one business function. They are built and maintained by a team (5-10 engineers). They are more resilient and get far more investment development and maintenance-wise than some of those early microservceis. [1] https://lobste.rs/s/mc3k1c/at_uber_we_re_moving_many_our https://lobste.rs/s/mc3k1c/at_uber_we_re_moving_many_our
  - kevindong 6y ago
    > We had a bunch of small services built and maintained by one person. That just seems like it'd be a disaster at any non-trivial scale.
FpUser 6y ago
I've never bought into this idea of microservices. Always wrote high performance native servers and have yet to encounter the situation when well designed monolith running on multicore monster with gobbles of RAM failed to satisfy client. Granted it will not work for Google but what does the world care. Most of the businesses (read potential clients) will never come anywhere close to those scales. Long time ago when computers were slow I did design some distributed applications myself - reliable IP multicast server, business process servers, map reduce type solutions like bill generation etc where it did make sense. That was long time ago and now most of those at their old scale could run on a single reasonable priced server.
- jzoch 6y ago
  If you dont have a team large enough or a business big enough to need multiple individual services than you aren't the target market. Unless you care about high availability (which small services can help you achieve) then a single server is fine.
  - FpUser 6y ago
    May I ask what high availability has to do with the microservices? Since many of my servers were designed for relatively large companies (like TELCO) high availability/reliability requirements were always there. Running 2 or more monoliths in parallel with the appropriate logic built in does not make them into micro services
    - jzoch 6y ago
      if they are large enough and have components that need to scale independently than a bottleneck in one of those components may affect availability of the service.
      - FpUser 6y ago
        Or may not. I design and create products that serve particular problem for customers while satisfying reasonable constraints. If it comes to a situation you've described than it will be spun into a separate component. I already told that I did and know how to do distributed applications. The keywords here are "WHEN NEEDED". And on modern hardware it is not in my cases.
barnyfried 6y ago
I disagree completely with this article. What I have seen that should give people pause about microservices is, if you are not good at building software (which most enterprise shops simply are not), microservices could potentially destroy you. But your legacy monolith already brought you to your knees so you have little to lose. Most enterprise shops simply are not mature enough to implement dumb pipe/smart endpoints not because theres something especially hard about doing it but because their teams and their business people don't really have the perspective or desire to successfully manage such distributed complexities. Where the monoliths fail is in the idea that this distribution is not needed (it always is) and instead offloads this on to the use of many different integration technologies such as queues, message buses, caches, document databases, RDBMS sidecar systems that will share data inappropriately, cause unresolvable data consistency issues, and generally only solve local problems while creating global ones. Microservices are the other end of the spectrum, if your design has anything to do with sharing significant amounts of data to complete a transaction, you are entering a world of pain. You can share some data at the API level but beyond that you will be in for a ton of trouble. Its true this requires more discipline at the design level and more discipline in implementation and operations. The moral of the story is that if you cant level up to managing microservices with the horses you currently have, personally, I would leave it alone. If your 3-5 years behind the curve already as an enterprise shop, you may want to consider putting your whole system into maintenance mode, and paying a competent organization to manage it, buy a startup or buy some COTS and call it a day. As for the single server guys know this, yes you can make a ton of money scaling that out on a limited basis and selling your startup or IPO-ing. But know that as soon as you cash out, it will be someones job to completely rewrite your single server drivel to serve even one more customer or business need in a predictable, scalable, cost effective manner. See JBoss, see ROR, see any already dead on the vine monolith technology. Its over there in the corner of the office. Its called the trash bin.
hn_throwaway_99 6y ago
Some other commenters have discussed it, but one thing to make very clear that I think a lot of people are missing: 1. I fervently believe, after having lots of experience with both microservices and monoliths, that microservices do not provide any technical advantage over monoliths. If anything, they can make a lot of technical details more difficult. 2. Microservices, however, can solve some particularly thorny organization problems for large teams with respect to how these large teams build complex software systems. Thus, in my experience microservices can serve a useful purpose, but they will only be as successful as your organizational-wide communication is. Furthermore, for the developers replying "I don't see why I'd ever use microservices for project xyz", well, if you're a team of 1, there is hardly ever a need to use microservices.
- gbuk2013 6y ago
  YMMV and also depending on what counts as a microservice but I currently write and maintain (as part of a very small team) several services that run on 200+ (physical, virtual and cloud) devices around the world and a monolith simply wouldn’t be able to do the job for boringly real technical reasons. Another team writes and maintains a few more. We also have some monoliths that provide customer facing UIs and aggregate data for our support teams and they are perfectly fine for that. Right hammer for the right nail and all that.
  - lukevp 6y ago
    You can have a distributed monolith. Monolith doesn’t mean there’s a single deployment. It’s about the interaction between domains. So if you have an app with a database on each device and the app layer is 1 application, one deploy, that’s still a monolith.
- doingmyting 6y ago
  I'll respectfully beg to differ here. A small app with a few functions at a small company, I completely agree. But once you have a large scale system you'll thank your lucky stars for microservices. Everything from deployments to development is a much better experience than in a large monolith. It's much easier to reason about and it's much easier to see where and why something failed if and when that happens.
jka 6y ago
Opinions are welcome on whether this kind of approach could help move forward from the microservice/monolith debate: - Teams commit to individual repositories so that team membership, changeset review, and issue management are kept close to the relevant code, and checkout sizes are minimal - Strong language support for inter-repository dependency management is introduced to enable effective rollback and rollforward of commits that affect other repositories - The function call becomes the unit of deployment -- not a microservice, and not a monolith. This allows for low bandwidth deployments and for small amounts of highly-utilized code to migrate to the devices where they are needed for computation - Function calls and return values are signed cryptographically so that inter-function calls have integrity, regardless whether within a single device or across the network - Functions that perform I/O calls are tagged with the virtual storage and/or network interfaces that they require access to - Kubernetes-like infrastructure manages the deployment, scaling, and resource allocation of functions to devices This seems to me like it would provide the organizational benefits of microservices while reducing developer hand-wringing over where to define service boundaries. It just becomes a question of 'should this be a function?'. Good tooling around binary-dependency-or-source-code retrieval could make it easy for developers to work on large applications built like this and inspect / debug / modify code as-needed without having to check out or download all the source code and binaries at once.
js8 6y ago
I find the discussion of monolith vs microservices to be very unhelpful. It is a discussion about how to split the code. But what you really want to understand, when building a distributed system (i.e. system to process large amounts of data), is how to partition the data, so it could be processed in parallel. Essentially, there are two options, vectorization (single instruction multiple data) and pipelining (multiple instruction single data). They both have different uses in different scenarios, based on the critical path dependencies in the data processing. Monoliths are easier to vectorize, microservices are easier to pipeline. But to choose which one you need before you understand the nature of data processing you need is a wrong way to do it.
- closeparen 6y ago
  The "vectorizing" and "pipelining" here seem to work when describing changes/deployments made to the system, but that seems orthogonal to the data processed by the system. If one part of your workload is suited to pipelining, and another part of your workload is suited to vectorizing, then that might be a reason to split the workload into different processes running on different clusters. Few but beefy nodes for the vectorized part. Many smaller nodes for the pipelined part.
  - js8 6y ago
    > The "vectorizing" and "pipelining" here seem to work when describing changes/deployments made to the system That's not what I mean. But you're correct, what you want to do with the data informs your decision of what should be separate processes, and you once you know, you might decide to split (modularize) the processing code accordingly. Doing that other way around (i.e. to design the modules before you understand the data flow) is just going to cause more trouble.
    - closeparen 6y ago
      Monolithic architecture implies the company is only willing to manage one production deployable. Someone solving a specific problem cannot introduce new processes / network boundaries even if warranted based on the characteristics of their problem.
tutfbhuf 6y ago
> You cannot build a system out of independent components! I disagree, the Unix philosophy proves otherwise. You can build very powerful applications, just by combining grep, sed, ls, ... and the like.
m0llusk 6y ago
This oversimplifies the measure of complexity by focusing on the difference between distributed and monolithic systems. A potentially interesting example I recently encountered was a monolithic system that used internal interfaces. Transitioning these internal interfaces from a synchronized design to an unsynchronized design effectively introduced much of the complexity of a distributed system into monolithic design. Ultimately everything about data flow, typing, abstraction, and so on has some effect on performance, encapsulation, complexity, organization. In a way all aspects of development from onboarding to builds and tests to release distribution need to be measured the same way we take into account of resource usage and basic performance criteria. If a piece of software masterfully meets all needs, but has internal interfaces that are simply too complex and brittle to be worth learning then that software will quickly decay and need to be replaced as the dynamic world alters the constraints of usage.

156 points by ablekh 6y ago

0xDEEPFAC 6y ago

Relevant comedy: https://www.youtube.com/watch?v=y8OnoxKotPQ https://www.youtube.com/watch?v=y8OnoxKotPQ

Frost1x 6y ago
This made my day, thank you.
ablekh 6y ago
Hilarious video (and most likely very true). Thank you for sharing.
- hinkley 6y ago
  Do you think he ever found love?
  - ablekh 6y ago
    I was referring to technology aspect only. Regarding love, you'd have to ask him. :-)
ecnahc515 6y ago
I knew this would be in the comments before even opening the page. This thing has been getting shared a ton lately since it was just uploaded 2 weeks ago. I think it kind of goes to show how many people empathize with the video.
tcgv 6y ago
Great satire! haha There should be one of these videos for reactive programming as well. Misuse of observable pattern can quite quickly turn an app into a mess, making code navigation / minor change requests a nightmare.
aledalgrande 6y ago
Great video, also in a couple of clicks I got to this, which is equally hilarious: https://www.youtube.com/watch?v=5WL_jkFS2gw https://www.youtube.com/watch?v=5WL_jkFS2gw
hn_throwaway_99 6y ago
Hah, I have seen this before but this was the first time I actually notice the PRODUCT SPEC at the beginning: GOAL: Surprise and delight users by displaying their birthday on the setting page. DELIVERABLES: Display User Birthday on Settings Page "Surprise and delight" has definitely become one of my new bullshit bingo favorites.

wdb 6y ago

The backlog example in the article is crazy! Do they really do services like that in the big world? Backlog service which encapsulates all the services I can follow (partly) but all these separate ones

jennyyang 6y ago

Uber is not reorganizing their microservices into "macroservices". The original tweeter refuted it, but it's still being propagated as fact. Granted the original tweet was poorly written but in the thread itself, he refutes it later on.

antoncohen 6y ago
Thank you. This is the core of it[1], IMO: > Back in the day, we’d spin up a microservice that did one, small thing just like that. We had a bunch of small services built and maintained by one person. This was great for autonomy, iteration speed, learning and making devops a no-brainer. You could spin up a service anytime: but you’d be oncall for it. > Now, as my area is maturing and it’s easier to look ahead, as we create new platforms, we’re doing far more thoughtful planning on new services. These services don’t just do one thing: they serve one business function. They are built and maintained by a team (5-10 engineers). They are more resilient and get far more investment development and maintenance-wise than some of those early microservceis. [1] https://lobste.rs/s/mc3k1c/at_uber_we_re_moving_many_our https://lobste.rs/s/mc3k1c/at_uber_we_re_moving_many_our
- kevindong 6y ago
  > We had a bunch of small services built and maintained by one person. That just seems like it'd be a disaster at any non-trivial scale.

FpUser 6y ago

I've never bought into this idea of microservices. Always wrote high performance native servers and have yet to encounter the situation when well designed monolith running on multicore monster with gobbles of RAM failed to satisfy client. Granted it will not work for Google but what does the world care. Most of the businesses (read potential clients) will never come anywhere close to those scales. Long time ago when computers were slow I did design some distributed applications myself - reliable IP multicast server, business process servers, map reduce type solutions like bill generation etc where it did make sense. That was long time ago and now most of those at their old scale could run on a single reasonable priced server.

jzoch 6y ago
If you dont have a team large enough or a business big enough to need multiple individual services than you aren't the target market. Unless you care about high availability (which small services can help you achieve) then a single server is fine.
- FpUser 6y ago
  May I ask what high availability has to do with the microservices? Since many of my servers were designed for relatively large companies (like TELCO) high availability/reliability requirements were always there. Running 2 or more monoliths in parallel with the appropriate logic built in does not make them into micro services
  - jzoch 6y ago
    if they are large enough and have components that need to scale independently than a bottleneck in one of those components may affect availability of the service.
    - FpUser 6y ago
      Or may not. I design and create products that serve particular problem for customers while satisfying reasonable constraints. If it comes to a situation you've described than it will be spun into a separate component. I already told that I did and know how to do distributed applications. The keywords here are "WHEN NEEDED". And on modern hardware it is not in my cases.

barnyfried 6y ago

I disagree completely with this article. What I have seen that should give people pause about microservices is, if you are not good at building software (which most enterprise shops simply are not), microservices could potentially destroy you. But your legacy monolith already brought you to your knees so you have little to lose. Most enterprise shops simply are not mature enough to implement dumb pipe/smart endpoints not because theres something especially hard about doing it but because their teams and their business people don't really have the perspective or desire to successfully manage such distributed complexities. Where the monoliths fail is in the idea that this distribution is not needed (it always is) and instead offloads this on to the use of many different integration technologies such as queues, message buses, caches, document databases, RDBMS sidecar systems that will share data inappropriately, cause unresolvable data consistency issues, and generally only solve local problems while creating global ones. Microservices are the other end of the spectrum, if your design has anything to do with sharing significant amounts of data to complete a transaction, you are entering a world of pain. You can share some data at the API level but beyond that you will be in for a ton of trouble. Its true this requires more discipline at the design level and more discipline in implementation and operations. The moral of the story is that if you cant level up to managing microservices with the horses you currently have, personally, I would leave it alone. If your 3-5 years behind the curve already as an enterprise shop, you may want to consider putting your whole system into maintenance mode, and paying a competent organization to manage it, buy a startup or buy some COTS and call it a day. As for the single server guys know this, yes you can make a ton of money scaling that out on a limited basis and selling your startup or IPO-ing. But know that as soon as you cash out, it will be someones job to completely rewrite your single server drivel to serve even one more customer or business need in a predictable, scalable, cost effective manner. See JBoss, see ROR, see any already dead on the vine monolith technology. Its over there in the corner of the office. Its called the trash bin.

hn_throwaway_99 6y ago

Some other commenters have discussed it, but one thing to make very clear that I think a lot of people are missing: 1. I fervently believe, after having lots of experience with both microservices and monoliths, that microservices do not provide any technical advantage over monoliths. If anything, they can make a lot of technical details more difficult. 2. Microservices, however, can solve some particularly thorny organization problems for large teams with respect to how these large teams build complex software systems. Thus, in my experience microservices can serve a useful purpose, but they will only be as successful as your organizational-wide communication is. Furthermore, for the developers replying "I don't see why I'd ever use microservices for project xyz", well, if you're a team of 1, there is hardly ever a need to use microservices.

gbuk2013 6y ago
YMMV and also depending on what counts as a microservice but I currently write and maintain (as part of a very small team) several services that run on 200+ (physical, virtual and cloud) devices around the world and a monolith simply wouldn’t be able to do the job for boringly real technical reasons. Another team writes and maintains a few more. We also have some monoliths that provide customer facing UIs and aggregate data for our support teams and they are perfectly fine for that. Right hammer for the right nail and all that.
- lukevp 6y ago
  You can have a distributed monolith. Monolith doesn’t mean there’s a single deployment. It’s about the interaction between domains. So if you have an app with a database on each device and the app layer is 1 application, one deploy, that’s still a monolith.
doingmyting 6y ago
I'll respectfully beg to differ here. A small app with a few functions at a small company, I completely agree. But once you have a large scale system you'll thank your lucky stars for microservices. Everything from deployments to development is a much better experience than in a large monolith. It's much easier to reason about and it's much easier to see where and why something failed if and when that happens.

jka 6y ago

Opinions are welcome on whether this kind of approach could help move forward from the microservice/monolith debate: - Teams commit to individual repositories so that team membership, changeset review, and issue management are kept close to the relevant code, and checkout sizes are minimal - Strong language support for inter-repository dependency management is introduced to enable effective rollback and rollforward of commits that affect other repositories - The function call becomes the unit of deployment -- not a microservice, and not a monolith. This allows for low bandwidth deployments and for small amounts of highly-utilized code to migrate to the devices where they are needed for computation - Function calls and return values are signed cryptographically so that inter-function calls have integrity, regardless whether within a single device or across the network - Functions that perform I/O calls are tagged with the virtual storage and/or network interfaces that they require access to - Kubernetes-like infrastructure manages the deployment, scaling, and resource allocation of functions to devices This seems to me like it would provide the organizational benefits of microservices while reducing developer hand-wringing over where to define service boundaries. It just becomes a question of 'should this be a function?'. Good tooling around binary-dependency-or-source-code retrieval could make it easy for developers to work on large applications built like this and inspect / debug / modify code as-needed without having to check out or download all the source code and binaries at once.

js8 6y ago

I find the discussion of monolith vs microservices to be very unhelpful. It is a discussion about how to split the code. But what you really want to understand, when building a distributed system (i.e. system to process large amounts of data), is how to partition the data, so it could be processed in parallel. Essentially, there are two options, vectorization (single instruction multiple data) and pipelining (multiple instruction single data). They both have different uses in different scenarios, based on the critical path dependencies in the data processing. Monoliths are easier to vectorize, microservices are easier to pipeline. But to choose which one you need before you understand the nature of data processing you need is a wrong way to do it.

closeparen 6y ago
The "vectorizing" and "pipelining" here seem to work when describing changes/deployments made to the system, but that seems orthogonal to the data processed by the system. If one part of your workload is suited to pipelining, and another part of your workload is suited to vectorizing, then that might be a reason to split the workload into different processes running on different clusters. Few but beefy nodes for the vectorized part. Many smaller nodes for the pipelined part.
- js8 6y ago
  > The "vectorizing" and "pipelining" here seem to work when describing changes/deployments made to the system That's not what I mean. But you're correct, what you want to do with the data informs your decision of what should be separate processes, and you once you know, you might decide to split (modularize) the processing code accordingly. Doing that other way around (i.e. to design the modules before you understand the data flow) is just going to cause more trouble.
  - closeparen 6y ago
    Monolithic architecture implies the company is only willing to manage one production deployable. Someone solving a specific problem cannot introduce new processes / network boundaries even if warranted based on the characteristics of their problem.

tutfbhuf 6y ago

> You cannot build a system out of independent components! I disagree, the Unix philosophy proves otherwise. You can build very powerful applications, just by combining grep, sed, ls, ... and the like.

m0llusk 6y ago

This oversimplifies the measure of complexity by focusing on the difference between distributed and monolithic systems. A potentially interesting example I recently encountered was a monolithic system that used internal interfaces. Transitioning these internal interfaces from a synchronized design to an unsynchronized design effectively introduced much of the complexity of a distributed system into monolithic design. Ultimately everything about data flow, typing, abstraction, and so on has some effect on performance, encapsulation, complexity, organization. In a way all aspects of development from onboarding to builds and tests to release distribution need to be measured the same way we take into account of resource usage and basic performance criteria. If a piece of software masterfully meets all needs, but has internal interfaces that are simply too complex and brittle to be worth learning then that software will quickly decay and need to be replaced as the dynamic world alters the constraints of usage.