8 ms·

I couldn't agree more with an article. Most people think a micro-service architecture is a panacea because "look at how simple X is," but it's not that simple.

I couldn't agree more with an article. Most people think a micro-service architecture is a panacea because "look at how simple X is," but it's not that simple. It's now a distributed system, and very likely, it's a the worst-of-the-worst a distributed monolith. Distributed system are hard, I know, I do it. Three signs you have a distributed monolith: 1. You're duplicating the tables (information), without transforming the data into something new (adding information), in another database (e.g. worst cache ever, enjoy the split-brain). [1] 2. Service X does not work without Y or Z, and/or you have no strategy for how to deal with one of them going down. 2.5 Bonus, there is likely no way to meaningfully decouple the services. Service X can be "tolerant" of service Y's failure, but it cannot ever function without service Y. 3. You push all your data over an event-bus to keep your services "in-sync" with each-other taking a hot shit on the idea of a "transaction." The event-bus over time pushes your data further out of sync, making you think you need an even better event bus... You need transactions and (clicks over to the Jepsen series and laughs) good luck rolling that on your own... I'm not saying service oriented architectures are bad, I'm not saying services are bad, they're absolutely not. They're a tool for a job, and one that comes with a lot of foot guns and pitfalls. Many of which people are not prepared for when they ship that first micro service. I didn't even touch on the additional infrastructure and testing burden that a fleet of micro-services bring about. [1] Simple tip: Don't duplicate data without adding value to it. Just don't.

foobarian 7y ago
> 2.5 Bonus, there is likely no way to meaningfully decouple the services. Service X can be "tolerant" of service Y's failure, but it cannot ever function without service Y. Nit: if service X could function without service Y, then it seems to follow service Y should not exist in the first place. And equivalently, the functionality of service Y before some microservice migration.
- titanomachy 7y ago
  Not necessarily. Maybe service X logs in a user and service Y sends them an email letting them know that someone logged in from a new location. Y adds value but it's not essential to X.
- sobani 7y ago
  Classic example is recommendations on a product page. If the personal recommendation service is not available / slow to respond, you might fall back on recommendations based on the best sellers or even fall back further by not giving recommendations at all. Recommendations are not necessary, and not showing them will significantly affect the bottom line, so you don't want to skip them if possible. But not showing the product page (in a timely manner), because the recommendation engine has a hiccup, is even worse for your bottom line.
mycall 7y ago
> Don't duplicate data without adding value to it. What about running multiple versions of a microservice in parallel -- don't each need their own yet separate databases that attempt to mirror each other as best they can?
- aliswe 7y ago
  Those are just different (versions of) interfaces for interacting with the same data source, ie no not several databases... But I'm no expert
- SideburnsOfDoom 7y ago
  No.
- rubyn00bie 7y ago
  I'm assuming you mean in production micro-service, if that's not the case, please elaborate a bit more... The short answer is "no," as succinctly stated by the, I assume from the name, majestic SideburnsOfDoom. The versions shouldn't _EVER_ be incompatible with each other. E.g. you need to rename a column. Do not: rename the column, e.g. `ALTER TABLE RENAME COLUMN...`. Because, your systems are going to break with the new schema. Do: Add a new column, with the new name, and migrate data to the new column, once it's good upgrade the rest of your instances, then drop the old column. Because, you can use both versions at the same time now without breaking anything. Yes, it can be a little tricky to get the data sync'd into the new column, but that's a lot less tricky than doing it for _every_ table and column.
rumanator 7y ago
> Most people think a micro-service architecture is a panacea because "look at how simple X is," but it's not that simple. It's now a distributed system, and very likely, it's a the worst-of-the-worst a distributed monolith. Distributed system are hard, I know, I do it. This line of argument fails to take into consideration any of the reasons why in general microservices are the right tool for the right job. Yes, it's challenging, and yes it's a distributes system. Yet, with microservices you actually are able to reuse specialized code, software packages, and even third-party services. That cuts down on a lot of dev time and cost, and makes the implementation of a lot of of POCs or even MVPs a trivial task. Take for example Celery. With Celery all you need to do to implement a queuable background task system that's trivially scalable is to write the background tasks, get a message broker up and running, launch worker instances, and that's it. What would you have to do to achieve the same goal with a monolith? Implement your own producer/consumer that runs on the same instance that serves requests? And aren't you actually developing a distributes system anyway?
- ar_lan 7y ago
  I'm completely in your camp, and I'm surprised by the lack of nuance HN seems to show (especially regarding micro-services & Kubernetes). There are many benefits to having microservices that people seem to forget because they think that everyone interested in microservices is interested in splitting their personal blog into 4 different services. They take coordination, good CICD, and a lot of forethought to ensure each service is cooperating in the ecosystem properly, but once established, it can do wonders to dev productivity.
  - sterlind 7y ago
    I can't tell if my project is a monolith or microservices, but it's going well so far. We use a single scalable database instance as a message broker and persistence source, and have a common framework implements distributed algorithms that every service uses to expose "OS-like" constructs (actor activation, persistent collections, workflows, Git etc. All communication is done through independent protocols.. there's not much coupling between protocols (except for common ones like "schedule this job on this device"), so it's not really a cobweb of dependencies, but everything relies on that single database. I think if the database gets too overloaded I'll partition certain tree nodes across multiple masters (this is feasible because the framework doesn't rely on a single timestream). With the level of shared code (the framework) and the single database, it's somewhat monolithic but the actors themselves are quite well-behaved and independent on top of it.
yes_man 7y ago
I don't want advocate one way or another (micro vs. monoliths) because tomato tomato. However here are a few arguments in defense of microservices regarding these three signs you commented: 1. Microservices do not have some inherent property of having to duplicate data. You can have data in single source and deliver that data to anyone who needs it through an API. There are infinitely many caching solutions if you are worried about this becoming a bottleneck 2 and 2.5. There are tools for microservice architectures that counter these problems to a degree, mainstream example being containers and container orchestration (e.g. Docker and Kubernetes). One can even make an argument that microservices force you to build your systems so they are more robust than your monolith would be. If the argument for monoliths is that it's easier to maintain reliability when every egg is in one basket then you are putting all your bets into that basket and it becomes a black hole for developer and operations resources, as well as making operations evolution very slow 3. There are again tools for handling data and "syncing" (although I don't like the idea of having to "sync") the services, for example message queues / streaming processing platforms (e.g. Kafka). If some data or information might be of interest to multiple services, you should then push such data to a message queue and consume it from services that need it. The "syncing" problem sounds like something that arises when you start duplicating your data across services which shouldn't happen (see my argument on 1.) Again not to say microservices are somehow universally better. Just coming into defense of their core concepts when they get unfairly accused
- abraxas 7y ago
  The trouble is that this process of streaming events over Kafka or Kinesis means that subscribed microservices will be duplicating the bus data in their own way in their local databases. If one of them falls out of the loop for whatever reason you are in trouble. Now, there is a pattern called Event Sourcing (ES) which proposes that the source of truth should be the event bus itself and microservice databases are mere projections of this data. This is all good and well except it's very hard to implement in practice. If a microservice needs to replay all business events from months or years in the past it may take hours or days to do this. What about the business in the meantime? If it's a service that significantly reduces the usability of you application you effectively have a long downtime anyway. Transactional activity becomes incredibly hard in the microservices world with either 2 phased commit (only good for very infrequent transactions due to performance) or with so called Sagas (which are very complex to get right and maintain). Any company that isn't delivering its service to billions of users daily will likely suffer far more from microservices than they will benefit.
smoyer 7y ago
We've moved a lot of services into Kubernetes and broken things up into smaller and smaller micro-services. It definitely eliminates a lot of the complexity for developers ... but you trade it for operational complexity (e.g. routing, security, mis-matched client/server versions, resiliency when dependency isn't responding). I still believe that overall software quality is higher with micro-services (our Swagger documents serve as living ICDs), but don't kid yourself that you're going to save development time. And don't fall into the trap of shrinking your micro-services too small.
- save_ferris 7y ago
  That’s an encouraging story to hear. The thing I’ve noticed is that the costs of moving a poorly written monolith to a microservice architecture can be incredibly high. I also think that microservice design really needs to be thought through and scrutinized, because poorly designed microservices start to suck really quickly in terms of maintenance.
- Aperocky 7y ago
  You’ll have endless race conditions to deal with, even when storage is central and unique. We learned and is continuing learning that.
  - idiocratic 7y ago
    Unique storage for multiple services sounds like a recipe for disaster. The purpose of splitting services, at least one of, is to decouple parts of the code at a fundamental level, including storage and overall ownership thereof. You're probably better served with a modular monolith if you really can't break storage up.
    - Aperocky 7y ago
      No, only one service is reading/writing, everything else just call that. Still, things get quite lost when it involves talking to multiple other teams and needing to keep everything in sync.
huherto 7y ago
I totally agree. Mostly we are doing microservices the wrong way. We are not drawing the boundaries correctly, they are too small, they have too many interdependencies, they don't really encapsulate the data, and you end up with many interdependencies. There is not enough guidance about sizing them. We are just building distributed monoliths. Which is great for cloud companies because they get to sell many boxes.
- galaxyLogic 7y ago
  Micro-services are just connected things which work together to accomplish something. But where are those connections described? In some tables somewhere. Maybe. Whereas if you write a single monolithic program its connections are described in code, preferably type-checked by a compiler. I think that gives you at least theoretically a better chance of understanding what are the things that connect, and how they connect. So if there was a good programming language for describing micro-services then that would probably resolve many management difficulties, and then the question would be simply do we get performance benefits from running on multiple processors.
  - IggleSniggle 7y ago
    There is such a language, and it is Erlang.
    - jb3689 7y ago
      Literally this. Reading the article I kept thinking to myself "which is why Erlang/Elixir is great because it doesn't make you choose up front". It's wild that with how popular Elixir has gotten that it still isn't seen as a serious contender for many companies
    - rubyn00bie 7y ago
      I write Elixir professionally, Erlang/Elixir and BEAM is no one-stop solution to these problems either. It has tools to help you, but you can very easily end up in the same boat.
    - 7y ago
ignoramous 7y ago
> Simple tip: Don't duplicate data without adding value to it. Just don't. Not much is said about S3's design principles in public, but that one was one of them. Disclaimer: Recalling from memory.
jiggawatts 7y ago
The most basic thing I see people neglecting is that inserting a network protocol is really adding many additional components that weren't there before, often doubling or even tripling the amount of code, config, and documentation required. If there's a single large project that combines "A+B" modules with no networking and you split this into networked services, then you now have: 1) "A" Component 2) "B" Component 3) "A" Server 4) "A" Client So for example if you started off with a single "project" in your favorite IDE, you now have 4, give or take. You might be able to code-gen your server and client code out of a single IDL file or something, but generally speaking you're going to be writing code like "B -> A client -> A server -> A" no matter what instead of simply "B -> A". Now you have to worry about network reliability, back-pressure, queuing, retries, security, bandwidth, latency, round-trips, serialization, load-balancing, affinity, transactions, secrets storage, threading, and on and on... A simple function call translates to a rats nest of dependency injection, configuration reads, callbacks, and retry loops. Then if you grow to 4 or more components you have to start worrying about the topology of the interconnections. Suddenly you may need to add a service bus or orchestrator to reduce the number of point-to-point connections. This is not avoidable, because if you have less than 4 components, then why bother to break things out into micro services in the first place!? Now when things go wrong in all sorts of creative ways, some of which are likely still the subject of research papers, heaven help you with the troubleshooting. First, it'll be the brownouts that the load balancer doesn't correctly flag as a failure, then the priority inversions, then the queue filling up, and then it'll get worse from there as the load ramps up. Meanwhile nothing stops you having a monolithic project with folders called "A", "B", "C", etc... with simple function calls or OO interfaces across the boundaries. For 99.9% of projects out there this is the right way to go. And then, if your business takes off into the stratosphere, nothing stops you converting those function call interfaces into a network interface and splitting up your servers. However, doing this when it's needed means that you know where the split makes sense, and you won't waste time introducing components that don't need individual scaling. For God's sake, I saw a government department roll out an Azure Service Fabric application with dozens of components for an application with a few hundred users total. Not concurrent. Total.
Cube189 7y ago
Sounds exactly like a project I am in. Not to mention that we have like three "microservices" that access the exact same database. Oh, and a single Git repo. Yeah...
- durden0 7y ago
  Google technically has one big repo. It just depends on how you do it.
thisisjustajob 7y ago
Ok, then I'll just keep calm and continue with a distributed monolith. In 5 years it will be the mainstream and a new way to go. I can even imagine titles of newsletters: "Distributed monolith - the golden mean of software architecture".

foobarian 7y ago

> 2.5 Bonus, there is likely no way to meaningfully decouple the services. Service X can be "tolerant" of service Y's failure, but it cannot ever function without service Y. Nit: if service X could function without service Y, then it seems to follow service Y should not exist in the first place. And equivalently, the functionality of service Y before some microservice migration.

titanomachy 7y ago
Not necessarily. Maybe service X logs in a user and service Y sends them an email letting them know that someone logged in from a new location. Y adds value but it's not essential to X.
sobani 7y ago
Classic example is recommendations on a product page. If the personal recommendation service is not available / slow to respond, you might fall back on recommendations based on the best sellers or even fall back further by not giving recommendations at all. Recommendations are not necessary, and not showing them will significantly affect the bottom line, so you don't want to skip them if possible. But not showing the product page (in a timely manner), because the recommendation engine has a hiccup, is even worse for your bottom line.

mycall 7y ago

> Don't duplicate data without adding value to it. What about running multiple versions of a microservice in parallel -- don't each need their own yet separate databases that attempt to mirror each other as best they can?

aliswe 7y ago
Those are just different (versions of) interfaces for interacting with the same data source, ie no not several databases... But I'm no expert
SideburnsOfDoom 7y ago
No.
rubyn00bie 7y ago
I'm assuming you mean in production micro-service, if that's not the case, please elaborate a bit more... The short answer is "no," as succinctly stated by the, I assume from the name, majestic SideburnsOfDoom. The versions shouldn't _EVER_ be incompatible with each other. E.g. you need to rename a column. Do not: rename the column, e.g. `ALTER TABLE RENAME COLUMN...`. Because, your systems are going to break with the new schema. Do: Add a new column, with the new name, and migrate data to the new column, once it's good upgrade the rest of your instances, then drop the old column. Because, you can use both versions at the same time now without breaking anything. Yes, it can be a little tricky to get the data sync'd into the new column, but that's a lot less tricky than doing it for _every_ table and column.

rumanator 7y ago

> Most people think a micro-service architecture is a panacea because "look at how simple X is," but it's not that simple. It's now a distributed system, and very likely, it's a the worst-of-the-worst a distributed monolith. Distributed system are hard, I know, I do it. This line of argument fails to take into consideration any of the reasons why in general microservices are the right tool for the right job. Yes, it's challenging, and yes it's a distributes system. Yet, with microservices you actually are able to reuse specialized code, software packages, and even third-party services. That cuts down on a lot of dev time and cost, and makes the implementation of a lot of of POCs or even MVPs a trivial task. Take for example Celery. With Celery all you need to do to implement a queuable background task system that's trivially scalable is to write the background tasks, get a message broker up and running, launch worker instances, and that's it. What would you have to do to achieve the same goal with a monolith? Implement your own producer/consumer that runs on the same instance that serves requests? And aren't you actually developing a distributes system anyway?

ar_lan 7y ago
I'm completely in your camp, and I'm surprised by the lack of nuance HN seems to show (especially regarding micro-services & Kubernetes). There are many benefits to having microservices that people seem to forget because they think that everyone interested in microservices is interested in splitting their personal blog into 4 different services. They take coordination, good CICD, and a lot of forethought to ensure each service is cooperating in the ecosystem properly, but once established, it can do wonders to dev productivity.
- sterlind 7y ago
  I can't tell if my project is a monolith or microservices, but it's going well so far. We use a single scalable database instance as a message broker and persistence source, and have a common framework implements distributed algorithms that every service uses to expose "OS-like" constructs (actor activation, persistent collections, workflows, Git etc. All communication is done through independent protocols.. there's not much coupling between protocols (except for common ones like "schedule this job on this device"), so it's not really a cobweb of dependencies, but everything relies on that single database. I think if the database gets too overloaded I'll partition certain tree nodes across multiple masters (this is feasible because the framework doesn't rely on a single timestream). With the level of shared code (the framework) and the single database, it's somewhat monolithic but the actors themselves are quite well-behaved and independent on top of it.

yes_man 7y ago

I don't want advocate one way or another (micro vs. monoliths) because tomato tomato. However here are a few arguments in defense of microservices regarding these three signs you commented: 1. Microservices do not have some inherent property of having to duplicate data. You can have data in single source and deliver that data to anyone who needs it through an API. There are infinitely many caching solutions if you are worried about this becoming a bottleneck 2 and 2.5. There are tools for microservice architectures that counter these problems to a degree, mainstream example being containers and container orchestration (e.g. Docker and Kubernetes). One can even make an argument that microservices force you to build your systems so they are more robust than your monolith would be. If the argument for monoliths is that it's easier to maintain reliability when every egg is in one basket then you are putting all your bets into that basket and it becomes a black hole for developer and operations resources, as well as making operations evolution very slow 3. There are again tools for handling data and "syncing" (although I don't like the idea of having to "sync") the services, for example message queues / streaming processing platforms (e.g. Kafka). If some data or information might be of interest to multiple services, you should then push such data to a message queue and consume it from services that need it. The "syncing" problem sounds like something that arises when you start duplicating your data across services which shouldn't happen (see my argument on 1.) Again not to say microservices are somehow universally better. Just coming into defense of their core concepts when they get unfairly accused

abraxas 7y ago
The trouble is that this process of streaming events over Kafka or Kinesis means that subscribed microservices will be duplicating the bus data in their own way in their local databases. If one of them falls out of the loop for whatever reason you are in trouble. Now, there is a pattern called Event Sourcing (ES) which proposes that the source of truth should be the event bus itself and microservice databases are mere projections of this data. This is all good and well except it's very hard to implement in practice. If a microservice needs to replay all business events from months or years in the past it may take hours or days to do this. What about the business in the meantime? If it's a service that significantly reduces the usability of you application you effectively have a long downtime anyway. Transactional activity becomes incredibly hard in the microservices world with either 2 phased commit (only good for very infrequent transactions due to performance) or with so called Sagas (which are very complex to get right and maintain). Any company that isn't delivering its service to billions of users daily will likely suffer far more from microservices than they will benefit.

smoyer 7y ago

We've moved a lot of services into Kubernetes and broken things up into smaller and smaller micro-services. It definitely eliminates a lot of the complexity for developers ... but you trade it for operational complexity (e.g. routing, security, mis-matched client/server versions, resiliency when dependency isn't responding). I still believe that overall software quality is higher with micro-services (our Swagger documents serve as living ICDs), but don't kid yourself that you're going to save development time. And don't fall into the trap of shrinking your micro-services too small.

save_ferris 7y ago
That’s an encouraging story to hear. The thing I’ve noticed is that the costs of moving a poorly written monolith to a microservice architecture can be incredibly high. I also think that microservice design really needs to be thought through and scrutinized, because poorly designed microservices start to suck really quickly in terms of maintenance.
Aperocky 7y ago
You’ll have endless race conditions to deal with, even when storage is central and unique. We learned and is continuing learning that.
- idiocratic 7y ago
  Unique storage for multiple services sounds like a recipe for disaster. The purpose of splitting services, at least one of, is to decouple parts of the code at a fundamental level, including storage and overall ownership thereof. You're probably better served with a modular monolith if you really can't break storage up.
  - Aperocky 7y ago
    No, only one service is reading/writing, everything else just call that. Still, things get quite lost when it involves talking to multiple other teams and needing to keep everything in sync.

huherto 7y ago

I totally agree. Mostly we are doing microservices the wrong way. We are not drawing the boundaries correctly, they are too small, they have too many interdependencies, they don't really encapsulate the data, and you end up with many interdependencies. There is not enough guidance about sizing them. We are just building distributed monoliths. Which is great for cloud companies because they get to sell many boxes.

galaxyLogic 7y ago
Micro-services are just connected things which work together to accomplish something. But where are those connections described? In some tables somewhere. Maybe. Whereas if you write a single monolithic program its connections are described in code, preferably type-checked by a compiler. I think that gives you at least theoretically a better chance of understanding what are the things that connect, and how they connect. So if there was a good programming language for describing micro-services then that would probably resolve many management difficulties, and then the question would be simply do we get performance benefits from running on multiple processors.
- IggleSniggle 7y ago
  There is such a language, and it is Erlang.
  - jb3689 7y ago
    Literally this. Reading the article I kept thinking to myself "which is why Erlang/Elixir is great because it doesn't make you choose up front". It's wild that with how popular Elixir has gotten that it still isn't seen as a serious contender for many companies
  - rubyn00bie 7y ago
    I write Elixir professionally, Erlang/Elixir and BEAM is no one-stop solution to these problems either. It has tools to help you, but you can very easily end up in the same boat.
  - 7y ago

ignoramous 7y ago

> Simple tip: Don't duplicate data without adding value to it. Just don't. Not much is said about S3's design principles in public, but that one was one of them. Disclaimer: Recalling from memory.

jiggawatts 7y ago

The most basic thing I see people neglecting is that inserting a network protocol is really adding many additional components that weren't there before, often doubling or even tripling the amount of code, config, and documentation required. If there's a single large project that combines "A+B" modules with no networking and you split this into networked services, then you now have: 1) "A" Component 2) "B" Component 3) "A" Server 4) "A" Client So for example if you started off with a single "project" in your favorite IDE, you now have 4, give or take. You might be able to code-gen your server and client code out of a single IDL file or something, but generally speaking you're going to be writing code like "B -> A client -> A server -> A" no matter what instead of simply "B -> A". Now you have to worry about network reliability, back-pressure, queuing, retries, security, bandwidth, latency, round-trips, serialization, load-balancing, affinity, transactions, secrets storage, threading, and on and on... A simple function call translates to a rats nest of dependency injection, configuration reads, callbacks, and retry loops. Then if you grow to 4 or more components you have to start worrying about the topology of the interconnections. Suddenly you may need to add a service bus or orchestrator to reduce the number of point-to-point connections. This is not avoidable, because if you have less than 4 components, then why bother to break things out into micro services in the first place!? Now when things go wrong in all sorts of creative ways, some of which are likely still the subject of research papers, heaven help you with the troubleshooting. First, it'll be the brownouts that the load balancer doesn't correctly flag as a failure, then the priority inversions, then the queue filling up, and then it'll get worse from there as the load ramps up. Meanwhile nothing stops you having a monolithic project with folders called "A", "B", "C", etc... with simple function calls or OO interfaces across the boundaries. For 99.9% of projects out there this is the right way to go. And then, if your business takes off into the stratosphere, nothing stops you converting those function call interfaces into a network interface and splitting up your servers. However, doing this when it's needed means that you know where the split makes sense, and you won't waste time introducing components that don't need individual scaling. For God's sake, I saw a government department roll out an Azure Service Fabric application with dozens of components for an application with a few hundred users total. Not concurrent. Total.

Cube189 7y ago

Sounds exactly like a project I am in. Not to mention that we have like three "microservices" that access the exact same database. Oh, and a single Git repo. Yeah...

durden0 7y ago
Google technically has one big repo. It just depends on how you do it.

thisisjustajob 7y ago

Ok, then I'll just keep calm and continue with a distributed monolith. In 5 years it will be the mainstream and a new way to go. I can even imagine titles of newsletters: "Distributed monolith - the golden mean of software architecture".