6 ms·

Has anybody legitimately had good experiences with microservices these days? Especially if you're not starting with a monolith that has done well? My experience

Has anybody legitimately had good experiences with microservices these days? Especially if you're not starting with a monolith that has done well? My experience at several companies, several projects, has been that what was actually just a poorly built, inflexible, monolithic application becomes a poorly built, distributed, networked, polyglot application that is now 100x less flexible. Sure you can reimplement any microservice interface any time you want and drop in the replacement, but good luck figuring out where that was being called and how many unique and interesting ways it might break if you don't implement the edge cases in the same way. Microservices and patterns like the ones in this article really add a serious amount of cost in the form of complexity, and you had better understand much better than this article explains the actual ways that your new and distributed system might fail. What happens when this circuit breaker opens and you stop creating accounts, but your transaction circuit breaker hasn't opened and because you're CQRS, you've already told your client "201 Accepted"? Be ready to deal with that. In my experience, you're much better off knowing how to build and iterate on a scalable monolithic architecture first. Built something that's good and that you can iterate on pretty easily. Figure out how to profile monolithic applications for performance and squeeze more throughput out of your system before you try your hand at splitting that architecture up and adding the myriad of distributed systems problems on top of the essential complexity. Don't underestimate the value of a compiler error or a right-click-to-refactor IDE. You don't get those when suddenly everything is loosely coupled microservices over a service mesh. There is plenty of tech to master to manage a half decent networked monolith without adding your own accidental complexity to the equation. Just to add a reference to a primary inspiration, on top of my anecdotes: https://www.martinfowler.com/bliki/MonolithFirst.html https://www.martinfowler.com/bliki/MonolithFirst.html

kid_atticus 6y ago
At my previous workplaces: no it has always been an unmitigated disaster. And in one case I think moving from the original monolith to microservice based architecture killed the business. At my current workplace: yes! We have about a dozen separate "microservices" and it works pretty well actually. The main difference I can see is that the boundary layers are extremely well defined and completely asynchronous. There are no synchronous REST or RPC calls between services at all. And it actually works! I think the key is loose coupling in the extreme.
- andrewstuart2 6y ago
  > I think the key is loose coupling in the extreme. Yeah, I agree for sure. I think that's kind of the key whether you're building a distributed system or a monolith, though. e.g. built on small interfaces that are easy to implement. In general, I think the thing about microservices is that they're a lot more sensitive to initial direction because of the added complexity of major architecture changes.
- radicalbyte 6y ago
  Before anyone even touches "microservices" they should be forced to read Domain-Driven Design. If you do not have a "Bounded Context" and are not splitting your "microservices" by that then you're doing it very wrong. A lot of the benefits I've seen people using "microservices" to fix what are just horrible design issues in their framework / codebase of choice.
  - matt-attack 6y ago
    Is that a book? Would love a reference.
    - andrenth 6y ago
      Domain-driven design by Eric Evans is the usual reference. There are some other well-regarded books on the subject too, e.g. by Vernon Vaughn.
- ratww 6y ago
  Exact same experience here. At my previous workplace it was a disaster. They hired too many developers in order to grow, and that led to too many people working on the same codebase. Microservices were adopted to solve that. After two years our product was exactly the same as before, except it had much more issues with latency and errors. Headcount was many times bigger, though. The problem was microservices being adopted for political rather than technical reasons. They wanted teams to own their stuff, but the boundaries were too fuzzy. We had a lot of duplicated data because leadership never decided which team owned what. Also, some features had to be made by duplicating data in two different services: if team #2 couldn't convince team #1 to send certain events to their microservice, team #2 had to duplicate the whole DB table using Kafka (Very appropriate, right?) and do a cron job. Another problem happened was when they banned "microservices" but asked to make "well-sized services". New teams had a limit of how many services they could have and to get around the limitation they started having to mix completely unrelated functionality in a single microservice (and yep, that was sanctioned by leadership). My current company has Microservices, but they exist purely for technical reasons and are extremely loosely coupled, so it works great.
jordwest 6y ago
In general I agree, but I think microservices applied in the right context can be useful. For example, say you have a system that manages bookings for a really popular musical act and frequently crashes with load when tickets go on sale. In that case it could be helpful to have the frontend service that deals with the booking requests isolated so it doesn't bring down the rest of the system. The problem is microservices for the sake of microservices, which seems to be the majority. When a design pattern is applied just because somebody is familiar with it and not because it's the best fit to the problem at hand - that's when things get messy.
- andrewstuart2 6y ago
  Maybe I'm just not up on the startup or smaller company scene, but isn't it still pretty important to acquire those customers before you worry about whether or not they'll take your systems down? I'd expect that to give you plenty of time to build the monolith, figure out if it's too brittle, too unscaleable, etc, and maybe ground-up rewrite at low cost before you solidify too much by breaking things up Even so, let's say you need to scale very very quickly right away, it's so much easier to profile and optimize a single process. I've been successful in optimizing e.g. a Go library from 100 KB/s to 700MB/s, with about two weeks of work (`pprof` and `go bench` are amazing tools), which then handled 20k tps in an application, and could have scaled out but didn't actually need to. I've also gotten millions of rows in a postgres database to be full-text searchable in under 10ms, also with maybe only a week of work. I can't readily think of a time (and I've had plenty of opportunity these days) where I've successfully diagnosed and optimized underperforming distributed systems, despite having spent months discussing and attempting to measure performance issues and trying various "yeah maybe that could work" fixes. Granted, It's almost certainly always been the case of microservices for the sake of microservices, but I always come back to the lack of tooling (no compiler to tell you that you broke some contract), the siren call of polyglot microservices that makes it a lot harder to build common tooling, and the temptation to prematurely send microservices to different teams (further adding to the lack of agility), as really solid reasons to stay smaller as long as possible.
  - jordwest 6y ago
Silhouette 6y ago
This often feels like a meme: "I had a scalability problem, so I made everything distributed and asynchronous. Now I have three problems!" It's not that the idea is fundamentally broken, but I suspect a lot of people dramatically misjudge the trade-offs. Never underestimate the power and efficiency of one big, fast server, nor the added complexity that is introduced by taking loose coupling to extreme levels.
Cthulhu_ 6y ago
I've worked with two major (well, relatively) microservices architectures and honestly it made things many times more complicated and expensive than they should have. In the one, I came in after a year and a half and they still had nothing to show for other than some marketing (this was fairly straightforward e-commerce). They tried to bolt Java applications together using Amazon Kinesis (which isn't intended as a service bus) and we quickly ran into issues because the stock / inventory service was separate from the order processing service, so any incoming order would have to do a weird round trip to confirm stock before it could confirm the order to the end-user. To name but one example. The other was for a shipping company, where every 'model' (users, addresses, shipments, etc) was put in its own microservice. That one started out with NodeJS services before - for some reason, probably political / hiring related) migrating to Scala, only adding to the complexity of what, in essence, was a CRUD application where the heavy lifting was still being done in the legacy back-end / mainframe systems. I don't get it. Glad I'm working on a project right now where I have full control over things, it's also a CRUD project but I can build it as a straight monolith. One other anecdote I have is where we built a serverless application. It too was e-commerce, but the heavy lifting was done by as-a-services; we used Commercetools as e-commerce back-end as as service, Contentful as CMS-as-a-service, Adyen for payment processing, and we used a heap (about 30) 'serverless' functions deployed in Netlify as the glue. Seemed to work all right, better than the microservices attempt (because the individual functions didn't have to talk to one another; they responded to either user / client requests, or events generated from one of the services doing the heavy lifting). Anyway if I had to do ALL of those again (and if I actually had a say in it, lol) I'd just write them as plain old Java monoliths. The reason that's not being considered an option much is because it's boring, and highly paid software engineers don't want to do boring stuff. Second reason is that if you have boring code, you get mediocre developers. That's why in some projects I've seen them push for Scala instead of plain Java (which would've done just fine), because if you pick Scala you weed out a good 95% or so percentile of developers - that is, the reasoning being that one already has to be a good developer to start to comprehend Scala. Of course, it backfired because these developers are so 'good' that they're also stubborn and will do things their own way, which may not be the same way as other devs. That may have just been my limited experience though, IDK.
- 6y ago
deepstack 6y ago
>Has anybody legitimately had good experiences with microservices these days? Especially if you're not starting with a monolith that has done well? My experience at several companies, several projects, has been that what was actually just a poorly built, inflexible, monolithic application becomes a poorly built, distributed, networked, polyglot application that is now 100x less flexible. Sure you can reimplement any microservice interface any time you want and drop in the replacement, but good luck figuring out where that was being called and how many unique and interesting ways it might break if you don't implement the edge cases in the same way. It depends on the team. If there is a large team, then yes micro-services could be a good choice. It is not silver bullet. In the end it comes down the the quality of developers. With a group of good quality developers then a good product can be build with micro-services or monolith. With functional programming or OO or declarative. Stop using marketing people who uses buzz word without understanding underling tech.
mytailorisrich 6y ago
People are always looking for a silver bullet but that obviously does not exist. There are technical pros and cons to both monoliths and microservices. Either way, the most important is to have a good, functional team. A team that has built a good monolith will probably build good microservices. A team that has built a messy monolith will probably do even worse with microservices. My view is that microservices are useful at scale and that a well-designed monolith can be split into microservices as needed later on.
TacoSteemers 6y ago
> Has anybody legitimately had good experiences with microservices these days? I have, but in a specific environment. Almost all of the final requirements were known in advance. Once installed, the final version would not receive any updates other than critical bug fixes. Even in that environment it became clear that any issue is a lot more work to debug and it will probably be the maintainer of the other service that will have to do troubleshooting. The places where interaction occurs become very important, but these are now in separate projects. It is easy to mess up code that you cannot see.
UK-Al05 6y ago
Yes at a large ecommerce company with many teams. Each team is able to treat their own microservice as a product, with its own release schedule and roadmap. Decoupled from other teams. If your having to touch multiple services to implement changes, you've sliced your services in the wrong way.
amelius 6y ago
Isn't a modern Linux system not a bunch of microservices?
- zbentley 6y ago
  Not really, it's a bunch of tightly connected non-networked components (the OS and userland stack--even with dbus increasingly in the mix, the reliability/failure domains are really different than those of web services).
alexchamberlain 6y ago
Yes; for us, it's a refinement of Service Oriented Architecture rather than hype mind. Schemas (with well defined major and minor versioning), routing (with timeouts) and service discovery (in the more general sense of being able to find the service you need) are all essential components. I fear Conway's law is at it's strongest with microservices and actually getting teams to collaborate is the biggest challenge.
flarg 6y ago
Monoliths always sound like a good idea but they always lead to some low code platform garbage that the org has settled upon. Microservices stay as proper code.
- fendy3002 6y ago
  Gotta disagree with you. A well built / designed / simple monolith beats any poorly implemented microservice of the same domain problem anytime. If you cannot separate the domain problem correctly in microservice architecture, be prepared for the worst.
  - flarg 6y ago
    Gotta agree with you - but my point was 'low code garbage' - this is what happens in enterprises if a monolith is the goal; if you rule out monolith then you ain't suffering with Pega, Appian, Salesforce etc. Monoliths are great but they are also an open manhole you need to close in enterprise development.
    - fendy3002 6y ago
      A giant shithole is what you get when you create a monolith that can fit into many business processes / cases. One additional case to handle introduce many more complexity to application.
      - flarg 6y ago
        For sure. Next after selecting a low code monolithic stack we are informed that we can reuse low code solution, which is like a doubling down on the monolithic approach. But of course with 30 consultants.
runT1ME 6y ago
>Has anybody legitimately had good experiences with microservices these days? Yes. I work for a company that started out with a few separate monoliths that were hacked together under intense time pressure by some smart folks that didn't anticipate the scale they were going to be operating under. As the team expanded and more talent was added, they were able to partition up some of the functionality into separate services. When I joined we had a small team with a mixture of legacy systems written in Java, Perl, PHP, Scala and Python. Some of the microservices were poorly done, others were fine. The new teams were able to completely overhaul each service one at a time to where they were stable and needing almost no maintenance/babysitting. Would microservices have been a good starting decision for this company, given its current level of engineering talent? Probably not. However, take a bootstrapped team crunched for time, with some foresight to know that there would need to be rewrites but not certain where, microservices can end up mitigating a lot of unknowable future issues.
Tainnor 6y ago
I think one of the reasons why so many devs immediately want to jump to microservices is because they haven't learned how to architect an application well and have been burned by entangled responsibilities. E.g. many Rails apps, if you're not super disciplined, devolve into a hot mess where your database is everywhere (even in your views), tests take forever to load because there is no isolation, framework updates take forever and so on. Other frameworks like Spring are a bit less bad, but they still don't force you to write properly layered architectures, use domain driven design, etc. Of course, the idea that this is magically fixed if you split things up into smaller services, is a bit misguided. Yes, API contracts are more explicit (especially in dynamically typed languages where you otherwise lack any sort of contract between pieces of your application), but there are still all sorts of implicit dependencies. At some point, someone might say "ah let's have service B also use the database of service A because of this one feature", or, "you can only call this service after you have called that other service in exactly that way" etc. and then you slowly start creating that distributed monolith...

kid_atticus 6y ago

At my previous workplaces: no it has always been an unmitigated disaster. And in one case I think moving from the original monolith to microservice based architecture killed the business. At my current workplace: yes! We have about a dozen separate "microservices" and it works pretty well actually. The main difference I can see is that the boundary layers are extremely well defined and completely asynchronous. There are no synchronous REST or RPC calls between services at all. And it actually works! I think the key is loose coupling in the extreme.

andrewstuart2 6y ago
> I think the key is loose coupling in the extreme. Yeah, I agree for sure. I think that's kind of the key whether you're building a distributed system or a monolith, though. e.g. built on small interfaces that are easy to implement. In general, I think the thing about microservices is that they're a lot more sensitive to initial direction because of the added complexity of major architecture changes.
radicalbyte 6y ago
Before anyone even touches "microservices" they should be forced to read Domain-Driven Design. If you do not have a "Bounded Context" and are not splitting your "microservices" by that then you're doing it very wrong. A lot of the benefits I've seen people using "microservices" to fix what are just horrible design issues in their framework / codebase of choice.
- matt-attack 6y ago
  Is that a book? Would love a reference.
  - andrenth 6y ago
    Domain-driven design by Eric Evans is the usual reference. There are some other well-regarded books on the subject too, e.g. by Vernon Vaughn.
ratww 6y ago
Exact same experience here. At my previous workplace it was a disaster. They hired too many developers in order to grow, and that led to too many people working on the same codebase. Microservices were adopted to solve that. After two years our product was exactly the same as before, except it had much more issues with latency and errors. Headcount was many times bigger, though. The problem was microservices being adopted for political rather than technical reasons. They wanted teams to own their stuff, but the boundaries were too fuzzy. We had a lot of duplicated data because leadership never decided which team owned what. Also, some features had to be made by duplicating data in two different services: if team #2 couldn't convince team #1 to send certain events to their microservice, team #2 had to duplicate the whole DB table using Kafka (Very appropriate, right?) and do a cron job. Another problem happened was when they banned "microservices" but asked to make "well-sized services". New teams had a limit of how many services they could have and to get around the limitation they started having to mix completely unrelated functionality in a single microservice (and yep, that was sanctioned by leadership). My current company has Microservices, but they exist purely for technical reasons and are extremely loosely coupled, so it works great.

jordwest 6y ago

In general I agree, but I think microservices applied in the right context can be useful. For example, say you have a system that manages bookings for a really popular musical act and frequently crashes with load when tickets go on sale. In that case it could be helpful to have the frontend service that deals with the booking requests isolated so it doesn't bring down the rest of the system. The problem is microservices for the sake of microservices, which seems to be the majority. When a design pattern is applied just because somebody is familiar with it and not because it's the best fit to the problem at hand - that's when things get messy.

andrewstuart2 6y ago
Maybe I'm just not up on the startup or smaller company scene, but isn't it still pretty important to acquire those customers before you worry about whether or not they'll take your systems down? I'd expect that to give you plenty of time to build the monolith, figure out if it's too brittle, too unscaleable, etc, and maybe ground-up rewrite at low cost before you solidify too much by breaking things up Even so, let's say you need to scale very very quickly right away, it's so much easier to profile and optimize a single process. I've been successful in optimizing e.g. a Go library from 100 KB/s to 700MB/s, with about two weeks of work (`pprof` and `go bench` are amazing tools), which then handled 20k tps in an application, and could have scaled out but didn't actually need to. I've also gotten millions of rows in a postgres database to be full-text searchable in under 10ms, also with maybe only a week of work. I can't readily think of a time (and I've had plenty of opportunity these days) where I've successfully diagnosed and optimized underperforming distributed systems, despite having spent months discussing and attempting to measure performance issues and trying various "yeah maybe that could work" fixes. Granted, It's almost certainly always been the case of microservices for the sake of microservices, but I always come back to the lack of tooling (no compiler to tell you that you broke some contract), the siren call of polyglot microservices that makes it a lot harder to build common tooling, and the temptation to prematurely send microservices to different teams (further adding to the lack of agility), as really solid reasons to stay smaller as long as possible.
- jordwest 6y ago

Silhouette 6y ago

This often feels like a meme: "I had a scalability problem, so I made everything distributed and asynchronous. Now I have three problems!" It's not that the idea is fundamentally broken, but I suspect a lot of people dramatically misjudge the trade-offs. Never underestimate the power and efficiency of one big, fast server, nor the added complexity that is introduced by taking loose coupling to extreme levels.

Cthulhu_ 6y ago

I've worked with two major (well, relatively) microservices architectures and honestly it made things many times more complicated and expensive than they should have. In the one, I came in after a year and a half and they still had nothing to show for other than some marketing (this was fairly straightforward e-commerce). They tried to bolt Java applications together using Amazon Kinesis (which isn't intended as a service bus) and we quickly ran into issues because the stock / inventory service was separate from the order processing service, so any incoming order would have to do a weird round trip to confirm stock before it could confirm the order to the end-user. To name but one example. The other was for a shipping company, where every 'model' (users, addresses, shipments, etc) was put in its own microservice. That one started out with NodeJS services before - for some reason, probably political / hiring related) migrating to Scala, only adding to the complexity of what, in essence, was a CRUD application where the heavy lifting was still being done in the legacy back-end / mainframe systems. I don't get it. Glad I'm working on a project right now where I have full control over things, it's also a CRUD project but I can build it as a straight monolith. One other anecdote I have is where we built a serverless application. It too was e-commerce, but the heavy lifting was done by as-a-services; we used Commercetools as e-commerce back-end as as service, Contentful as CMS-as-a-service, Adyen for payment processing, and we used a heap (about 30) 'serverless' functions deployed in Netlify as the glue. Seemed to work all right, better than the microservices attempt (because the individual functions didn't have to talk to one another; they responded to either user / client requests, or events generated from one of the services doing the heavy lifting). Anyway if I had to do ALL of those again (and if I actually had a say in it, lol) I'd just write them as plain old Java monoliths. The reason that's not being considered an option much is because it's boring, and highly paid software engineers don't want to do boring stuff. Second reason is that if you have boring code, you get mediocre developers. That's why in some projects I've seen them push for Scala instead of plain Java (which would've done just fine), because if you pick Scala you weed out a good 95% or so percentile of developers - that is, the reasoning being that one already has to be a good developer to start to comprehend Scala. Of course, it backfired because these developers are so 'good' that they're also stubborn and will do things their own way, which may not be the same way as other devs. That may have just been my limited experience though, IDK.

6y ago

deepstack 6y ago

>Has anybody legitimately had good experiences with microservices these days? Especially if you're not starting with a monolith that has done well? My experience at several companies, several projects, has been that what was actually just a poorly built, inflexible, monolithic application becomes a poorly built, distributed, networked, polyglot application that is now 100x less flexible. Sure you can reimplement any microservice interface any time you want and drop in the replacement, but good luck figuring out where that was being called and how many unique and interesting ways it might break if you don't implement the edge cases in the same way. It depends on the team. If there is a large team, then yes micro-services could be a good choice. It is not silver bullet. In the end it comes down the the quality of developers. With a group of good quality developers then a good product can be build with micro-services or monolith. With functional programming or OO or declarative. Stop using marketing people who uses buzz word without understanding underling tech.

mytailorisrich 6y ago

People are always looking for a silver bullet but that obviously does not exist. There are technical pros and cons to both monoliths and microservices. Either way, the most important is to have a good, functional team. A team that has built a good monolith will probably build good microservices. A team that has built a messy monolith will probably do even worse with microservices. My view is that microservices are useful at scale and that a well-designed monolith can be split into microservices as needed later on.

TacoSteemers 6y ago

> Has anybody legitimately had good experiences with microservices these days? I have, but in a specific environment. Almost all of the final requirements were known in advance. Once installed, the final version would not receive any updates other than critical bug fixes. Even in that environment it became clear that any issue is a lot more work to debug and it will probably be the maintainer of the other service that will have to do troubleshooting. The places where interaction occurs become very important, but these are now in separate projects. It is easy to mess up code that you cannot see.

UK-Al05 6y ago

Yes at a large ecommerce company with many teams. Each team is able to treat their own microservice as a product, with its own release schedule and roadmap. Decoupled from other teams. If your having to touch multiple services to implement changes, you've sliced your services in the wrong way.

amelius 6y ago

Isn't a modern Linux system not a bunch of microservices?

zbentley 6y ago
Not really, it's a bunch of tightly connected non-networked components (the OS and userland stack--even with dbus increasingly in the mix, the reliability/failure domains are really different than those of web services).

alexchamberlain 6y ago

Yes; for us, it's a refinement of Service Oriented Architecture rather than hype mind. Schemas (with well defined major and minor versioning), routing (with timeouts) and service discovery (in the more general sense of being able to find the service you need) are all essential components. I fear Conway's law is at it's strongest with microservices and actually getting teams to collaborate is the biggest challenge.

flarg 6y ago

Monoliths always sound like a good idea but they always lead to some low code platform garbage that the org has settled upon. Microservices stay as proper code.

fendy3002 6y ago
Gotta disagree with you. A well built / designed / simple monolith beats any poorly implemented microservice of the same domain problem anytime. If you cannot separate the domain problem correctly in microservice architecture, be prepared for the worst.
- flarg 6y ago
  Gotta agree with you - but my point was 'low code garbage' - this is what happens in enterprises if a monolith is the goal; if you rule out monolith then you ain't suffering with Pega, Appian, Salesforce etc. Monoliths are great but they are also an open manhole you need to close in enterprise development.
  - fendy3002 6y ago
    A giant shithole is what you get when you create a monolith that can fit into many business processes / cases. One additional case to handle introduce many more complexity to application.
    - flarg 6y ago
      For sure. Next after selecting a low code monolithic stack we are informed that we can reuse low code solution, which is like a doubling down on the monolithic approach. But of course with 30 consultants.

runT1ME 6y ago

>Has anybody legitimately had good experiences with microservices these days? Yes. I work for a company that started out with a few separate monoliths that were hacked together under intense time pressure by some smart folks that didn't anticipate the scale they were going to be operating under. As the team expanded and more talent was added, they were able to partition up some of the functionality into separate services. When I joined we had a small team with a mixture of legacy systems written in Java, Perl, PHP, Scala and Python. Some of the microservices were poorly done, others were fine. The new teams were able to completely overhaul each service one at a time to where they were stable and needing almost no maintenance/babysitting. Would microservices have been a good starting decision for this company, given its current level of engineering talent? Probably not. However, take a bootstrapped team crunched for time, with some foresight to know that there would need to be rewrites but not certain where, microservices can end up mitigating a lot of unknowable future issues.

Tainnor 6y ago

I think one of the reasons why so many devs immediately want to jump to microservices is because they haven't learned how to architect an application well and have been burned by entangled responsibilities. E.g. many Rails apps, if you're not super disciplined, devolve into a hot mess where your database is everywhere (even in your views), tests take forever to load because there is no isolation, framework updates take forever and so on. Other frameworks like Spring are a bit less bad, but they still don't force you to write properly layered architectures, use domain driven design, etc. Of course, the idea that this is magically fixed if you split things up into smaller services, is a bit misguided. Yes, API contracts are more explicit (especially in dynamically typed languages where you otherwise lack any sort of contract between pieces of your application), but there are still all sorts of implicit dependencies. At some point, someone might say "ah let's have service B also use the database of service A because of this one feature", or, "you can only call this service after you have called that other service in exactly that way" etc. and then you slowly start creating that distributed monolith...