4 ms·

In my experience, it's easy enough to have services which have a large blast radius themselves and can become points of failure for your entire ecosystem. I don

by whakim 6y ago

In my experience, it's easy enough to have services which have a large blast radius themselves and can become points of failure for your entire ecosystem. I don't find this to be a huge point of difference from a monolith, although of course it depends a lot on what you're working on. To pick on the OP a bit here (sorry!), they said that their entire legacy system "could go down because of issues to do with completely unrelated, non critical things like eg internal staff vacation hours." To me that sounds like poorly written software regardless of architecture. I can't imagine a scenario in any of the recent codebases I've worked on (microservices and monoliths both) where errors in what sounds like an internal CRUD tool would cause an entire production application to crash. I find it even harder to imagine if the application has a halfway decent test suite. To add to that, when you have hundreds of services running around and something goes wrong, it ends up being a lot harder to track down exactly what's happening. So when you do get that critical error, oftentimes the downtime is worsened. As for dev velocity, I find the claims of the microservice gospel a little bit exaggerated. Your layers of nested services all talk to each other, and any of them could be a point of failure. This isn't really all that different from calling another function in your monolithic app - you've just distributed that function call across a network boundary. You still need to know the callee's API, and you'll still spend a decent amount of time trying to understand the ways that the callee might fail. But you've also created a huge amount of additional developer work whenever you need to do something that spans the boundaries of existing services. I think microservices certainly have their advantages but a lot of the simplistic claims made by their biggest proponents only hold up prima facie.

dodobirdlord 6y ago
> In my experience, it's easy enough to have services which have a large blast radius themselves and can become points of failure for your entire ecosystem. I don't find this to be a huge point of difference from a monolith, although of course it depends a lot on what you're working on. It's a huge different because if some core critical service starts causing problems it's almost certainly because the last binary push was bad, and you roll it back. You only have to roll back that particular service any everything starts behaving correctly again. Moreover, you probably detected the problem in the first place when the rollout of that service began by replacing a single instance of the updated service with the new binary. Monitoring picks up a spike in errors/latency/database-load/whatever and the push is stopped and rolled back. Monoliths have inventive ways to address this problem without having to roll the entire binary back, like pushing patches or using feature flags, but few would argue that the microservice approach to handling bad pushes isn't superior. > To me that sounds like poorly written software regardless of architecture. I can't imagine a scenario in any of the recent codebases I've worked on (microservices and monoliths both) where errors in what sounds like an internal CRUD tool would cause an entire production application to crash. I find it even harder to imagine if the application has a halfway decent test suite. Easy enough with a sufficiently large codebase in C or C++. Somebody's parser encounters an input that was supposed to never happen and now it's off clobbering the memory of who-knows-what with garbage.
- whakim 6y ago
  If you find yourself routinely pushing a bad binary that has (1) passed your test suite; and (2) passed whatever manual QA process you have on sandbox/staging deployments, then I would again suggest the problem is the process and not the architecture. Not to mention that if the service is indeed critical, you're not rolling out a deployment to every production server at once (or you're low enough scale that it doesn't matter). Either way, easy enough to roll back a bad deploy before things get hairy regardless of architecture. Also, I'm not sure what kind of internal CRUD tools you're writing, but "malicious input" doesn't really seem likely to come from your coworkers.
  - dodobirdlord 6y ago
    Input doesn’t have to be malicious to cause problems, just unexpected. Separating memory-unsafe code into the finest granularity memory spaces possible is just good practice regardless of whether it’s microservices or just process isolation in a monolith.
  - bedobi 6y ago
    When you're dealing with a giant piece of software that has existed since the 90s, with hundreds of developers working on it at any given moment, what you're describing doesn't work. You can read about a famous example here https://news.ycombinator.com/item?id=18442941 https://news.ycombinator.com/item?id=18442941
    - whakim 6y ago
      Well, you began by saying that "monoliths are universally bad in every imaginable way." You're now saying that a select few poorly-maintained, 25-year old monoliths are bad, which is an entirely different claim. That doesn't really prove that microservices are any better (are 25-year old poorly maintained clusters of services really going to be an improvement?). Let alone "better in every imaginable way."

by whakim 6y ago

dodobirdlord 6y ago

> In my experience, it's easy enough to have services which have a large blast radius themselves and can become points of failure for your entire ecosystem. I don't find this to be a huge point of difference from a monolith, although of course it depends a lot on what you're working on. It's a huge different because if some core critical service starts causing problems it's almost certainly because the last binary push was bad, and you roll it back. You only have to roll back that particular service any everything starts behaving correctly again. Moreover, you probably detected the problem in the first place when the rollout of that service began by replacing a single instance of the updated service with the new binary. Monitoring picks up a spike in errors/latency/database-load/whatever and the push is stopped and rolled back. Monoliths have inventive ways to address this problem without having to roll the entire binary back, like pushing patches or using feature flags, but few would argue that the microservice approach to handling bad pushes isn't superior. > To me that sounds like poorly written software regardless of architecture. I can't imagine a scenario in any of the recent codebases I've worked on (microservices and monoliths both) where errors in what sounds like an internal CRUD tool would cause an entire production application to crash. I find it even harder to imagine if the application has a halfway decent test suite. Easy enough with a sufficiently large codebase in C or C++. Somebody's parser encounters an input that was supposed to never happen and now it's off clobbering the memory of who-knows-what with garbage.

whakim 6y ago
If you find yourself routinely pushing a bad binary that has (1) passed your test suite; and (2) passed whatever manual QA process you have on sandbox/staging deployments, then I would again suggest the problem is the process and not the architecture. Not to mention that if the service is indeed critical, you're not rolling out a deployment to every production server at once (or you're low enough scale that it doesn't matter). Either way, easy enough to roll back a bad deploy before things get hairy regardless of architecture. Also, I'm not sure what kind of internal CRUD tools you're writing, but "malicious input" doesn't really seem likely to come from your coworkers.
- dodobirdlord 6y ago
  Input doesn’t have to be malicious to cause problems, just unexpected. Separating memory-unsafe code into the finest granularity memory spaces possible is just good practice regardless of whether it’s microservices or just process isolation in a monolith.
- bedobi 6y ago
  When you're dealing with a giant piece of software that has existed since the 90s, with hundreds of developers working on it at any given moment, what you're describing doesn't work. You can read about a famous example here https://news.ycombinator.com/item?id=18442941 https://news.ycombinator.com/item?id=18442941
  - whakim 6y ago
    Well, you began by saying that "monoliths are universally bad in every imaginable way." You're now saying that a select few poorly-maintained, 25-year old monoliths are bad, which is an entirely different claim. That doesn't really prove that microservices are any better (are 25-year old poorly maintained clusters of services really going to be an improvement?). Let alone "better in every imaginable way."