8 ms·

I’ve worked for decades now as both a developer and an SRE and have never once thought either “man I wish these microservices were monolithic” nor “man this mon

I’ve worked for decades now as both a developer and an SRE and have never once thought either “man I wish these microservices were monolithic” nor “man this monolith is so great I’m glad it’s not a collection of microservices”. These kinds of articles seem to be written for people who work in environments I’ve never even heard of let alone experienced.

honkycat 4y ago
There are a lot of small shops who aren't dealing with scale, and assume that what works for them works for everyone. Personally, I think micro-services should be approached very carefully but I understand the idea of them.
- jnash 4y ago
  Companies like Jane Street use Monoliths to handle millions of transactions per second. How many transactions per second do you have?
DJBunnies 4y ago
I think that all the time. Or more specifically, "I wish these services were a monolith so we could have better type checks/easier logging/debugging."
- bluefirebrand 4y ago
  "I wish these services were a monolith so there was any amount of discoverability at all, and I don't have to beg for time from busy people on other teams to help me use their undocumented APIs"
  - arinlen 4y ago
    > I wish these services were a monolith so there was any amount of discoverability at all, Why do you conflate microservices with discoverability? What's wrong with simply calling a web service? > (...) and I don't have to beg for time from busy people on other teams to help me use their undocumented APIs And you believe that the same hypothetical team you claim doesn't document their API would all of a sudden documented all its internal?
    - bluefirebrand 4y ago
      If it was a monolith I would have access to the same codebase, so I could just go look at stuff in the code? Versus microservices where I might not have permissions to look at their repos.
      - thaneross 4y ago
        It's a fascinating set of priories that lead to these policies. "We trust you enough to run your code in production but we can't let you read another team's code because you might steal it. Yes, the odds that anyone wants to steal the code for this mundane microservice are staggeringly low... and yes you're going to be less productive and more prone to serious bugs by being in the dark, but that's your problem."
        bluefirebrand 4y ago
        I honestly don't even know if it's "you might steal it" It's just this vague notion of "if you aren't contributing to this repo you don't need to access it" or something.
ericbarrett 4y ago
I've seen the degenerate states of both extremes: - a monolith with too many engineers contributing: no continuous deployment but rather "release day," which was a shitshow; said day was an extensive process where hundreds of engineers (anybody who was on git blame) had to be online and sitting by for four hours while it rolled; the release team often had to hand-revert bad patches; bullying of engineers who "broke the build" reached levels that would raise HR eyebrows; there were still often rollbacks and site breakages. - microservice hell: there were often 2-3 APIs for the same service; platform engineering (for the protobuf RPC generation) had to support five different languages; security auditing was NP-hard; every team had its own release process; services that were still highly used were "deprecated" and left languishing for years until somebody took up the mantle and released their own parallel service that did nearly the same thing but with a different API, so now other services had to use both; etc. "But that wouldn't happen at $my_company, we know the pitfalls and we'd never be that bad at engineering!" Sure...sure. That's what these companies said to themselves too :)
- hahahKk 4y ago
- otagekki 4y ago
  I have also known both albeit on a lesser scale. At a company we as a dev team had to maintain a constellation of desktop applications and homemade ETL Airflow-like software that scaled really badly. For the time I'd been there I had managed to automate deployment almost totally on my own. Sadly it had been somewhat controversial as it was done at the expense of other development tasks (my manager agreed but not his nontechnical manager), and probably too late too. Before I had had enough grasp on the complicated context we had to deliver on a big project and needless to say it was a shitshow and we developers took basically all the blame, with pretty much nothing to say back. The tech stack and code had been in a state of neglect for 8+ years but that was not an acceptable answer at that time, it seemed, and I'd only been there for 6 months when it happened. On the other end of the spectrum, at the above's parent company, I had to maintain several major versions of an API on several cloud regions. Undocumented peculiarities regarding the object storage that was specific to one region caused a production incident that took the dev and support team 3 days to untangle and solve. I have similar stories on monorepo vs. multirepo too
throwaway894345 4y ago
Same. I keep hearing that people have had bad experiences with microservices, but I'm not sure what those bad experiences are. I certainly don't relate to them. We had a monolith, and it devolved into a mess--managers would insist on people taking dependencies on stuff they didn't own in the name of expedience ("yeah, we'll totally circle back and do it the right way lol"). Microservices kept things neat by making dumb things hard. We were also able to deploy individual services much more frequently without needing to coordinate with a bunch of other teams, and builds, tests, etc were much faster helping shorten our iteration loops. Local development environments were a bit more tedious in certain cases, but that was the only issue I recall.
- willcipriano 4y ago
  Microservices suck when you need to make a process faster, so you run a profiler and figure out that 90% of the workload is handling the http requests between the services.
  - bb88 4y ago
    I feel like HTTP is too heavyweight for microservices. For monoliths, it's great. And the only way to proceed in your case is to convert microservices into mini-monoliths. Particularly the authentication side, which needs to be revalidated with every request. gRPC might be better, but supposedly so was SOAP, CORBA, DCOM, XMLRPC, and finally REST.
    - throwaway894345 4y ago
      How does gRPC make authentication lighter-weight? Presumably you still have a token in either case which needs to be validated? There aren't many places where parsing HTTP requests is really a significant bottleneck, and I'd guess the majority of the remainder are an architectural problem (someone using a "get" endpoint in a tight loop instead of a "list" endpoint or some such).
      - bb88 4y ago
atleta 4y ago
It still depends on the organization size (or the size of the user base, or the complexity of the product). So you may only have experience with services where it was the right choice. I did see two instances where it wasn't. (I mainly work with/for small companies, startups.) In one instance I was called in as a tech lead/expert for a small startup having a kind of a product/software crisis. They've been working on their service for a year or too (yes, way too long) and 2-3 months before the planned release at some random conference (Slush, TNW or whatnot) one of the developers figured out that the whole codebase was a piece of shit and there was no way they could be ready in time, but they should rewrite it as a collection of node microservices and that could work. The monolith they had was PHP, just to add to the fun, so switching over would mean switching languages too. The guy, he was a smart and motivated chap, even started implementing one service in his free time, the user registration/user handling I think (the least important and least complex one, of course) which somehow screwed up the monotlith and made it start crashing. (Or so they said, I don't know what was up with that.) Obviously, it was a 100% stupid idea and we went on with fixing their development process (starting to do scrum and teaching the stakeholders that they need to stop phoning the developers directly and asking for features, fixes, introducing automated testing, etc.) Oh, and it was a team of 2-2.5 people. (With the group manager doing some backend work too, but also managing another, totally unrelated project for another client.) The other one was a bit different story, where I just shared my insights over a call. A guy I've known took over a project that was built by a small team (2-5 people, can't remember) for a startup and wanted some external opinion for himself and the founder. That one was built as a set of microservices but they did have all kinds of stability issues. The idea was that it had to be very-very-super scalable. Because, you know you launch and they will come and there's nothing worse than not being able to handle the load. Except, there is: they had been building the thing for over 2 years back then. It was an online medical consultation solution (you describe your problem, pick a doctor, do a f2f call and pay by the time). The funny thing is that I've built a very similar system, as a startup cofounder, 3-4 years earlier for psychology consultation with the help of 2 other guys, who didn't even work full time (one of them came after the first one quit). The MVP was up in, I think, 2 maybe 3 months. Ours was a monolith-ish thing and theirs definitely looked better, maybe scaled better and would have been cheaper to operate at scale (we used an external service for the video calls). But ours was a lot cheaper to build and launch and we could test (validate) our solution a lot earlier with real customers. If it worked out, we could have started breaking it down into multiple services as/if/when needed.
mbrodersen 4y ago
A friend of mine is working on a 20 years old Microservices application. It is literally the worst piece of garbage system I have ever heard about. I am sure my friend would have loved for it to be a monolith instead of 50+ services with massively complicated distributed dependencies and timing issues.
nailer 4y ago
Really? You’ve never seen an over-engineered microservices design that could be a much simpler express app?
- fud101 4y ago
  What is an express app?
  - nailer 4y ago
    An app in express.js, a web framework.
KronisLV 4y ago
When I'm forced to work with a monolith that has 4000+ source files and takes a few minutes to even start up, and has about 20 configuration files that all need to be setup properly for it to even start, I wish for microservices. When I'm forced to work with microservices that need Skaffold and Helm charts just to run locally, but with the configuration in the monorepo being kind of mismanaged and strewn around a bunch of folders with no documentation, in addition to debugging in the IDE not working because nobody set it all up, I wish for monoliths. Really, you can have good monolithic systems and you can have good microservices as well, in addition to something in the middle: https://blog.kronis.dev/articles/modulith-because-we-need-to-scale-but-we-also-cannot-afford-micro-services https://blog.kronis.dev/articles/modulith-because-we-need-to... (actually the first blog post that I wrote, the casual language probably shows its age). But there can also be plenty of poorly developed projects with either. It just so happens that people hate monoliths more in the mainstream culture because most of the older and worse projects out there were monolithic, much like many hate Java and other languages because of being exposed to bad legacy projects written in them: https://earthly.dev/blog/brown-green-language/ https://earthly.dev/blog/brown-green-language/ Just wait for 5-10 years and people's disposition towards both monoliths and microservices will even out, the advantages and disadvantages of either will become clearly recognized and the hype will shift towards something like serverless. Much like now we know the advantages and disadvantages of languages with/without GC as well as higher/lower abtraction levels pretty well (consider Python vs Rust, for example). Maybe things will slow down a bit because Kubernetes will also become a bit easier to use, possibly thanks to projects like K3s, K0s, Portainer and Rancher.
- jnash 4y ago
  Nothing stops microservices from having the same problems. Plus all the extra problems you get with distributed systems (timing issues, logical dependencies etc.)
- barking_biscuit 4y ago
  It turns out the secret is just not writing crappy software and having crappy processes. It turns out the caveat is that the conditional probability of those two things is vanishingly small, so regardless of monolith or microservices, most devs are exposed to crappy stuff and because their sample sizes are tiny they assume well we just did it wrong, when in fact they did it average and average is crappy for any and all architectures.

honkycat 4y ago

There are a lot of small shops who aren't dealing with scale, and assume that what works for them works for everyone. Personally, I think micro-services should be approached very carefully but I understand the idea of them.

jnash 4y ago
Companies like Jane Street use Monoliths to handle millions of transactions per second. How many transactions per second do you have?

DJBunnies 4y ago

I think that all the time. Or more specifically, "I wish these services were a monolith so we could have better type checks/easier logging/debugging."

bluefirebrand 4y ago
"I wish these services were a monolith so there was any amount of discoverability at all, and I don't have to beg for time from busy people on other teams to help me use their undocumented APIs"
- arinlen 4y ago
  > I wish these services were a monolith so there was any amount of discoverability at all, Why do you conflate microservices with discoverability? What's wrong with simply calling a web service? > (...) and I don't have to beg for time from busy people on other teams to help me use their undocumented APIs And you believe that the same hypothetical team you claim doesn't document their API would all of a sudden documented all its internal?
  - bluefirebrand 4y ago
    If it was a monolith I would have access to the same codebase, so I could just go look at stuff in the code? Versus microservices where I might not have permissions to look at their repos.
    - thaneross 4y ago
      It's a fascinating set of priories that lead to these policies. "We trust you enough to run your code in production but we can't let you read another team's code because you might steal it. Yes, the odds that anyone wants to steal the code for this mundane microservice are staggeringly low... and yes you're going to be less productive and more prone to serious bugs by being in the dark, but that's your problem."
      - bluefirebrand 4y ago
        I honestly don't even know if it's "you might steal it" It's just this vague notion of "if you aren't contributing to this repo you don't need to access it" or something.

ericbarrett 4y ago

I've seen the degenerate states of both extremes: - a monolith with too many engineers contributing: no continuous deployment but rather "release day," which was a shitshow; said day was an extensive process where hundreds of engineers (anybody who was on git blame) had to be online and sitting by for four hours while it rolled; the release team often had to hand-revert bad patches; bullying of engineers who "broke the build" reached levels that would raise HR eyebrows; there were still often rollbacks and site breakages. - microservice hell: there were often 2-3 APIs for the same service; platform engineering (for the protobuf RPC generation) had to support five different languages; security auditing was NP-hard; every team had its own release process; services that were still highly used were "deprecated" and left languishing for years until somebody took up the mantle and released their own parallel service that did nearly the same thing but with a different API, so now other services had to use both; etc. "But that wouldn't happen at $my_company, we know the pitfalls and we'd never be that bad at engineering!" Sure...sure. That's what these companies said to themselves too :)

hahahKk 4y ago
otagekki 4y ago
I have also known both albeit on a lesser scale. At a company we as a dev team had to maintain a constellation of desktop applications and homemade ETL Airflow-like software that scaled really badly. For the time I'd been there I had managed to automate deployment almost totally on my own. Sadly it had been somewhat controversial as it was done at the expense of other development tasks (my manager agreed but not his nontechnical manager), and probably too late too. Before I had had enough grasp on the complicated context we had to deliver on a big project and needless to say it was a shitshow and we developers took basically all the blame, with pretty much nothing to say back. The tech stack and code had been in a state of neglect for 8+ years but that was not an acceptable answer at that time, it seemed, and I'd only been there for 6 months when it happened. On the other end of the spectrum, at the above's parent company, I had to maintain several major versions of an API on several cloud regions. Undocumented peculiarities regarding the object storage that was specific to one region caused a production incident that took the dev and support team 3 days to untangle and solve. I have similar stories on monorepo vs. multirepo too

throwaway894345 4y ago

Same. I keep hearing that people have had bad experiences with microservices, but I'm not sure what those bad experiences are. I certainly don't relate to them. We had a monolith, and it devolved into a mess--managers would insist on people taking dependencies on stuff they didn't own in the name of expedience ("yeah, we'll totally circle back and do it the right way lol"). Microservices kept things neat by making dumb things hard. We were also able to deploy individual services much more frequently without needing to coordinate with a bunch of other teams, and builds, tests, etc were much faster helping shorten our iteration loops. Local development environments were a bit more tedious in certain cases, but that was the only issue I recall.

willcipriano 4y ago
Microservices suck when you need to make a process faster, so you run a profiler and figure out that 90% of the workload is handling the http requests between the services.
- bb88 4y ago
  I feel like HTTP is too heavyweight for microservices. For monoliths, it's great. And the only way to proceed in your case is to convert microservices into mini-monoliths. Particularly the authentication side, which needs to be revalidated with every request. gRPC might be better, but supposedly so was SOAP, CORBA, DCOM, XMLRPC, and finally REST.
  - throwaway894345 4y ago
    How does gRPC make authentication lighter-weight? Presumably you still have a token in either case which needs to be validated? There aren't many places where parsing HTTP requests is really a significant bottleneck, and I'd guess the majority of the remainder are an architectural problem (someone using a "get" endpoint in a tight loop instead of a "list" endpoint or some such).
    - bb88 4y ago

atleta 4y ago

It still depends on the organization size (or the size of the user base, or the complexity of the product). So you may only have experience with services where it was the right choice. I did see two instances where it wasn't. (I mainly work with/for small companies, startups.) In one instance I was called in as a tech lead/expert for a small startup having a kind of a product/software crisis. They've been working on their service for a year or too (yes, way too long) and 2-3 months before the planned release at some random conference (Slush, TNW or whatnot) one of the developers figured out that the whole codebase was a piece of shit and there was no way they could be ready in time, but they should rewrite it as a collection of node microservices and that could work. The monolith they had was PHP, just to add to the fun, so switching over would mean switching languages too. The guy, he was a smart and motivated chap, even started implementing one service in his free time, the user registration/user handling I think (the least important and least complex one, of course) which somehow screwed up the monotlith and made it start crashing. (Or so they said, I don't know what was up with that.) Obviously, it was a 100% stupid idea and we went on with fixing their development process (starting to do scrum and teaching the stakeholders that they need to stop phoning the developers directly and asking for features, fixes, introducing automated testing, etc.) Oh, and it was a team of 2-2.5 people. (With the group manager doing some backend work too, but also managing another, totally unrelated project for another client.) The other one was a bit different story, where I just shared my insights over a call. A guy I've known took over a project that was built by a small team (2-5 people, can't remember) for a startup and wanted some external opinion for himself and the founder. That one was built as a set of microservices but they did have all kinds of stability issues. The idea was that it had to be very-very-super scalable. Because, you know you launch and they will come and there's nothing worse than not being able to handle the load. Except, there is: they had been building the thing for over 2 years back then. It was an online medical consultation solution (you describe your problem, pick a doctor, do a f2f call and pay by the time). The funny thing is that I've built a very similar system, as a startup cofounder, 3-4 years earlier for psychology consultation with the help of 2 other guys, who didn't even work full time (one of them came after the first one quit). The MVP was up in, I think, 2 maybe 3 months. Ours was a monolith-ish thing and theirs definitely looked better, maybe scaled better and would have been cheaper to operate at scale (we used an external service for the video calls). But ours was a lot cheaper to build and launch and we could test (validate) our solution a lot earlier with real customers. If it worked out, we could have started breaking it down into multiple services as/if/when needed.

mbrodersen 4y ago

A friend of mine is working on a 20 years old Microservices application. It is literally the worst piece of garbage system I have ever heard about. I am sure my friend would have loved for it to be a monolith instead of 50+ services with massively complicated distributed dependencies and timing issues.

nailer 4y ago

Really? You’ve never seen an over-engineered microservices design that could be a much simpler express app?

fud101 4y ago
What is an express app?
- nailer 4y ago
  An app in express.js, a web framework.

KronisLV 4y ago

When I'm forced to work with a monolith that has 4000+ source files and takes a few minutes to even start up, and has about 20 configuration files that all need to be setup properly for it to even start, I wish for microservices. When I'm forced to work with microservices that need Skaffold and Helm charts just to run locally, but with the configuration in the monorepo being kind of mismanaged and strewn around a bunch of folders with no documentation, in addition to debugging in the IDE not working because nobody set it all up, I wish for monoliths. Really, you can have good monolithic systems and you can have good microservices as well, in addition to something in the middle: https://blog.kronis.dev/articles/modulith-because-we-need-to-scale-but-we-also-cannot-afford-micro-services https://blog.kronis.dev/articles/modulith-because-we-need-to... (actually the first blog post that I wrote, the casual language probably shows its age). But there can also be plenty of poorly developed projects with either. It just so happens that people hate monoliths more in the mainstream culture because most of the older and worse projects out there were monolithic, much like many hate Java and other languages because of being exposed to bad legacy projects written in them: https://earthly.dev/blog/brown-green-language/ https://earthly.dev/blog/brown-green-language/ Just wait for 5-10 years and people's disposition towards both monoliths and microservices will even out, the advantages and disadvantages of either will become clearly recognized and the hype will shift towards something like serverless. Much like now we know the advantages and disadvantages of languages with/without GC as well as higher/lower abtraction levels pretty well (consider Python vs Rust, for example). Maybe things will slow down a bit because Kubernetes will also become a bit easier to use, possibly thanks to projects like K3s, K0s, Portainer and Rancher.

jnash 4y ago
Nothing stops microservices from having the same problems. Plus all the extra problems you get with distributed systems (timing issues, logical dependencies etc.)
barking_biscuit 4y ago
It turns out the secret is just not writing crappy software and having crappy processes. It turns out the caveat is that the conditional probability of those two things is vanishingly small, so regardless of monolith or microservices, most devs are exposed to crappy stuff and because their sample sizes are tiny they assume well we just did it wrong, when in fact they did it average and average is crappy for any and all architectures.