19 ms·

Monoliths are not dinosaurs

266 points by davidrupp 3y ago

jpalomaki 3y ago
"Since its launch in 2006 with just a few microservices, (AWS) S3 has grown to over 300," I would love if articles like this would give a bit more context on the size of codebase and team. Like is it two pizzas per microservice or 10 microservices per one pizza.
- fredrikholm 3y ago
  > Like is it two pizzas per microservice or 10 microservices per one pizza. Nearly spat out my coffee. Thank you for making my morning.
  - przemo_li 3y ago
    I think it's reference to a team that can be satisfied with a pizza in take out event. Increasing number of pizzas you get bigger teams. Increasing number of microservices you assign more of them to a single team.
Zanfa 3y ago
A good rule of thumb is that if you’re starting a new project and immediately implement microservices, you’ve already failed. Never seen it work, don’t think I ever will. The only way microservices can ever work is if they’re extracted over time as the monolith gets fleshed out, likely over the course of years. And even they’re probably a bad idea.
- NohatCoder 3y ago
  I think that splitting things can be a good idea once in a while. The important part is that you only make a new service because some job fits being its own service. The microservice syndrome begin once you start splitting services simply because you arbitrarily declare them too large.
  - KyeRussell 3y ago
    > because some job fits being its own service Determining this at all takes skill. Determining it that far ahead of time requires so much skill and foresight that suggesting that people try is bad advice.
  - ProZsolt 3y ago
    But you can make a "new service" in code in the same binary communicating through memory. You can then separate it to a separate binary, when the network and serialisation overhead worth it. 80% of the time it will never come.
    - selcuka 3y ago
      Microservices are not only "function calls over TCP" though. There are other concerns such as database per service vs a single shared database. There are also security implications that you don't have with a monolity. Designing it that way from the beginning could easily be as complicated as making them separate services.
      - roflyear 3y ago
        I wouldn't call that microservices.
- tinco 3y ago
  I've never done it myself, but it seems startups are succesfully deploying using for example auth0. Outsourcing auth like that is using microservices right?
  - tracker1 3y ago
    Kinda... it's breaking off the domain into a different service (SaaS in this case). I usually separate auth out even for localized authentication, as stronger passphrase hashing is a relatively low bar for DDoS attacks in practice. If hashing a passphrase takes 0.5s of a single core, then it wouldn't take THAT many authentication requests to overload a system... and even then if you use other mitigations, that can have its' own design overhead. So I generally try to design with a separated authentication from the beginning. I'll start with a "dev" auth service that will just have a form you can fill out whatever you want for the "user" and permissions, then sign a token to get into your local/dev environment application. From there it's really easy to create a more robust system or wrappers for external systems (Auth0, Okta, Azure AD, etc). In general, however, MicroServices is about completely separating a given context domain from other services in a larger system. Account Management is separate from Assets, etc. In practice this means you will have a much more complex orchestration, deployment and communications system in place, often with some sort of Distributed RPC, Queue or other layers on top. You also have to be much more concerned with communications between teams and service versioning and availability. This complexity isn't less complex, it just shifts the complexity, which can help with larger organizations, but for smaller teams it can really bottleneck everything. This is why the general concensus is to start with a more monolithic codebase designed to still scale horizontally, then break off systems as the need arizes. The exception being if you are in an organization that has already paid the overhead/debt of setting up for microservice orchestrations.
  - roflyear 3y ago
    That's a different thing really. It's like using sendgrid.
- tracker1 3y ago
  Or... if you're working in an organization that already has a Microservice based infrastructure in place. Otherwise, I generally agree... I'll usually take a monolith approach and break things off in ways that make sense. Usually starting with long running processes that can simply be workers off of queues. Sometimes potential bottlenecks that have higher compute overhead, such as passphrase hashing and comparison which is relatively easy to DDoS, but if broken off only effects new logins and password changes.
  - charrondev 3y ago
    Isn’t something paraphrase hashing something that should be heavily rate limited? In order to DoS your typical site through passphrase hashing you would need to be: - have a ton of valid usernames/emails of accounts that need to be checked (because a typical password check will rate limit by account) - send in a massive torrent of traffic from a ton of IP addresses (because a typical password check will be rate limited by IP, even more than typical IP based rate limiting) While this is not impossible if you had those resources it still might be easier to just DoS the site though standard pages/ endpoints by sheer traffic.
    - roflyear 3y ago
      Correct microservices don't magically prevent DDOS attacks. They can actually make things much worse.
vinay_ys 3y ago
> My rule of thumb has been that with every order of magnitude of growth you should revisit your architecture, and determine whether it can still support the next order level of growth. The last hyper growth startup I worked at grew 10x in scale every 2-3 quarters for nearly 3 years at meaningfully large scale (millions of monthly transacting users). In that time, the number of different business-lines/categories and amount of functional flows and their intersecting/overlapping complexity also grew multiple folds. So, we were adding whole new things and throwing away old things and basically refactoring everything every 18-months. Without knowing consciously, the superpower we had was our ability to refactor large live systems so well. In hindsight it became clear to me that our ability to do this hinged on a few different things: 1. A critical mass of engineers at both senior and junior levels understood the whole systems and flows end to end. A lot of engineers stayed with their own team developing strong functional-domain understanding. Similarly a good number of senior engineers rotated across teams. 2. The devops culture was extreme – every team (of 10-12 engineers) managed all aspects (dev-qa-ops-oncall etc) of building and operating their systems. This meant even very junior engineers were expected to know both functional and non-functional characteristics of their systems. Senior engineers (5-10 yr experience) were expected to evaluate new tech stacks and make choices for their team and live with the consequences. 3. Design proposals were openly shared and sought critical feedback. Technical peer reviews were rigorous. Engineers were genuinely curious to learn things, ask and understand things, challenge/debate things etc. Strong emphasis on first-principles thinking/reasoning and focusing on actual end-to-end problem-solving without being territorial or having dogmas was strongly encouraged and the opposite was strongly discouraged. 4. Doing live-migrations – we mastered the art of doing safe live migrations of services whose API schema or implementation was changing and of datastore whose schema or underlying tech was changing. We had a lot of different database tech migrations – from monolith SQL dbs, to NOSQL clusters to distributed SQL dbs and their equivalent in-memory dbs and caches. Surprisingly, the things we didn't do so well but didn't really hurt our ability to refactor safely were: 1. Documentation – we had informal whiteboard diagrams photographed and stuck in wiki pages. We didn't have reams and reams of documentation. 2. Tests – we didn't have a formal and rigorous test coverage. We had end-to-end tests for load-testing and we had a small manual QA for doing end-to-end integration testing for critical flows. These came about a bit later – but trying to scale them effectively proved very challenging. But these were not seen as hurdles for doing refactors. 3. Formal architecture councils and formal approval processes – we didn't have these. Instead we had strong people to people connect and strong team level accountability – culturally people owned up their mistakes and do everything they could to fix things and do better next time. Humility was high. Later, I worked at a large mature company with very large scale – and everything was exactly flipped on all the points above and any major refactors were a serious pain and migrations took forever and actually never completed. The contrast was very eye-opening and I realized in hindsight the above contributing factors.
Garlef 3y ago
The monolith/microservice dichotomy is a red herring. What even is a microservice? There are other distinctions, borders and splits that are more important to consider. Here's a koan that highlights a few of these considerations: > If you run a kubernetes cluster on a single physical server, is it a monolith or is it a microservice architecture?
sverhagen 3y ago
For many years to come, for better or for worse, people will point to the Prime team's blog post as the definitive proof that microservices are inferior. And instead of perspective of nuance, it'll be used for absolutist arguments. I'm already tired of it in anticipation...
ranting-moth 3y ago
Microservices are, like Prozac, to help with a very specific problem and not without side effects. Don't do Prozac if you don't need it.
andyjohnson0 3y ago
There was a perceptive comment on HN a few days ago [1] to the effect that microservices are a useful way to package a body of code if the team that is consuming the code doesn't trust the team that built it. This brings to mind Conway's law - "Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure" [2] - and also that meme from a few years back about the internal structure of tech companies [3]. So you can argue that monolith/non-monolith it not purely a tech-driven consideration [1] Too lazy to keep looking right now [2] https://en.wikipedia.org/wiki/Conway%27s_law https://en.wikipedia.org/wiki/Conway%27s_law [3] https://www.reddit.com/r/ProgrammerHumor/comments/6jw33z/internal_structure_of_tech_companies/ https://www.reddit.com/r/ProgrammerHumor/comments/6jw33z/int...
Havoc 3y ago
The whole debate is rather silly. Just pick the right architecture for the given problem. Sometimes it's monolith, sometimes it is not. The end.
- mrg3_2013 3y ago
  This really is the issue. When you have no informed opinion, it may be better to start with monolith. This Microservice culture was all about "let me reduce the design complexity by using MS"
  - Havoc 3y ago
    Definitely less footguns too. The thing that makes me lean towards microservice thinking overall is the abundance of free tier building blocks. Obviously doesn't scale but aluring anyway
yuvadam 3y ago
Microservices were a zero-interest rate phenomena that benefited no one but cloud service marketing teams. As money becomes more expensive and as we inch further towards a massive economic crisis, companies that have allowed their R&D budget to bloat out of proportion with needlessly-distributed architectures are NGMI.
lakomen 3y ago
So how does that all help me, who is seeing "microservices AWSGCPAzure EDA" job ads exclusively, not a single regular job ad anymore. It's all hypeshit shit shit.
retrac98 3y ago
Added to https://www.microservice-stories.com https://www.microservice-stories.com - thanks!
- davidrupp 3y ago
  Bookmarked; thanks for that link. Bonus points for sentiment analysis.
- richardbarosky 3y ago
  Nice, love it
- evntdrvn 3y ago
  Neat! It would be awesome if you could set up a parallel “monolith-stories.com” site with the exact same content, except for inverting the sentiment scale :) Seriously!!
akhayam 3y ago
Werner's advice in my own words: 1. Don't pick an architecture because it's all the rage right now. 2. Don't pick an architecture that mimics your org's structure. Aka don't fall prey to Conway's law: https://en.wikipedia.org/wiki/Conway%27s_law https://en.wikipedia.org/wiki/Conway%27s_law. 3. Don't pick an architecture that your team can't operationalize--e.g. due to lack of skills or due to business constraints.
- davidrupp 3y ago
  (Some of) Werner's advice in his own words (from the linked article): * "I always urge builders to consider the evolution of their systems over time and make sure the foundation is such that you can change and expand them with the minimum number of dependencies." * "There are few one-way doors. Evaluating your systems regularly is as important, if not more so, than building them in the first place."
  - asdfman123 3y ago
    > Evaluating your systems regularly is as important, if not more so, than building them in the first place. This is useful advice to people who can make high level org decisions. But they don’t necessarily know what’s going on at a software level. The people who do know what’s going on often have a very hard time getting buy in for refactoring. Many of them (rightfully) conclude it’s better to just make management happy churning out features.
- marcosdumay 3y ago
  > Aka don't fall prey to Conway's law You can't not fall prey to Conway's law. You can only choose how you organize your people and their interfaces.
  - thanatos519 3y ago
    Right! Decide on the shape of the software, then rearrange your organization to match.
rr808 3y ago
Can't we all just go back? Seriously every system I've worked on in the last 10 years seems worse in every metric than what I worked on 2000-2010.
- pjmlp 3y ago
  Kubernetes makes Websphere 5 development feel refreshing.
- keyle 3y ago
  There is no going back. The industry has seen a 5X raise in money and people. It's only natural that a job that used to take 2 now takes 10 people because of those facts. And I agree. We've traded safety and a few other things that could have been ironed out in exchange of massive gridlocks, headaches and colossal facepalms. Performance is only on par if you consider the hardware improvements, it's crazy.
  - papito 3y ago
    We will be going back. After the last binge, torching through piles of cash to do resume-driven development, companies are now looking to be lean. You can't really survive if you are doing microservices and trying to actually build something with a team cut by 75%. I am already seeing this newly discovered love of simple, fast monoliths and the developer ergonomics they offer. What is old is new again - we are at the beginning of this cycle. The older generation learned that "complexity kills", the new generation is beginning to get it as well.
  - Aeolun 3y ago
    If you can have a monolith for $500/month with a team of 2, or a distributed system for $50,000/month with a team of 50, what manager in their right mind would choose the former. Zero prestige in that.
    - anonzzzies 3y ago
      The first thing I hear from the tech lead/CTO or whatever when I walk into a company is 'we are going to use microservices', then a lot of kubernetes, sqs, lambda etc. And THEN they talk about what the project is. I only didn't manage to keep away from a monolith for a fraction of the money and time; it was a huge fail; 20m$ burnt on engineers and aws and the endresult was a painful burning pile of nuclear waste. They had money to burn so they are still alive, but I should've walked out after the CTO pushed microservices past his team (his team + my team) who said it was nonsense in this case (it's in all cases, but whatever, I cannot prove that so).
tonetheman 3y ago
[dead]
bovermyer 3y ago
You know, I'm pretty sure you could build a PHP monolith in 2023 without a framework and it would do what it needed to do.
- tored 3y ago
  Yes, I recently built a small PHP example project for educational purposes and I intentionally avoided all external dependencies for demonstrating how you would solve things without a framework or library. You can do almost everything you need with plain PHP without too much effort except a few exceptions that you shouldn't waste your time on, most of them are tooling however and not for the application itself, like unit testing, error tracking and logging, database migration, but for that you of course use libraries.
- asdfman123 3y ago
  Bad coding is infinitely worse than a “bad” language. I don’t know how many of you are musicians, but many people are surprised to find that the tone quality of a guitar has way more to do with the person playing it than the way it was built.
  - dmbche 3y ago
    Surprising! Will look into
  - emodendroket 3y ago
    Bad languages, more than anything else, frustrate your expectations and encourage developers to do things that are difficult to maintain. Sure, a great basketball player can dribble with gloves on his hands, but he’s making the task harder on himself, and anyone who’s less than great is less likely to be able to hold on to the ball at all in that case.
  - gabereiser 3y ago
    A great player can make a Squire sound like a Studio Strat. Indeed. The type of strings make a difference too.
kristianpaul 3y ago
Cloud computing is not a religion could be another good title
quadrifoliate 3y ago
Like most articles in distributed systems, this makes wild assumptions about the most important, i.e. the human layer. I would bet $100 that this is written by the same sort of person who thinks "Managers – what do they do all day exactly?" [1] > If you hire the best engineers.... Guess what, there is no broad consensus on what "best engineer" means. I bet your org is rejecting really good engineers right now because they don't know what Kubernetes is. Same goes for literally any other technology that has been part of a hype cycle (Java in 2001, Ruby on Rails in 2011, ML in 2011; no the precise years don't matter). > ...trust them to make the best decisions. A lot of work encapsulated there in less than ten words. If you hire a bunch of people and tell them "you are the best", you think they are going to sit around and run the Raft protocol for consensus on deciding how to architect the system? No, each of them is going to reinvent Kubernetes, and likely not in an amazing way. Microservices are often best deployed when there is a mixture of cultural and engineering factors that makes it easy to split up a system into different parts that have interfaces with each other. It has little to do with the superiority of the technical architecture. ---------------------------------------- [1] Looks like the article was written by the CTO of Amazon, which...surprises me a bit. Then again, from all accounts, Amazon's not exactly known as the best place to work; so maybe I'm right? In any case, anything written by Amazon is not directly applicable to the vast majority of small-to-medium companies.
- deleted 3y ago
  [deleted]
- jabradoodle 3y ago
  Their point on hiring the best engineers was specifically about not following the hype, and allowing engineers to engineer. Also, they never claimed technical superiority of microservices, quote: > For example, a startup with five engineers may choose a monolithic architecture because it is easier to deploy and doesn’t require their small team to learn multiple programming languages.
  - quadrifoliate 3y ago
    > Also, they never claimed technical superiority of microservices, quote: No, but they are arguing against the strawman of the perceived technical inferiority of monoliths. Look at the title of the article. I am simply calling out that strawman as such. If they wanted to convey the message of "It all depends, use the best architecture for the job", they should reflect that in the title.
paulddraper 3y ago
Software would be 50% better if every developer understood Tesler's Law: "Complexity can neither be created nor destroyed, only moved somewhere else." The drive to simplify is virtuous, but often people are blind to the complexity that it adds. Okay, so your microservices are each very simple, but that made the interactions and resource provisioning very complex. What was the net gain? The correct solution depends on the circumstances. There are excellent uses of microservices. There are excellent uses of monoliths. There are excellent uses of monorepos. There are excellent uses of ... (wait never mind monorepos are just better). Understand what is ESSENTIAL complexity and what is INCIDENTAL complexity. Simplify your system to remove as much incidental complexity as possible, until you are left with the essential and be satisfied with that.
- deterministic 3y ago
  Not true. You can make any system arbitrarily complex. And 95% of software developers IMHO are hell-bent on proving that true every single day. Micro-services is a GREAT example of this.
  - zambal 3y ago
    > You can make any system arbitrarily complex. But isn't that introducing incidental complexity? Not sure if you actually disagree.
- ed_elliott_asc 3y ago
  Does Tesler’s law apply to lines of code or architecture? I have absolutely seen complex code that was created (often for perceived “best practices” like DRY) which could be removed by simplifying the code.
  - sebhans 3y ago
    I think it applies to problems, not solutions. The complexity of a given problem cannot change. If you try to ignore part of the inherent complexity of a problem (also called essential complexity) in your solution, it does not disappear but someone else must solve it somewhere else, or the problem is not really solved. If you build a solution that is more complex than the problem itself (in other words, if you add incidental complexity), this does not increase the complexity of the problem either, only the complexity of the solution. I think a good solution to any problem needs to match it in complexity. I regularly use this comparison as a benchmark for solutions. Of course, you can also see it this way: The complexity you remove from the code by making it cleaner is added to your team communication because you now have to defend your decision. (Only half joking.)
andrewstuart 3y ago
I imagine there are not many video companies running on AWS. It's really only Amazon themselves that can afford to do so.
- andrewguenther 3y ago
  Netflix is one of AWS largest customers.
  - andrewstuart 3y ago
    True, but I'd say there's zero relationship between the AWS price list and what Netflix pays. So Netflix doesn't really count as an example of how AWS is a practical option for video hosting.
    - yibg 3y ago
      Because Netflix doesn’t do video hosting on AWS. “Running” on AWS isn’t necessarily hosting on AWS.
osigurdson 3y ago
I think the first step toward sanity is to stop factoring services by team sizes - “we have 100 people so require 20 microservices”. Instead, factor services along natural fault lines. These are areas in the solution that scale differently from other parts and can tolerate communicating over http or message queue. It is fine to have lots of people work on a single service. We compose things using 3rd party libraries all the time. Just treat internal code a bit more like 3rd party libraries.
- kqr 3y ago
  Could you elaborate on what you mean by "natural fault lines"? The rest of that paragraph seems to refer to performance -- is that the criterion? Do the natural fault lines shift if you manage to optimise the performance of a component, so that it starts scaling at the same rates as its neighbours?
  - osigurdson 3y ago
    It is hard to say without understanding the system. In my own case it is auth, output (1D,2D,3D), logging and sim but that is meaningless outside of my application's context.
- dasil003 3y ago
  You're right that factoring into services should be based on natural domain boundaries. That said, it feels like a bit of a strawman to suggest that people are driving their architecture with naive math like this. I've definitely seen journeyman engineers coming out of FAANG and other big companies proposing overly complex service topographies, but there is always at least a veneer of semantic justification. That's not to say there's no relationship between team size and the applicability of a service-oriented architecture. Microservices are a way of drawing hard boundaries around blocks of logic. These boundaries come with cognitive and operational costs, so they represent significant overhead, but they are a tool for abstracting both logic and physical operations to the maximum degree possible (100% is never possible for a single application). In order to get a net benefit, you have to have enough engineers that they can be experts in a subset of services, and the interfaces to their peer services have to be reliable and stable enough that they can be productive without knowing anything about their internals. So while I agree with you there is no universal floor of microservices that makes based purely on team size, there definitely is a ceiling.
mullingitover 3y ago
I can’t believe it’s news that someone said this. I thought everyone understood: you don’t try to do microservices until you have to. Before you get to that point you make your monolith modular enough that if you ever need microservices you’re prepared to break them out.
- 8organicbits 3y ago
  A lot of people are focused on microservices as a way to address scaling (of load, team size, etc.) but there's other reasons to choose a microservice. A pretty basic one is when an existing microservice already does what you need and no suitable module or library exists. In that case, there's no "until". Start with a microservice. There's a number of open source projects that are pluggable microservices, for example.
- mattbillenstein 3y ago
  This is not well understood at all - and a lot of frameworks don't really lend to simply splitting a piece off. I've seen terrible spaghetti code apps where literally nothing can be refactored because of the model / view, God object dependency stuff all over the place.
- 0x6c6f6c 3y ago
  This is really it. You can have everything in a monolith and scalable and modular by following good architectural practices. If the need arises, you could move into SOA, and break out a couple of your larger domain modules. Continue following these same rules though. If the need arises still, you could move into micro services, and break out more / all of your domain modules. Truly understand first whether you actually need this first. But the "let's do micro services this monolith is old junk" trope, abandonwaring the codebase, building out a bunch of services without strong, fundamental domain knowledge, and then complaining when shit is expensive and fragile and broken- it's getting tiring.
vishnugupta 3y ago
For the context this is most likely in response to DHH’s post [1] where he heavily came down on AWS’s serverless offerings. He’s been ranting against cloud too. [1] https://world.hey.com/dhh/even-amazon-can-t-make-sense-of-serverless-or-microservices-59625580 https://world.hey.com/dhh/even-amazon-can-t-make-sense-of-se...
ChicagoDave 3y ago
Lot of false comparisons here. It’s not “monolith vs distributed”. It’s “good monolith vs bad monolith or big ball of mud vs domain-driven design” It depends on what your primary domain is, level of complexity, number and makeup of enterprise integrations, and more. Some monoliths are very bad. Some distributed systems are very bad. My rundown is: - is it a simple crud system? —-- monolith Otherwise: - model it, identify bounded contexts, proceed accordingly
jeswin 3y ago
HN could be a little less pessimistic. People aren't choosing microservices merely because of the hype. Here's why I'd choose microservices for a large project: 1. People don't produce uniform code quality. With microservices, the damage is often contained. 2. Every monolith is riddled with exceptional cases in a few years. Only a few people know about corner cases after a few years, and the company becomes dependent on those developers. 3. It's easier for junior developers to start contributing. With a monolith you'd need to be rigid with code reviews, whereas you could be a little lax with microservices. Again, ties into (1) above. This also allows a company to hire faster. 4. Different modules have different performance and non-functional requirements. For example, consider reading a large file. You don't want such an expensive process to compete for resources with say a product search flow. Even with a monolith, you wouldn't do this - you'd make an exception. In a few years, the monolith is full of special cases which only a few people know about. When those employees leave, the project sometimes stalls and code quality drops. Related to (2). 5. Microservices have become a lot easier thanks to k8s and docker. If you think about it, microservices were becoming popular even before k8s became mainstream. If it was viable then, it's a lot easier today. 6. It helps with organizing teams and assigning responsibility. 7. You don't need super small microservices. A microservice could very well handle all of a module - say all of payments (payment processing, refunds, coupon codes etc), or all of authentication (oauth, mfa etc). 8. Broken Windows Theory more often applies to monoliths, and much less to microservices. Delivery pressure is unavoidable in product development at various points. Which means that you'll often make compromises. Once you start making these compromises, people will keep making them more often. 9. It allows you the agility to choose a more efficient tech/process when available. Monoliths are rigid in tech choices, and don't easily allow you to adopt a different programming language or a framework. With Microservices, you could choose the stack that best solves the problem at hand. In addition, this allows a company to scale up the team faster. Add: 10. It's difficult to fix schemas, contracts and data structures once they're in production. Refactoring is easier with microservices, given that the implications are local compared to monoliths.
- dlisboa 3y ago
  Yes! Finally! People keep making the assumption that going for Services/Microservices is merely technical. It’s almost all about people and organization. Point 7 is the most important of the technical considerations: just make your services big enough to make sense as a separate unit and small enough to not be another unchangeable monolith. Yes, it’s not sane to have 300 Lambdas that add one number to another talking to each other over network, so just don’t do it. Microservices the “Netflix way”, with the huge graph of services, gives a bad name to the idea of factoring out modules into independently deployable parts, which always made sense. Kubernetes just helps with deployment, but how coarse or fine you factor is on you.

266 points by davidrupp 3y ago

"Since its launch in 2006 with just a few microservices, (AWS) S3 has grown to over 300," I would love if articles like this would give a bit more context on the size of codebase and team. Like is it two pizzas per microservice or 10 microservices per one pizza.

fredrikholm 3y ago
> Like is it two pizzas per microservice or 10 microservices per one pizza. Nearly spat out my coffee. Thank you for making my morning.
- przemo_li 3y ago
  I think it's reference to a team that can be satisfied with a pizza in take out event. Increasing number of pizzas you get bigger teams. Increasing number of microservices you assign more of them to a single team.

Zanfa 3y ago

A good rule of thumb is that if you’re starting a new project and immediately implement microservices, you’ve already failed. Never seen it work, don’t think I ever will. The only way microservices can ever work is if they’re extracted over time as the monolith gets fleshed out, likely over the course of years. And even they’re probably a bad idea.

NohatCoder 3y ago
I think that splitting things can be a good idea once in a while. The important part is that you only make a new service because some job fits being its own service. The microservice syndrome begin once you start splitting services simply because you arbitrarily declare them too large.
- KyeRussell 3y ago
  > because some job fits being its own service Determining this at all takes skill. Determining it that far ahead of time requires so much skill and foresight that suggesting that people try is bad advice.
- ProZsolt 3y ago
  But you can make a "new service" in code in the same binary communicating through memory. You can then separate it to a separate binary, when the network and serialisation overhead worth it. 80% of the time it will never come.
  - selcuka 3y ago
    Microservices are not only "function calls over TCP" though. There are other concerns such as database per service vs a single shared database. There are also security implications that you don't have with a monolity. Designing it that way from the beginning could easily be as complicated as making them separate services.
    - roflyear 3y ago
      I wouldn't call that microservices.
tinco 3y ago
I've never done it myself, but it seems startups are succesfully deploying using for example auth0. Outsourcing auth like that is using microservices right?
- tracker1 3y ago
  Kinda... it's breaking off the domain into a different service (SaaS in this case). I usually separate auth out even for localized authentication, as stronger passphrase hashing is a relatively low bar for DDoS attacks in practice. If hashing a passphrase takes 0.5s of a single core, then it wouldn't take THAT many authentication requests to overload a system... and even then if you use other mitigations, that can have its' own design overhead. So I generally try to design with a separated authentication from the beginning. I'll start with a "dev" auth service that will just have a form you can fill out whatever you want for the "user" and permissions, then sign a token to get into your local/dev environment application. From there it's really easy to create a more robust system or wrappers for external systems (Auth0, Okta, Azure AD, etc). In general, however, MicroServices is about completely separating a given context domain from other services in a larger system. Account Management is separate from Assets, etc. In practice this means you will have a much more complex orchestration, deployment and communications system in place, often with some sort of Distributed RPC, Queue or other layers on top. You also have to be much more concerned with communications between teams and service versioning and availability. This complexity isn't less complex, it just shifts the complexity, which can help with larger organizations, but for smaller teams it can really bottleneck everything. This is why the general concensus is to start with a more monolithic codebase designed to still scale horizontally, then break off systems as the need arizes. The exception being if you are in an organization that has already paid the overhead/debt of setting up for microservice orchestrations.
- roflyear 3y ago
  That's a different thing really. It's like using sendgrid.
tracker1 3y ago
Or... if you're working in an organization that already has a Microservice based infrastructure in place. Otherwise, I generally agree... I'll usually take a monolith approach and break things off in ways that make sense. Usually starting with long running processes that can simply be workers off of queues. Sometimes potential bottlenecks that have higher compute overhead, such as passphrase hashing and comparison which is relatively easy to DDoS, but if broken off only effects new logins and password changes.
- charrondev 3y ago
  Isn’t something paraphrase hashing something that should be heavily rate limited? In order to DoS your typical site through passphrase hashing you would need to be: - have a ton of valid usernames/emails of accounts that need to be checked (because a typical password check will rate limit by account) - send in a massive torrent of traffic from a ton of IP addresses (because a typical password check will be rate limited by IP, even more than typical IP based rate limiting) While this is not impossible if you had those resources it still might be easier to just DoS the site though standard pages/ endpoints by sheer traffic.
  - roflyear 3y ago
    Correct microservices don't magically prevent DDOS attacks. They can actually make things much worse.

vinay_ys 3y ago

> My rule of thumb has been that with every order of magnitude of growth you should revisit your architecture, and determine whether it can still support the next order level of growth. The last hyper growth startup I worked at grew 10x in scale every 2-3 quarters for nearly 3 years at meaningfully large scale (millions of monthly transacting users). In that time, the number of different business-lines/categories and amount of functional flows and their intersecting/overlapping complexity also grew multiple folds. So, we were adding whole new things and throwing away old things and basically refactoring everything every 18-months. Without knowing consciously, the superpower we had was our ability to refactor large live systems so well. In hindsight it became clear to me that our ability to do this hinged on a few different things: 1. A critical mass of engineers at both senior and junior levels understood the whole systems and flows end to end. A lot of engineers stayed with their own team developing strong functional-domain understanding. Similarly a good number of senior engineers rotated across teams. 2. The devops culture was extreme – every team (of 10-12 engineers) managed all aspects (dev-qa-ops-oncall etc) of building and operating their systems. This meant even very junior engineers were expected to know both functional and non-functional characteristics of their systems. Senior engineers (5-10 yr experience) were expected to evaluate new tech stacks and make choices for their team and live with the consequences. 3. Design proposals were openly shared and sought critical feedback. Technical peer reviews were rigorous. Engineers were genuinely curious to learn things, ask and understand things, challenge/debate things etc. Strong emphasis on first-principles thinking/reasoning and focusing on actual end-to-end problem-solving without being territorial or having dogmas was strongly encouraged and the opposite was strongly discouraged. 4. Doing live-migrations – we mastered the art of doing safe live migrations of services whose API schema or implementation was changing and of datastore whose schema or underlying tech was changing. We had a lot of different database tech migrations – from monolith SQL dbs, to NOSQL clusters to distributed SQL dbs and their equivalent in-memory dbs and caches. Surprisingly, the things we didn't do so well but didn't really hurt our ability to refactor safely were: 1. Documentation – we had informal whiteboard diagrams photographed and stuck in wiki pages. We didn't have reams and reams of documentation. 2. Tests – we didn't have a formal and rigorous test coverage. We had end-to-end tests for load-testing and we had a small manual QA for doing end-to-end integration testing for critical flows. These came about a bit later – but trying to scale them effectively proved very challenging. But these were not seen as hurdles for doing refactors. 3. Formal architecture councils and formal approval processes – we didn't have these. Instead we had strong people to people connect and strong team level accountability – culturally people owned up their mistakes and do everything they could to fix things and do better next time. Humility was high. Later, I worked at a large mature company with very large scale – and everything was exactly flipped on all the points above and any major refactors were a serious pain and migrations took forever and actually never completed. The contrast was very eye-opening and I realized in hindsight the above contributing factors.

Garlef 3y ago

The monolith/microservice dichotomy is a red herring. What even is a microservice? There are other distinctions, borders and splits that are more important to consider. Here's a koan that highlights a few of these considerations: > If you run a kubernetes cluster on a single physical server, is it a monolith or is it a microservice architecture?

sverhagen 3y ago

For many years to come, for better or for worse, people will point to the Prime team's blog post as the definitive proof that microservices are inferior. And instead of perspective of nuance, it'll be used for absolutist arguments. I'm already tired of it in anticipation...

ranting-moth 3y ago

Microservices are, like Prozac, to help with a very specific problem and not without side effects. Don't do Prozac if you don't need it.

andyjohnson0 3y ago

There was a perceptive comment on HN a few days ago [1] to the effect that microservices are a useful way to package a body of code if the team that is consuming the code doesn't trust the team that built it. This brings to mind Conway's law - "Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure" [2] - and also that meme from a few years back about the internal structure of tech companies [3]. So you can argue that monolith/non-monolith it not purely a tech-driven consideration [1] Too lazy to keep looking right now [2] https://en.wikipedia.org/wiki/Conway%27s_law https://en.wikipedia.org/wiki/Conway%27s_law [3] https://www.reddit.com/r/ProgrammerHumor/comments/6jw33z/internal_structure_of_tech_companies/ https://www.reddit.com/r/ProgrammerHumor/comments/6jw33z/int...

Havoc 3y ago

The whole debate is rather silly. Just pick the right architecture for the given problem. Sometimes it's monolith, sometimes it is not. The end.

mrg3_2013 3y ago
This really is the issue. When you have no informed opinion, it may be better to start with monolith. This Microservice culture was all about "let me reduce the design complexity by using MS"
- Havoc 3y ago
  Definitely less footguns too. The thing that makes me lean towards microservice thinking overall is the abundance of free tier building blocks. Obviously doesn't scale but aluring anyway

yuvadam 3y ago

Microservices were a zero-interest rate phenomena that benefited no one but cloud service marketing teams. As money becomes more expensive and as we inch further towards a massive economic crisis, companies that have allowed their R&D budget to bloat out of proportion with needlessly-distributed architectures are NGMI.

lakomen 3y ago

So how does that all help me, who is seeing "microservices AWSGCPAzure EDA" job ads exclusively, not a single regular job ad anymore. It's all hypeshit shit shit.

retrac98 3y ago

Added to https://www.microservice-stories.com https://www.microservice-stories.com - thanks!

davidrupp 3y ago
Bookmarked; thanks for that link. Bonus points for sentiment analysis.
richardbarosky 3y ago
Nice, love it
evntdrvn 3y ago
Neat! It would be awesome if you could set up a parallel “monolith-stories.com” site with the exact same content, except for inverting the sentiment scale :) Seriously!!

akhayam 3y ago

Werner's advice in my own words: 1. Don't pick an architecture because it's all the rage right now. 2. Don't pick an architecture that mimics your org's structure. Aka don't fall prey to Conway's law: https://en.wikipedia.org/wiki/Conway%27s_law https://en.wikipedia.org/wiki/Conway%27s_law. 3. Don't pick an architecture that your team can't operationalize--e.g. due to lack of skills or due to business constraints.

davidrupp 3y ago
(Some of) Werner's advice in his own words (from the linked article): * "I always urge builders to consider the evolution of their systems over time and make sure the foundation is such that you can change and expand them with the minimum number of dependencies." * "There are few one-way doors. Evaluating your systems regularly is as important, if not more so, than building them in the first place."
- asdfman123 3y ago
  > Evaluating your systems regularly is as important, if not more so, than building them in the first place. This is useful advice to people who can make high level org decisions. But they don’t necessarily know what’s going on at a software level. The people who do know what’s going on often have a very hard time getting buy in for refactoring. Many of them (rightfully) conclude it’s better to just make management happy churning out features.
marcosdumay 3y ago
> Aka don't fall prey to Conway's law You can't not fall prey to Conway's law. You can only choose how you organize your people and their interfaces.
- thanatos519 3y ago
  Right! Decide on the shape of the software, then rearrange your organization to match.

rr808 3y ago

Can't we all just go back? Seriously every system I've worked on in the last 10 years seems worse in every metric than what I worked on 2000-2010.

pjmlp 3y ago
Kubernetes makes Websphere 5 development feel refreshing.
keyle 3y ago
There is no going back. The industry has seen a 5X raise in money and people. It's only natural that a job that used to take 2 now takes 10 people because of those facts. And I agree. We've traded safety and a few other things that could have been ironed out in exchange of massive gridlocks, headaches and colossal facepalms. Performance is only on par if you consider the hardware improvements, it's crazy.
- papito 3y ago
  We will be going back. After the last binge, torching through piles of cash to do resume-driven development, companies are now looking to be lean. You can't really survive if you are doing microservices and trying to actually build something with a team cut by 75%. I am already seeing this newly discovered love of simple, fast monoliths and the developer ergonomics they offer. What is old is new again - we are at the beginning of this cycle. The older generation learned that "complexity kills", the new generation is beginning to get it as well.
- Aeolun 3y ago
  If you can have a monolith for $500/month with a team of 2, or a distributed system for $50,000/month with a team of 50, what manager in their right mind would choose the former. Zero prestige in that.
  - anonzzzies 3y ago
    The first thing I hear from the tech lead/CTO or whatever when I walk into a company is 'we are going to use microservices', then a lot of kubernetes, sqs, lambda etc. And THEN they talk about what the project is. I only didn't manage to keep away from a monolith for a fraction of the money and time; it was a huge fail; 20m$ burnt on engineers and aws and the endresult was a painful burning pile of nuclear waste. They had money to burn so they are still alive, but I should've walked out after the CTO pushed microservices past his team (his team + my team) who said it was nonsense in this case (it's in all cases, but whatever, I cannot prove that so).

tonetheman 3y ago

[dead]

bovermyer 3y ago

You know, I'm pretty sure you could build a PHP monolith in 2023 without a framework and it would do what it needed to do.

tored 3y ago
Yes, I recently built a small PHP example project for educational purposes and I intentionally avoided all external dependencies for demonstrating how you would solve things without a framework or library. You can do almost everything you need with plain PHP without too much effort except a few exceptions that you shouldn't waste your time on, most of them are tooling however and not for the application itself, like unit testing, error tracking and logging, database migration, but for that you of course use libraries.
asdfman123 3y ago
Bad coding is infinitely worse than a “bad” language. I don’t know how many of you are musicians, but many people are surprised to find that the tone quality of a guitar has way more to do with the person playing it than the way it was built.
- dmbche 3y ago
  Surprising! Will look into
- emodendroket 3y ago
  Bad languages, more than anything else, frustrate your expectations and encourage developers to do things that are difficult to maintain. Sure, a great basketball player can dribble with gloves on his hands, but he’s making the task harder on himself, and anyone who’s less than great is less likely to be able to hold on to the ball at all in that case.
- gabereiser 3y ago
  A great player can make a Squire sound like a Studio Strat. Indeed. The type of strings make a difference too.

kristianpaul 3y ago

Cloud computing is not a religion could be another good title

quadrifoliate 3y ago

Like most articles in distributed systems, this makes wild assumptions about the most important, i.e. the human layer. I would bet $100 that this is written by the same sort of person who thinks "Managers – what do they do all day exactly?" [1] > If you hire the best engineers.... Guess what, there is no broad consensus on what "best engineer" means. I bet your org is rejecting really good engineers right now because they don't know what Kubernetes is. Same goes for literally any other technology that has been part of a hype cycle (Java in 2001, Ruby on Rails in 2011, ML in 2011; no the precise years don't matter). > ...trust them to make the best decisions. A lot of work encapsulated there in less than ten words. If you hire a bunch of people and tell them "you are the best", you think they are going to sit around and run the Raft protocol for consensus on deciding how to architect the system? No, each of them is going to reinvent Kubernetes, and likely not in an amazing way. Microservices are often best deployed when there is a mixture of cultural and engineering factors that makes it easy to split up a system into different parts that have interfaces with each other. It has little to do with the superiority of the technical architecture. ---------------------------------------- [1] Looks like the article was written by the CTO of Amazon, which...surprises me a bit. Then again, from all accounts, Amazon's not exactly known as the best place to work; so maybe I'm right? In any case, anything written by Amazon is not directly applicable to the vast majority of small-to-medium companies.

deleted 3y ago
[deleted]
jabradoodle 3y ago
Their point on hiring the best engineers was specifically about not following the hype, and allowing engineers to engineer. Also, they never claimed technical superiority of microservices, quote: > For example, a startup with five engineers may choose a monolithic architecture because it is easier to deploy and doesn’t require their small team to learn multiple programming languages.
- quadrifoliate 3y ago
  > Also, they never claimed technical superiority of microservices, quote: No, but they are arguing against the strawman of the perceived technical inferiority of monoliths. Look at the title of the article. I am simply calling out that strawman as such. If they wanted to convey the message of "It all depends, use the best architecture for the job", they should reflect that in the title.

paulddraper 3y ago

Software would be 50% better if every developer understood Tesler's Law: "Complexity can neither be created nor destroyed, only moved somewhere else." The drive to simplify is virtuous, but often people are blind to the complexity that it adds. Okay, so your microservices are each very simple, but that made the interactions and resource provisioning very complex. What was the net gain? The correct solution depends on the circumstances. There are excellent uses of microservices. There are excellent uses of monoliths. There are excellent uses of monorepos. There are excellent uses of ... (wait never mind monorepos are just better). Understand what is ESSENTIAL complexity and what is INCIDENTAL complexity. Simplify your system to remove as much incidental complexity as possible, until you are left with the essential and be satisfied with that.

deterministic 3y ago
Not true. You can make any system arbitrarily complex. And 95% of software developers IMHO are hell-bent on proving that true every single day. Micro-services is a GREAT example of this.
- zambal 3y ago
  > You can make any system arbitrarily complex. But isn't that introducing incidental complexity? Not sure if you actually disagree.
ed_elliott_asc 3y ago
Does Tesler’s law apply to lines of code or architecture? I have absolutely seen complex code that was created (often for perceived “best practices” like DRY) which could be removed by simplifying the code.
- sebhans 3y ago
  I think it applies to problems, not solutions. The complexity of a given problem cannot change. If you try to ignore part of the inherent complexity of a problem (also called essential complexity) in your solution, it does not disappear but someone else must solve it somewhere else, or the problem is not really solved. If you build a solution that is more complex than the problem itself (in other words, if you add incidental complexity), this does not increase the complexity of the problem either, only the complexity of the solution. I think a good solution to any problem needs to match it in complexity. I regularly use this comparison as a benchmark for solutions. Of course, you can also see it this way: The complexity you remove from the code by making it cleaner is added to your team communication because you now have to defend your decision. (Only half joking.)

andrewstuart 3y ago

I imagine there are not many video companies running on AWS. It's really only Amazon themselves that can afford to do so.

andrewguenther 3y ago
Netflix is one of AWS largest customers.
- andrewstuart 3y ago
  True, but I'd say there's zero relationship between the AWS price list and what Netflix pays. So Netflix doesn't really count as an example of how AWS is a practical option for video hosting.
  - yibg 3y ago
    Because Netflix doesn’t do video hosting on AWS. “Running” on AWS isn’t necessarily hosting on AWS.

osigurdson 3y ago

I think the first step toward sanity is to stop factoring services by team sizes - “we have 100 people so require 20 microservices”. Instead, factor services along natural fault lines. These are areas in the solution that scale differently from other parts and can tolerate communicating over http or message queue. It is fine to have lots of people work on a single service. We compose things using 3rd party libraries all the time. Just treat internal code a bit more like 3rd party libraries.

kqr 3y ago
Could you elaborate on what you mean by "natural fault lines"? The rest of that paragraph seems to refer to performance -- is that the criterion? Do the natural fault lines shift if you manage to optimise the performance of a component, so that it starts scaling at the same rates as its neighbours?
- osigurdson 3y ago
  It is hard to say without understanding the system. In my own case it is auth, output (1D,2D,3D), logging and sim but that is meaningless outside of my application's context.
dasil003 3y ago
You're right that factoring into services should be based on natural domain boundaries. That said, it feels like a bit of a strawman to suggest that people are driving their architecture with naive math like this. I've definitely seen journeyman engineers coming out of FAANG and other big companies proposing overly complex service topographies, but there is always at least a veneer of semantic justification. That's not to say there's no relationship between team size and the applicability of a service-oriented architecture. Microservices are a way of drawing hard boundaries around blocks of logic. These boundaries come with cognitive and operational costs, so they represent significant overhead, but they are a tool for abstracting both logic and physical operations to the maximum degree possible (100% is never possible for a single application). In order to get a net benefit, you have to have enough engineers that they can be experts in a subset of services, and the interfaces to their peer services have to be reliable and stable enough that they can be productive without knowing anything about their internals. So while I agree with you there is no universal floor of microservices that makes based purely on team size, there definitely is a ceiling.

mullingitover 3y ago

I can’t believe it’s news that someone said this. I thought everyone understood: you don’t try to do microservices until you have to. Before you get to that point you make your monolith modular enough that if you ever need microservices you’re prepared to break them out.

8organicbits 3y ago
A lot of people are focused on microservices as a way to address scaling (of load, team size, etc.) but there's other reasons to choose a microservice. A pretty basic one is when an existing microservice already does what you need and no suitable module or library exists. In that case, there's no "until". Start with a microservice. There's a number of open source projects that are pluggable microservices, for example.
mattbillenstein 3y ago
This is not well understood at all - and a lot of frameworks don't really lend to simply splitting a piece off. I've seen terrible spaghetti code apps where literally nothing can be refactored because of the model / view, God object dependency stuff all over the place.
0x6c6f6c 3y ago
This is really it. You can have everything in a monolith and scalable and modular by following good architectural practices. If the need arises, you could move into SOA, and break out a couple of your larger domain modules. Continue following these same rules though. If the need arises still, you could move into micro services, and break out more / all of your domain modules. Truly understand first whether you actually need this first. But the "let's do micro services this monolith is old junk" trope, abandonwaring the codebase, building out a bunch of services without strong, fundamental domain knowledge, and then complaining when shit is expensive and fragile and broken- it's getting tiring.

vishnugupta 3y ago

For the context this is most likely in response to DHH’s post [1] where he heavily came down on AWS’s serverless offerings. He’s been ranting against cloud too. [1] https://world.hey.com/dhh/even-amazon-can-t-make-sense-of-serverless-or-microservices-59625580 https://world.hey.com/dhh/even-amazon-can-t-make-sense-of-se...

ChicagoDave 3y ago

Lot of false comparisons here. It’s not “monolith vs distributed”. It’s “good monolith vs bad monolith or big ball of mud vs domain-driven design” It depends on what your primary domain is, level of complexity, number and makeup of enterprise integrations, and more. Some monoliths are very bad. Some distributed systems are very bad. My rundown is: - is it a simple crud system? —-- monolith Otherwise: - model it, identify bounded contexts, proceed accordingly

jeswin 3y ago

HN could be a little less pessimistic. People aren't choosing microservices merely because of the hype. Here's why I'd choose microservices for a large project: 1. People don't produce uniform code quality. With microservices, the damage is often contained. 2. Every monolith is riddled with exceptional cases in a few years. Only a few people know about corner cases after a few years, and the company becomes dependent on those developers. 3. It's easier for junior developers to start contributing. With a monolith you'd need to be rigid with code reviews, whereas you could be a little lax with microservices. Again, ties into (1) above. This also allows a company to hire faster. 4. Different modules have different performance and non-functional requirements. For example, consider reading a large file. You don't want such an expensive process to compete for resources with say a product search flow. Even with a monolith, you wouldn't do this - you'd make an exception. In a few years, the monolith is full of special cases which only a few people know about. When those employees leave, the project sometimes stalls and code quality drops. Related to (2). 5. Microservices have become a lot easier thanks to k8s and docker. If you think about it, microservices were becoming popular even before k8s became mainstream. If it was viable then, it's a lot easier today. 6. It helps with organizing teams and assigning responsibility. 7. You don't need super small microservices. A microservice could very well handle all of a module - say all of payments (payment processing, refunds, coupon codes etc), or all of authentication (oauth, mfa etc). 8. Broken Windows Theory more often applies to monoliths, and much less to microservices. Delivery pressure is unavoidable in product development at various points. Which means that you'll often make compromises. Once you start making these compromises, people will keep making them more often. 9. It allows you the agility to choose a more efficient tech/process when available. Monoliths are rigid in tech choices, and don't easily allow you to adopt a different programming language or a framework. With Microservices, you could choose the stack that best solves the problem at hand. In addition, this allows a company to scale up the team faster. Add: 10. It's difficult to fix schemas, contracts and data structures once they're in production. Refactoring is easier with microservices, given that the implications are local compared to monoliths.

dlisboa 3y ago
Yes! Finally! People keep making the assumption that going for Services/Microservices is merely technical. It’s almost all about people and organization. Point 7 is the most important of the technical considerations: just make your services big enough to make sense as a separate unit and small enough to not be another unchangeable monolith. Yes, it’s not sane to have 300 Lambdas that add one number to another talking to each other over network, so just don’t do it. Microservices the “Netflix way”, with the huge graph of services, gives a bad name to the idea of factoring out modules into independently deployable parts, which always made sense. Kubernetes just helps with deployment, but how coarse or fine you factor is on you.