9 ms·

Meta’s Microservice Architecture [pdf]

115 points by mfiguiere 3y ago

rapsey 3y ago
So many companies shoot themselves in the foot chasing Google/Meta/Netflix backend architectures. If your developer count is in the 10s, you are committing professional negligence chasing a microservices architecture.
- danwee 3y ago
  Indeed, but I think we are too late for that. It's not like the usage of GOTO: someone who mattered wrote that its usage should be considered harmful, and hence nowadays GOTO has practically a niche usage. If only someone that mattered would have written on time an essay titled "Microservices Architecture Considered Harmful".
  - randomdata 3y ago
    > hence nowadays GOTO has practically a niche usage. Except where it was rebranded as throw/catch. There, goto remains quite popular. All while bridled (C-style) gotos were considered acceptable by Dijstrka, but are now looked upon with disdain. Amazing what a little marketing can do.
    - gordian-not 3y ago
      Aren't you talking about break and continue? throw and catch unwind the stack and goto doesn't
      - randomdata 3y ago
        break, continue, and even goto in any language created in the last several decades cannot leave the current function. These are not what Dijkstra was talking about. Dijkstra wrote the piece when structured programming was just starting to become a thing and was urging people towards it. Think more like goto in BASIC, where there is no bridling of the operator.
        gordian-not 3y ago
        Yes, however I am not sure if Dijkstra meant goto in the sense of jump outside of a function. I don't know enough about Algol 60 and languages of the time, but if you allow usage of goto between different functions you will have a corrupt stack in no time, so I'll be surprised if it was implemented. Going back to the original discussion, my point was that the only statements that are real gotos still in usage (except for C cleanup gotos) is break and continue, which even support labels
        randomdata 3y ago
        Functions do not exist in the context of the "Go to statement considered harmful" paper, so there is no good analog in there. Further, he is specific that it is only about unbridled gotos. continue and break are decidedly bridled. In reality, the paper just doesn't apply to any language created in the last several decades, even those which use the goto keyword. We bought in to structured programming. Except I posit that throw/catch still suffers the same problem he speaks of. It becomes difficult to follow when and where the code will jump to an arbitrary spot. The harmful parts of goto live on. Granted, we are learning. Modern languages are abandoning the throw/catch concept.
        gordian-not 3y ago
        I agree about the somewhat unexpectedness of exceptions, although it is much structured than gotos (you just go up the stack) virtual polymorphism for example is completely unexpected, as well as function pointers and other constructs that in my opinion are harder to track than exceptions
    - whstl 3y ago
      Almost everything is a "rebranded goto". Functions, conditions, iteration, break/continue. Doesn't mean that Dijkstra was wrong, or that they are the same as goto. Also, throw/catch do quite more than a goto, though. It's not only about stack unwinding, it's about the ergonomics of not needing code that uses throw to have any information about where catch is located. Sure you can simulate some use cases of try/catch with goto, but implementing the general case is much harder. throw/catch is exactly the kind of good goto-replacement provided by "higher level" programming languages that Dijkstra is talking about in his paper.
      - gjvc 3y ago
        > Almost everything is a "rebranded goto". Functions, conditions, iteration, break/continue. That's like saying all maths is just the application of addition.
        ubercore 3y ago
        I think that's the point -- it's nearly as useless to call throw/catch a rebranded goto as it is anything else.
        whstl 3y ago
        Yeah. It’s a useless pedantism that serves no purpose in most discussions. That’s what I have “rebranded goto” in quotes. And that’s what I address in the second paragraph.
        gjvc 3y ago
        it failed to satisfy me
      - randomdata 3y ago
        > Almost everything is a "rebranded goto". Functions, conditions, iteration, break/continue. Not within the context of discussion, where goto refers to the harmful kind. Throw/catch exhibits the very problem Dijkstra was talking about, with execution jumping all over the place haphazardly. Functions, conditions, iteration, break/continue, even (C-style) goto does not exhibit the same problem. They are strictly bridled in the execution scope. There is good reason why modern languages are moving away from throw/catch. However, there is little question that it is still widely used at this time.
    - patmorgan23 3y ago
      Throw is much worse than GOTO. GOTO is explicit you always know where it goes. Throw has no idea where catch is and catch has no idea where throw is. It's hidden control flow. Go/Rust/Zig Errors as values is a much better system forcing you to explicitly deal with the error, crash or pass it on. Rather than hoping you handled all the correct exceptions/someone else will handle all of them.
      - o1y32 3y ago
        If this were a top level comment I am sure you'll get 20 people disagree with you on this
        randomdata 3y ago
        Even Dijkstra followed up the "Go To Statement Considered Harmful" letter with the "On a Somewhat Disappointing Correspondence" letter because "20" people disagreed with him. There will always be naysayers. The above is starting to become generally accepted, though, and modern languages are moving away from the practice just as languages moved away from unbridled gotos.
  - fvdessen 3y ago
    The thing is they are not harmful, they are valuable above a certain scale. To caricature, monoliths scale with O(n) and microservices O(nlogn). At some point the lines cross and microservices start to get better. The problem is there's no clear answer to where is that point as it also depends on the product being built and other company specifics. But I wouldn't see it being usually worthwhile below a hundred devs.
    - jcparkyn 3y ago
      Your analogy is backwards, nlogn > n.
      - pc86 3y ago
        They're saying that microservices scale much worse than monoliths at the beginning, and need to reach a certain scale before they're worth the effort. The debate around is usually around where that inflection point is, and to a lesser degree if there are other non-scale advances to microservices that might make it worth adopting sooner.
      - nonameiguess 3y ago
        This comment was grayed out at the time I upvoted and am commenting. Either readers don't appreciate pedantry or don't realize what the point was, but for any n > 2, n * logn is a larger number than n. I believe what the original poster meant to say was microservices grow like O(klogn) and monoliths O(n), i.e. microservices have some upfront constant cost that is large enough that, for small projects, monoliths will still perform adequately and be cheaper to develop and deploy. Once you exceed n large enough to overcome that upfront constant cost, microservices become the superior option. I'm not agreeing or disagreeing with this, by the way.
        fvdessen 3y ago
        Indeed that's what I meant, but cannot edit my comment.
        inkyoto 3y ago
  - pjmlp 3y ago
    Thing is, contrary to GOTO, microservices help selling books, conferences and consulting services.
- guiriduro 3y ago
  It often ends up as a micro/monolith, a hybrid architecture with few benefits and many costs especially around deployment and distributed tracing, with many resumes polished on the way, but at least the return to a sensible monolith isn't quite as far.
- SanderNL 3y ago
  I count myself among those who are almost irrationally opposed to microservice anything. However, in my quest for the holy monolith I came across small teams who against all odds were adequately functional despite this heretical paradigm. They were definitely building a distributed monolith which was a monster to run locally. But, it worked. They were shipping and most importantly it matched their culture of small isolated islands of functional specialists with as little communication between them as possible. I will refrain from commenting on the virtue of communicating as little as possible, but sometimes you just got to make the best of a situation.
  - neon_electro 3y ago
    Did it seem like that team was aware of Conway's Law and was actively considering it in their architectural/system design decisions?
    - 0xbadcafebee 3y ago
      They can consider it all they want, that won't change the outcome. It is inevitable.
      - jerf 3y ago
        You can't avoid its impact, but you can either work with it or work against it. I assume that was more the point.
        0xbadcafebee 3y ago
        I don't think you can, actually, from the level of a team. If you're designing a bridge, you can work with gravity, or against it. But if you're a car on the bridge, you can't work with or against it; the bridge either will hold you up or it won't, your car will either go or it won't. Conway's Law is about how your organization's communication structures work, and how that inevitably leads to specific outcomes. That outcome is an eventuality for the team, because the team can't change the communication structure. They can design the best damn microservice in the world, and the resulting use of it could still be shit, because the other teams aren't necessarily up to snuff. And you still have all the baggage of dealing with those intra-team, intra-service issues.
- afhammad 3y ago
  "chasing" is a good word
- perryizgr8 3y ago
  Why is that? Can you point me to some material that explains this in detail?
- marcus_holmes 3y ago
  but... but we'll need this so we can cope with webscale request volumes when our product (that has zero customers currently) launches and immediately goes viral! (/s, obviously, but I've heard variations on this plenty of times)
- scns 3y ago
  Many may know it but for those that don't: Enjoy https://www.youtube.com/watch?v=y8OnoxKotPQ https://www.youtube.com/watch?v=y8OnoxKotPQ
  - marcosdumay 3y ago
    Is there some Firefox mod that reads that y8Onox... part and displays something like "galactus" for easiness to reference? The fact that one has seen the video doesn't make the link recognizable. (And that's a problem with Youtube.)
  - ldjkfkdsjnv 3y ago
    One of the greatest depictions of writing software at a big tech company
  - pjmlp 3y ago
    I discovered it a couple of weeks ago. It is great.
  - belter 3y ago
    As somebody said in the comments..Not a Comedy but a Documentary....
- FragrantRiver 3y ago
  Microservices are great, even for small companies. Change my mind.
  - onion-soup 3y ago
    You can safely continue your professionnal jjourney elsewhere but not in IT. Thank you!
- YetAnotherNick 3y ago
  This is hn so obviously your comment has to be the top voted comment. I am really tired of this trend here. No. microservice vs monolith is not the deciding factor of the developer speed or bugs. It wouldn't even be top 5 factor for good developers. Difference between microservices and monolith technically is just network rpc rather than function call. In itself, it doesn't make much of a difference. If a developer finds themselves stuck because of microservice architecture, they are probably very bad developer in first place
  - onion-soup 3y ago
    We all appreciate your contribution to micro services. But let us now choose simplicity and move on with our lives. You do you with your zoo of dockerized babel towers
    - YetAnotherNick 3y ago
      I use monolith 90% of the time. I think you missed my point. I just said that I could be similar level of productive in well thought of microservices division.
  - fvdessen 3y ago
    The fact some people think there's little difference between function calls and RPCs is exactly why microservices is such a dangerous idea.
  - vilunov 3y ago
    Placing a network call in your system also introduces: - distributed systems problems - think consistency and ACID - increased refactoring complexity - you now have to change the api three times instead of 1 to deploy the change with 0 downtime - requirements for more generic CI/CD pipelines - you will develop many microservices, it's important to do it fast it's not "just" a network rpc, even technically
    - Cthulhu_ 3y ago
      If it's a network RPC style microservices architecture, you've built a distributed monolith; every service in a proper microservices architecture should be developed, maintained and deployed 100% independently from any other service. If two services are closely tied, they should be merged together. Yes this causes overhead; every microservice will have a public API of sorts that will need to be documented and communicated, and any other service consuming it will need to eventually be updated to keep up with changes. If this overhead is not something your organization can afford, you should not be doing a microservices architecture.
- DeathArrow 3y ago
  For us at my present and former workplaces, the decision of using microservices didn't depend solely on number of developers. We needed something very scalable, something that different teams can work on without stepping on each other's foot, something that survives even if part of it fails temporarily, something that auto-heals. We did it with developers in the tens and we didn't have much issues with this approach. In fact, at one of my workplaces we had much more issues with a monolithic app than with the microservice based app we replaced it with.
  - fvdessen 3y ago
    How the hell does microservices 'auto-heals' ? Do they autogenerate bug fixes and patch themselves ?
    - DeathArrow 3y ago
      Simple, if a service is not answering for a certain amount of time, a new Kubernetes pod is brought up and the old one is killed. :)
      - fvdessen 3y ago
        That also works with monoliths ... Usually you would make your monolith stateless and distribute the incoming requests / events across many instances that can be spawned / killed depending on volume of requests and health status of instances.
        tomthumb 3y ago
        The same logic holds true for micro services. Statelessness is the key. But most micro services implementations end up being distributed monoliths
        philjohn 3y ago
        In a past job, the benefit of microservices was that some of the operations performed by the system were fare more CPU intensive than others - by having them in their own service that could be scaled independently led to lower overall hardware requirements, and made keeping latency of the other services sensible much easier.
- lijok 3y ago
  > If your developer count is in the 10s, you are committing professional negligence chasing a microservices architecture. Such a strong statement. Might a lack of industry experience be driving such strong convictions of yours? Here are some reasons, off the top of my head, why a company would want to embrace microservice architecture, with all its benefits and complexities, with a developer count in the 10s: 1. You're building a product that touches multiple deep domains and the business has modelled a very aggressive headcount growth 2. You've outsourced a large portion of your development to a number of agencies 3. You have a team of individuals who know nothing but microservices 4. Your chief compliance officer is intimately familiar with the data protection benefits that microservices bring and is leaning heavily into it in their regulatory submissions as a way to compensate for some other gap in the business 5. You're building anything to do with image processing at scale 6. You are a subsidiary owned by a parent company with tons of experience and tooling for microservices 7. One of your VCs has offered up a dev team they own to speedboat your MVP, who specialize in microservices 8. You've received a buyout offer by a party interested in specific IP within your product, with the condition that the IP is isolated from other parts of the system
  - rkwz 3y ago
    > Your chief compliance officer is intimately familiar with the data protection benefits that microservices bring and is leaning heavily into it in their regulatory submissions as a way to compensate for some other gap in the business Just curious, how does microservices help here?
    - lijok 3y ago
      Great question. Two things you're aiming for with a regulatory submission: 1) Get it approved. 2) Receive back as few rounds of questions as possible from the case officer, as each round delays approval. You can improve 2) by painting a very clear and concise narrative around key concerns pertaining to the particular application you're submitting. Microservice architecture helps to paint that narrative (for CCOs that understand them) as at the high level, it addresses the key concerns (primarily data protection) with simple to convey concepts such as data segregation, service ownership, etc. The implementation of those controls is not as simple, but for the regulatory application, all that matters is that they are sound in practice and simple to convey.
- collyw 3y ago
  Agreed. In my last company we had everything running in Kubernetes, despite having less than 300 active users. It did have some benefits, but scaling wasn't one that we needed and did cost lot of developer time. Debugging is painful.
- outworlder 3y ago
  Many companies implement microservices for the wrong reasons. The most common reason is because other companies are doing it. The second most common bad reason is because business units don't want to talk to one another. AKA they are shipping their org chart. It's great for cloud providers too. It's often cheaper to just run entire copies of monoliths than it is to run microservices. All that synchronization and API calls has a huge cost, even more so when you add support infrastructure (RMQ? Kubernetes?) Most companies would work just fine with a monolith, with the occasional service split off from that when it makes sense.
  - ibejoeb 3y ago
    I don't read it as your intention, but that comes off putting it on non-technical management. It sure seems like a lot of the microservices plays come from technologists wanting to use it as OTJ training. There are lots of neat things to experiment with, and there are lots of ways to glue things together. There's a whole resume to build with all of the ancillary tech required just to run a process. Your point about cloud is right on, too. That goes both ways. Some CEOs chase the deals. But some CTOs chase the greener-grass dragon. One thing leads to another, and suddenly you're migrating from k8s on GCP to nomad on DO. > with the occasional service split off from that when it makes sense. Exactly what meta does, according to the article. There it is.
  - rewmie 3y ago
    > The second most common bad reason is because business units don't want to talk to one another. This sounds like a terribly naive take. It matters nothing if units talk to one another. Nothing. All it matters is ownership and accountability. What happens if the service managed by team A goes down and takes the whole org with it? Is team B going to take the blame because one of their developers posted a PR to tweak the project's README?
    - MBCook 3y ago
      It’s real. I’ve seen it. They can’t get their internal silos to work together on a single project, so they have each silo make services for the others to use in service of the project they’re trying to make that needs multiple teams. But… all the same problems come up. Teams building to their requirements and not what the consumers need, refusal to cooperate in design/requirements/etc, scheduling issues or project priority issues. “Microservices are cool and what Google does and will help us avoid our organizational issues” is very loosely paraphrased from what I heard as the exact pitch for why it was being used.
- gooseclip 3y ago
  The big differentiator is how micro do you go. My rule of thumb is split by scaling requirements (and hardware requirements if you have them). For example splitting off your general business API from your TURN servers/service since the TURN should be scaled by connections and throughput plus possibly requires a higher network bandwidth.
who_is_this 3y ago
water
deleted 3y ago
[deleted]
nologic01 3y ago
> Services are defined as units of software with well-defined API interfaces, called endpoints ( in Figure 1). Each service satisfies a specific business use case (e.g., cachinga photo feed). There is significant room for interpretation in defining the scope of a business use case... It seems to me the problem with the microservice concept does not come from the decomposition of a monolith into well defined stateful/stateless services but indeed the lack of further principles and tools for doing so in an optimal way. Imagine a decision process that would ingest some quantitative details (including a topology of dependencies) and volumes of information flow and computation etc and spitting out an architecture. In many cases the outcome might be "monolith". While fads and bandwagons promising utopia are prevalent in tech as in almost all other domain, there usually is a core truth that activates them.
- oersted 3y ago
  As I understand it, the concept of microservices mainly originated to solve organisational bottlenecks, rather than to improve the quality of the software itself. Microservices are organized by business use-case in order to isolate a team so they can focus in deeply understanding the requirements of this specific domain. This is fundamentally a fuzzy and human division of concerns, and has little to do with the performance factors you state, which seems to be a secondary priority. Mantainability and code-quality are related concepts, but they don't seem to be the top priority either, for them it seems that it is more important to enable a team to attain deep knowledge of the problem-space assigned to them, rather than ease in implementing solutions. Sure microservices are supposed to handle scale, but are we talking about computational scale or organizational scale? It is ambiguous, but I think it's more of the latter. That's probably why so many technical issues have surfaced surrounding this architecture after the initial hype, it was not about technical efficiency to begin with, and it only makes sense for very large teams of engineers, if at that. Just my interpretation, I don't claim to be more qualified than any other professional here to have an opinion on it.
gigatexal 3y ago
obligatory: you are not Netflix/Meta/Google... so you don't need to do this architecture for your own startup. Just spin up a monolith until you have something that needs breaking down and even then it might not need it or you might pivot to doing something else entirely.
- Cthulhu_ 3y ago
  Exactly that. I also think a lot of developer teams are both too big, too unorganized, and not diligent enough so they start to interfere with each other's work, causing them to want to have their own codebase. But no, you need a strong lead developer and development practices. There may be too many cooks in the kitchen as well, at which point you should probably consider downsizing.
onion-soup 3y ago
Holy shit. No wonder it failed miserably.
- yankoff 3y ago
  in what way did it fail?
phyrex 3y ago
I find this highly misleading, Facebook is famously a big monolith, and I think so is instagram. There’s plenty of services and a couple micro services as well, but I don’t think anybody would characterize meta as having a micro services architecture. I have no idea what the authors’ agendas are, but something isn’t right there.
- MarkSweep 3y ago
  The article mentions that www, the monolithic PHP code base, is 4.6% of the service instances. Though the paper does not mention the compute allocation. I do not recall how www is deployed, but it’s easy to imagine it’s allocated a lot more resources than a lot of other services are.
  - phyrex 3y ago
    > At Meta, 0.1% and 1% of the largest services consume 40% and 80% of the total fleet capacity, respectively. from https://www.usenix.org/system/files/osdi23-grubic.pdf https://www.usenix.org/system/files/osdi23-grubic.pdf
  - yuliyp 3y ago
    Large, unsharded services are generally going to run one instance per machine alone as that's the most efficient way to run most of them (reduce per-instance overhead as much as possible). Small (true "micro") services or other things which have instances per use case (like the inference platform example given in the paper) may use a small fraction of a machine.
- yuliyp 3y ago
  There is a Hack monolith (the www tier). There are also a huge number of other services, ranging from micro with a couple instances globally to "larger than almost anyone elses's monolith".
jposor 3y ago
[dead]
AndrewPGameDev 3y ago
I've had this thought for a long time that if you have a completely functional code base (as in no side-effects), making the decision between a microservice approach and a monolith approach is fundamentally transparent. Any functional code can not only be split onto multiple threads, but multiple servers (at the cost of latency).
- Cthulhu_ 3y ago
  This was the Scala / Akka approach, I'm sure there's similar ones in other languages; basically you'd work via communicating, 'pushing' messages to actors instead of calling functions. From a developer's point of view, it then didn't matter if that message went to an actor on the same machine or something halfway across the world. "Don't communicate by sharing memory, share memory by communicating." is the modern adage.
- eldenring 3y ago
  Multiple servers adds a pretty hefty layer of networking, orchestration, security, failure handling (even in the datacenter), and serialization - even if the business logic stays mostly the same.
- jcparkyn 3y ago
  Any purely functional app still needs a mutable data store _somewhere_. And if that's all owned by one service (the "root" of your functional app, since everything else is pure by definition), then you're hardly doing microservices.
- dugmartin 3y ago
  Except in the microservice approach with multiple servers you have introduced the possibility of network partitioning where microservice A can't reach microservice B. Unless your language/runtime model can deal with that you have hidden that error case in your abstraction. I think a lot of microservice architectures are implemented with the YOLO model where there is no thought to network errors and all calls are assumed to succeed. Language/runtimes like Erlang & Elixir on the BEAM/OTP that were built with this in mind work well. They are completely functional with all state contained with lightweight processes and can send messages with timeout handling transparently across nodes in a cluster. With OTP you get a supervision tree for those processes that can automatically restart nodes and child processes of nodes.
revskill 3y ago
Universe is composed of atom, and so is software. Microservice is the atom.
ibejoeb 3y ago
One of the services exposes 11,359 endpoints. Well, one interpretation is that Meta runs predominantly on a monolith but has factored out a few services.
- phyrex 3y ago
  that's exactly what it is
steveBK123 3y ago
The trick is to use a slow enough language so no one notices the IPC hit of micro services.
- steveBK123 3y ago
  Man the quickest way to get downvoted here is to dunk on Python huh?
  - mentaltomorrow 3y ago
    What Python? Who said Python? lol
    - steveBK123 3y ago
      Exactly - a microservice architecture in python is an idea architecture for any small business!
covi 3y ago
Meta's infra and system teams have been on a roll. Feels like Google in the 2000s. Check out quite a few interesting papers from them in OSDI 2023 (top systems conference, co-located with ATC where the OP URL links to): https://www.usenix.org/conference/osdi23/technical-sessions https://www.usenix.org/conference/osdi23/technical-sessions
MarkSweep 3y ago
> Many frontend and some backend services also expose numerous HTTP (REST and GraphQL) endpoints; however, they do not have canonical names that we can use for our analyses. For this reason, we limit the endpoint analysis only to Thrift RPCs reported in the dataset from the routing library. That’s too bad. One of the funnest thing is when a backend service calls into www, the monolith PHP code base. For example, callbacks when an item enters a queue or video call changes state. And by fun, I mean annoying to debug.
totallywrong 3y ago
It's cool to hate on microservices here on HN. In my opinion it's not always the answer but can bring many benefits. For example the ability to use different languages or frameworks that might be better suited for parts of the application, the separation of concerns, the ability to split work between teams, outsourcing parts of the app, scaling parts of the app individually, etc. They key is to have clear API contracts for each service, and enforce them ruthlessly.

115 points by mfiguiere 3y ago

So many companies shoot themselves in the foot chasing Google/Meta/Netflix backend architectures. If your developer count is in the 10s, you are committing professional negligence chasing a microservices architecture.

danwee 3y ago
Indeed, but I think we are too late for that. It's not like the usage of GOTO: someone who mattered wrote that its usage should be considered harmful, and hence nowadays GOTO has practically a niche usage. If only someone that mattered would have written on time an essay titled "Microservices Architecture Considered Harmful".
- randomdata 3y ago
  > hence nowadays GOTO has practically a niche usage. Except where it was rebranded as throw/catch. There, goto remains quite popular. All while bridled (C-style) gotos were considered acceptable by Dijstrka, but are now looked upon with disdain. Amazing what a little marketing can do.
  - gordian-not 3y ago
    Aren't you talking about break and continue? throw and catch unwind the stack and goto doesn't
    - randomdata 3y ago
      break, continue, and even goto in any language created in the last several decades cannot leave the current function. These are not what Dijkstra was talking about. Dijkstra wrote the piece when structured programming was just starting to become a thing and was urging people towards it. Think more like goto in BASIC, where there is no bridling of the operator.
      - gordian-not 3y ago
        Yes, however I am not sure if Dijkstra meant goto in the sense of jump outside of a function. I don't know enough about Algol 60 and languages of the time, but if you allow usage of goto between different functions you will have a corrupt stack in no time, so I'll be surprised if it was implemented. Going back to the original discussion, my point was that the only statements that are real gotos still in usage (except for C cleanup gotos) is break and continue, which even support labels
        randomdata 3y ago
        Functions do not exist in the context of the "Go to statement considered harmful" paper, so there is no good analog in there. Further, he is specific that it is only about unbridled gotos. continue and break are decidedly bridled. In reality, the paper just doesn't apply to any language created in the last several decades, even those which use the goto keyword. We bought in to structured programming. Except I posit that throw/catch still suffers the same problem he speaks of. It becomes difficult to follow when and where the code will jump to an arbitrary spot. The harmful parts of goto live on. Granted, we are learning. Modern languages are abandoning the throw/catch concept.
        gordian-not 3y ago
        I agree about the somewhat unexpectedness of exceptions, although it is much structured than gotos (you just go up the stack) virtual polymorphism for example is completely unexpected, as well as function pointers and other constructs that in my opinion are harder to track than exceptions
  - whstl 3y ago
    Almost everything is a "rebranded goto". Functions, conditions, iteration, break/continue. Doesn't mean that Dijkstra was wrong, or that they are the same as goto. Also, throw/catch do quite more than a goto, though. It's not only about stack unwinding, it's about the ergonomics of not needing code that uses throw to have any information about where catch is located. Sure you can simulate some use cases of try/catch with goto, but implementing the general case is much harder. throw/catch is exactly the kind of good goto-replacement provided by "higher level" programming languages that Dijkstra is talking about in his paper.
    - gjvc 3y ago
      > Almost everything is a "rebranded goto". Functions, conditions, iteration, break/continue. That's like saying all maths is just the application of addition.
      - ubercore 3y ago
        I think that's the point -- it's nearly as useless to call throw/catch a rebranded goto as it is anything else.
      - whstl 3y ago
        Yeah. It’s a useless pedantism that serves no purpose in most discussions. That’s what I have “rebranded goto” in quotes. And that’s what I address in the second paragraph.
        gjvc 3y ago
        it failed to satisfy me
    - randomdata 3y ago
      > Almost everything is a "rebranded goto". Functions, conditions, iteration, break/continue. Not within the context of discussion, where goto refers to the harmful kind. Throw/catch exhibits the very problem Dijkstra was talking about, with execution jumping all over the place haphazardly. Functions, conditions, iteration, break/continue, even (C-style) goto does not exhibit the same problem. They are strictly bridled in the execution scope. There is good reason why modern languages are moving away from throw/catch. However, there is little question that it is still widely used at this time.
  - patmorgan23 3y ago
    Throw is much worse than GOTO. GOTO is explicit you always know where it goes. Throw has no idea where catch is and catch has no idea where throw is. It's hidden control flow. Go/Rust/Zig Errors as values is a much better system forcing you to explicitly deal with the error, crash or pass it on. Rather than hoping you handled all the correct exceptions/someone else will handle all of them.
    - o1y32 3y ago
      If this were a top level comment I am sure you'll get 20 people disagree with you on this
      - randomdata 3y ago
        Even Dijkstra followed up the "Go To Statement Considered Harmful" letter with the "On a Somewhat Disappointing Correspondence" letter because "20" people disagreed with him. There will always be naysayers. The above is starting to become generally accepted, though, and modern languages are moving away from the practice just as languages moved away from unbridled gotos.
- fvdessen 3y ago
  The thing is they are not harmful, they are valuable above a certain scale. To caricature, monoliths scale with O(n) and microservices O(nlogn). At some point the lines cross and microservices start to get better. The problem is there's no clear answer to where is that point as it also depends on the product being built and other company specifics. But I wouldn't see it being usually worthwhile below a hundred devs.
  - jcparkyn 3y ago
    Your analogy is backwards, nlogn > n.
    - pc86 3y ago
      They're saying that microservices scale much worse than monoliths at the beginning, and need to reach a certain scale before they're worth the effort. The debate around is usually around where that inflection point is, and to a lesser degree if there are other non-scale advances to microservices that might make it worth adopting sooner.
    - nonameiguess 3y ago
      This comment was grayed out at the time I upvoted and am commenting. Either readers don't appreciate pedantry or don't realize what the point was, but for any n > 2, n * logn is a larger number than n. I believe what the original poster meant to say was microservices grow like O(klogn) and monoliths O(n), i.e. microservices have some upfront constant cost that is large enough that, for small projects, monoliths will still perform adequately and be cheaper to develop and deploy. Once you exceed n large enough to overcome that upfront constant cost, microservices become the superior option. I'm not agreeing or disagreeing with this, by the way.
      - fvdessen 3y ago
        Indeed that's what I meant, but cannot edit my comment.
      - inkyoto 3y ago
- pjmlp 3y ago
  Thing is, contrary to GOTO, microservices help selling books, conferences and consulting services.
guiriduro 3y ago
It often ends up as a micro/monolith, a hybrid architecture with few benefits and many costs especially around deployment and distributed tracing, with many resumes polished on the way, but at least the return to a sensible monolith isn't quite as far.
SanderNL 3y ago
I count myself among those who are almost irrationally opposed to microservice anything. However, in my quest for the holy monolith I came across small teams who against all odds were adequately functional despite this heretical paradigm. They were definitely building a distributed monolith which was a monster to run locally. But, it worked. They were shipping and most importantly it matched their culture of small isolated islands of functional specialists with as little communication between them as possible. I will refrain from commenting on the virtue of communicating as little as possible, but sometimes you just got to make the best of a situation.
- neon_electro 3y ago
  Did it seem like that team was aware of Conway's Law and was actively considering it in their architectural/system design decisions?
  - 0xbadcafebee 3y ago
    They can consider it all they want, that won't change the outcome. It is inevitable.
    - jerf 3y ago
      You can't avoid its impact, but you can either work with it or work against it. I assume that was more the point.
      - 0xbadcafebee 3y ago
        I don't think you can, actually, from the level of a team. If you're designing a bridge, you can work with gravity, or against it. But if you're a car on the bridge, you can't work with or against it; the bridge either will hold you up or it won't, your car will either go or it won't. Conway's Law is about how your organization's communication structures work, and how that inevitably leads to specific outcomes. That outcome is an eventuality for the team, because the team can't change the communication structure. They can design the best damn microservice in the world, and the resulting use of it could still be shit, because the other teams aren't necessarily up to snuff. And you still have all the baggage of dealing with those intra-team, intra-service issues.
afhammad 3y ago
"chasing" is a good word
perryizgr8 3y ago
Why is that? Can you point me to some material that explains this in detail?
marcus_holmes 3y ago
but... but we'll need this so we can cope with webscale request volumes when our product (that has zero customers currently) launches and immediately goes viral! (/s, obviously, but I've heard variations on this plenty of times)
scns 3y ago
Many may know it but for those that don't: Enjoy https://www.youtube.com/watch?v=y8OnoxKotPQ https://www.youtube.com/watch?v=y8OnoxKotPQ
- marcosdumay 3y ago
  Is there some Firefox mod that reads that y8Onox... part and displays something like "galactus" for easiness to reference? The fact that one has seen the video doesn't make the link recognizable. (And that's a problem with Youtube.)
- ldjkfkdsjnv 3y ago
  One of the greatest depictions of writing software at a big tech company
- pjmlp 3y ago
  I discovered it a couple of weeks ago. It is great.
- belter 3y ago
  As somebody said in the comments..Not a Comedy but a Documentary....
FragrantRiver 3y ago
Microservices are great, even for small companies. Change my mind.
- onion-soup 3y ago
  You can safely continue your professionnal jjourney elsewhere but not in IT. Thank you!
YetAnotherNick 3y ago
This is hn so obviously your comment has to be the top voted comment. I am really tired of this trend here. No. microservice vs monolith is not the deciding factor of the developer speed or bugs. It wouldn't even be top 5 factor for good developers. Difference between microservices and monolith technically is just network rpc rather than function call. In itself, it doesn't make much of a difference. If a developer finds themselves stuck because of microservice architecture, they are probably very bad developer in first place
- onion-soup 3y ago
  We all appreciate your contribution to micro services. But let us now choose simplicity and move on with our lives. You do you with your zoo of dockerized babel towers
  - YetAnotherNick 3y ago
    I use monolith 90% of the time. I think you missed my point. I just said that I could be similar level of productive in well thought of microservices division.
- fvdessen 3y ago
  The fact some people think there's little difference between function calls and RPCs is exactly why microservices is such a dangerous idea.
- vilunov 3y ago
  Placing a network call in your system also introduces: - distributed systems problems - think consistency and ACID - increased refactoring complexity - you now have to change the api three times instead of 1 to deploy the change with 0 downtime - requirements for more generic CI/CD pipelines - you will develop many microservices, it's important to do it fast it's not "just" a network rpc, even technically
  - Cthulhu_ 3y ago
    If it's a network RPC style microservices architecture, you've built a distributed monolith; every service in a proper microservices architecture should be developed, maintained and deployed 100% independently from any other service. If two services are closely tied, they should be merged together. Yes this causes overhead; every microservice will have a public API of sorts that will need to be documented and communicated, and any other service consuming it will need to eventually be updated to keep up with changes. If this overhead is not something your organization can afford, you should not be doing a microservices architecture.
DeathArrow 3y ago
For us at my present and former workplaces, the decision of using microservices didn't depend solely on number of developers. We needed something very scalable, something that different teams can work on without stepping on each other's foot, something that survives even if part of it fails temporarily, something that auto-heals. We did it with developers in the tens and we didn't have much issues with this approach. In fact, at one of my workplaces we had much more issues with a monolithic app than with the microservice based app we replaced it with.
- fvdessen 3y ago
  How the hell does microservices 'auto-heals' ? Do they autogenerate bug fixes and patch themselves ?
  - DeathArrow 3y ago
    Simple, if a service is not answering for a certain amount of time, a new Kubernetes pod is brought up and the old one is killed. :)
    - fvdessen 3y ago
      That also works with monoliths ... Usually you would make your monolith stateless and distribute the incoming requests / events across many instances that can be spawned / killed depending on volume of requests and health status of instances.
      - tomthumb 3y ago
        The same logic holds true for micro services. Statelessness is the key. But most micro services implementations end up being distributed monoliths
      - philjohn 3y ago
        In a past job, the benefit of microservices was that some of the operations performed by the system were fare more CPU intensive than others - by having them in their own service that could be scaled independently led to lower overall hardware requirements, and made keeping latency of the other services sensible much easier.
lijok 3y ago
> If your developer count is in the 10s, you are committing professional negligence chasing a microservices architecture. Such a strong statement. Might a lack of industry experience be driving such strong convictions of yours? Here are some reasons, off the top of my head, why a company would want to embrace microservice architecture, with all its benefits and complexities, with a developer count in the 10s: 1. You're building a product that touches multiple deep domains and the business has modelled a very aggressive headcount growth 2. You've outsourced a large portion of your development to a number of agencies 3. You have a team of individuals who know nothing but microservices 4. Your chief compliance officer is intimately familiar with the data protection benefits that microservices bring and is leaning heavily into it in their regulatory submissions as a way to compensate for some other gap in the business 5. You're building anything to do with image processing at scale 6. You are a subsidiary owned by a parent company with tons of experience and tooling for microservices 7. One of your VCs has offered up a dev team they own to speedboat your MVP, who specialize in microservices 8. You've received a buyout offer by a party interested in specific IP within your product, with the condition that the IP is isolated from other parts of the system
- rkwz 3y ago
  > Your chief compliance officer is intimately familiar with the data protection benefits that microservices bring and is leaning heavily into it in their regulatory submissions as a way to compensate for some other gap in the business Just curious, how does microservices help here?
  - lijok 3y ago
    Great question. Two things you're aiming for with a regulatory submission: 1) Get it approved. 2) Receive back as few rounds of questions as possible from the case officer, as each round delays approval. You can improve 2) by painting a very clear and concise narrative around key concerns pertaining to the particular application you're submitting. Microservice architecture helps to paint that narrative (for CCOs that understand them) as at the high level, it addresses the key concerns (primarily data protection) with simple to convey concepts such as data segregation, service ownership, etc. The implementation of those controls is not as simple, but for the regulatory application, all that matters is that they are sound in practice and simple to convey.
collyw 3y ago
Agreed. In my last company we had everything running in Kubernetes, despite having less than 300 active users. It did have some benefits, but scaling wasn't one that we needed and did cost lot of developer time. Debugging is painful.
outworlder 3y ago
Many companies implement microservices for the wrong reasons. The most common reason is because other companies are doing it. The second most common bad reason is because business units don't want to talk to one another. AKA they are shipping their org chart. It's great for cloud providers too. It's often cheaper to just run entire copies of monoliths than it is to run microservices. All that synchronization and API calls has a huge cost, even more so when you add support infrastructure (RMQ? Kubernetes?) Most companies would work just fine with a monolith, with the occasional service split off from that when it makes sense.
- ibejoeb 3y ago
  I don't read it as your intention, but that comes off putting it on non-technical management. It sure seems like a lot of the microservices plays come from technologists wanting to use it as OTJ training. There are lots of neat things to experiment with, and there are lots of ways to glue things together. There's a whole resume to build with all of the ancillary tech required just to run a process. Your point about cloud is right on, too. That goes both ways. Some CEOs chase the deals. But some CTOs chase the greener-grass dragon. One thing leads to another, and suddenly you're migrating from k8s on GCP to nomad on DO. > with the occasional service split off from that when it makes sense. Exactly what meta does, according to the article. There it is.
- rewmie 3y ago
  > The second most common bad reason is because business units don't want to talk to one another. This sounds like a terribly naive take. It matters nothing if units talk to one another. Nothing. All it matters is ownership and accountability. What happens if the service managed by team A goes down and takes the whole org with it? Is team B going to take the blame because one of their developers posted a PR to tweak the project's README?
  - MBCook 3y ago
    It’s real. I’ve seen it. They can’t get their internal silos to work together on a single project, so they have each silo make services for the others to use in service of the project they’re trying to make that needs multiple teams. But… all the same problems come up. Teams building to their requirements and not what the consumers need, refusal to cooperate in design/requirements/etc, scheduling issues or project priority issues. “Microservices are cool and what Google does and will help us avoid our organizational issues” is very loosely paraphrased from what I heard as the exact pitch for why it was being used.
gooseclip 3y ago
The big differentiator is how micro do you go. My rule of thumb is split by scaling requirements (and hardware requirements if you have them). For example splitting off your general business API from your TURN servers/service since the TURN should be scaled by connections and throughput plus possibly requires a higher network bandwidth.

who_is_this 3y ago

water

deleted 3y ago

[deleted]

nologic01 3y ago

> Services are defined as units of software with well-defined API interfaces, called endpoints ( in Figure 1). Each service satisfies a specific business use case (e.g., cachinga photo feed). There is significant room for interpretation in defining the scope of a business use case... It seems to me the problem with the microservice concept does not come from the decomposition of a monolith into well defined stateful/stateless services but indeed the lack of further principles and tools for doing so in an optimal way. Imagine a decision process that would ingest some quantitative details (including a topology of dependencies) and volumes of information flow and computation etc and spitting out an architecture. In many cases the outcome might be "monolith". While fads and bandwagons promising utopia are prevalent in tech as in almost all other domain, there usually is a core truth that activates them.

oersted 3y ago
As I understand it, the concept of microservices mainly originated to solve organisational bottlenecks, rather than to improve the quality of the software itself. Microservices are organized by business use-case in order to isolate a team so they can focus in deeply understanding the requirements of this specific domain. This is fundamentally a fuzzy and human division of concerns, and has little to do with the performance factors you state, which seems to be a secondary priority. Mantainability and code-quality are related concepts, but they don't seem to be the top priority either, for them it seems that it is more important to enable a team to attain deep knowledge of the problem-space assigned to them, rather than ease in implementing solutions. Sure microservices are supposed to handle scale, but are we talking about computational scale or organizational scale? It is ambiguous, but I think it's more of the latter. That's probably why so many technical issues have surfaced surrounding this architecture after the initial hype, it was not about technical efficiency to begin with, and it only makes sense for very large teams of engineers, if at that. Just my interpretation, I don't claim to be more qualified than any other professional here to have an opinion on it.

gigatexal 3y ago

obligatory: you are not Netflix/Meta/Google... so you don't need to do this architecture for your own startup. Just spin up a monolith until you have something that needs breaking down and even then it might not need it or you might pivot to doing something else entirely.

Cthulhu_ 3y ago
Exactly that. I also think a lot of developer teams are both too big, too unorganized, and not diligent enough so they start to interfere with each other's work, causing them to want to have their own codebase. But no, you need a strong lead developer and development practices. There may be too many cooks in the kitchen as well, at which point you should probably consider downsizing.

onion-soup 3y ago

Holy shit. No wonder it failed miserably.

yankoff 3y ago
in what way did it fail?

phyrex 3y ago

I find this highly misleading, Facebook is famously a big monolith, and I think so is instagram. There’s plenty of services and a couple micro services as well, but I don’t think anybody would characterize meta as having a micro services architecture. I have no idea what the authors’ agendas are, but something isn’t right there.

MarkSweep 3y ago
The article mentions that www, the monolithic PHP code base, is 4.6% of the service instances. Though the paper does not mention the compute allocation. I do not recall how www is deployed, but it’s easy to imagine it’s allocated a lot more resources than a lot of other services are.
- phyrex 3y ago
  > At Meta, 0.1% and 1% of the largest services consume 40% and 80% of the total fleet capacity, respectively. from https://www.usenix.org/system/files/osdi23-grubic.pdf https://www.usenix.org/system/files/osdi23-grubic.pdf
- yuliyp 3y ago
  Large, unsharded services are generally going to run one instance per machine alone as that's the most efficient way to run most of them (reduce per-instance overhead as much as possible). Small (true "micro") services or other things which have instances per use case (like the inference platform example given in the paper) may use a small fraction of a machine.
yuliyp 3y ago
There is a Hack monolith (the www tier). There are also a huge number of other services, ranging from micro with a couple instances globally to "larger than almost anyone elses's monolith".

jposor 3y ago

[dead]

AndrewPGameDev 3y ago

I've had this thought for a long time that if you have a completely functional code base (as in no side-effects), making the decision between a microservice approach and a monolith approach is fundamentally transparent. Any functional code can not only be split onto multiple threads, but multiple servers (at the cost of latency).

Cthulhu_ 3y ago
This was the Scala / Akka approach, I'm sure there's similar ones in other languages; basically you'd work via communicating, 'pushing' messages to actors instead of calling functions. From a developer's point of view, it then didn't matter if that message went to an actor on the same machine or something halfway across the world. "Don't communicate by sharing memory, share memory by communicating." is the modern adage.
eldenring 3y ago
Multiple servers adds a pretty hefty layer of networking, orchestration, security, failure handling (even in the datacenter), and serialization - even if the business logic stays mostly the same.
jcparkyn 3y ago
Any purely functional app still needs a mutable data store _somewhere_. And if that's all owned by one service (the "root" of your functional app, since everything else is pure by definition), then you're hardly doing microservices.
dugmartin 3y ago
Except in the microservice approach with multiple servers you have introduced the possibility of network partitioning where microservice A can't reach microservice B. Unless your language/runtime model can deal with that you have hidden that error case in your abstraction. I think a lot of microservice architectures are implemented with the YOLO model where there is no thought to network errors and all calls are assumed to succeed. Language/runtimes like Erlang & Elixir on the BEAM/OTP that were built with this in mind work well. They are completely functional with all state contained with lightweight processes and can send messages with timeout handling transparently across nodes in a cluster. With OTP you get a supervision tree for those processes that can automatically restart nodes and child processes of nodes.

revskill 3y ago

Universe is composed of atom, and so is software. Microservice is the atom.

ibejoeb 3y ago

One of the services exposes 11,359 endpoints. Well, one interpretation is that Meta runs predominantly on a monolith but has factored out a few services.

phyrex 3y ago
that's exactly what it is

steveBK123 3y ago

The trick is to use a slow enough language so no one notices the IPC hit of micro services.

steveBK123 3y ago
Man the quickest way to get downvoted here is to dunk on Python huh?
- mentaltomorrow 3y ago
  What Python? Who said Python? lol
  - steveBK123 3y ago
    Exactly - a microservice architecture in python is an idea architecture for any small business!

covi 3y ago

Meta's infra and system teams have been on a roll. Feels like Google in the 2000s. Check out quite a few interesting papers from them in OSDI 2023 (top systems conference, co-located with ATC where the OP URL links to): https://www.usenix.org/conference/osdi23/technical-sessions https://www.usenix.org/conference/osdi23/technical-sessions

MarkSweep 3y ago

> Many frontend and some backend services also expose numerous HTTP (REST and GraphQL) endpoints; however, they do not have canonical names that we can use for our analyses. For this reason, we limit the endpoint analysis only to Thrift RPCs reported in the dataset from the routing library. That’s too bad. One of the funnest thing is when a backend service calls into www, the monolith PHP code base. For example, callbacks when an item enters a queue or video call changes state. And by fun, I mean annoying to debug.

totallywrong 3y ago

It's cool to hate on microservices here on HN. In my opinion it's not always the answer but can bring many benefits. For example the ability to use different languages or frameworks that might be better suited for parts of the application, the separation of concerns, the ability to split work between teams, outsourcing parts of the app, scaling parts of the app individually, etc. They key is to have clear API contracts for each service, and enforce them ruthlessly.