6 ms·

I am working on a project that uses a microservice architecture to make the individual components scalable and separate the concerns. However one of the unexpec

• tofuahdude 4y ago
  "Unexpected consequences" for a core piece of the technical approach suggests there are architects that are in the wrong role.
• lowbloodsugar 4y ago
  >We are now attempting to solve this with caches and doing batch requests, but all of this created additional overhead that could have all been avoided by not using microservices. > especially since some of these services are not even in the same data center. I think you need to answer why? If you can't put all of the services in one data center, then by definition you can't write a monolith either. If the monolith would happily run in one datacenter, then you should have all instances of your microservices in that one datacenter. It surprised me that you would conclude that this is a problem with microservices. It's like if a particularly architect always punches you in the groin in every meeting, and you've concluded that architects are bad people, rather than this one architect is a bad person.
• nova22033 4y ago
  especially since some of these services are not even in the same data center. Why is the other service in another data center? Does it need to be in another data center? If it does, how will a monolith help?
• 0xblinq 4y ago
  > However one of the unexpected consequences How the hell... like... who decided to do Microservices in the first place if they didn't know this? This is such a rookie mistake. It's like somebody right out of high school just read on a blog the new way is "microservices" and then went ahead with it.
  • Pamar 4y ago
    well, this is a surprisingly common case in my experience, except that the people were not "right out of high school" but "right out of a company funded seminar on microservices".
  • marcosdumay 4y ago
    There is a high amount of correlation between people that start with microservices (instead of using them to solve specific problems after they have the problem) and people that lack any awareness about the resources they need. And both are way more common than they should be.
  • dogleash 4y ago
    >How the hell... like... who decided to do Microservices in the first place if they didn't know this? Microservices can have both design utility and simultaneously been a major fad. Lurking reddit and HN you can watch development fashions come and go. It's really really profoundly hard to hold a sober conversation on splitting merits from limitations in the middle of the boom. tl;dr: gartner_hype_cycle.png
• padjo 4y ago
  I don’t mean to be snarky but how is that an “unexpected consequence”? Were the pros and cons just never considered when deciding to use micro services? Additional networks calls are one of the most obvious things to go on the cons list!
  • arkh 4y ago
    Someone must have read a blog post about microservice and decided it would look good on their resume.
    • c-fe 4y ago
      Unfortunately this is also partly correct - The team was young and people were eager to play around and learn how to work with microservices.
      • donkeyd 4y ago
        This is one of the reasons I dislike working in teams that lack 'old people'. Young devs still have a lot of mistakes to make that old devs have already made or seen. The young ones seem to see them as slow and difficult, but they also create stability and control. In a start-up, having a team of young people will allow you to move fast, pivot and deliver. What you usually end up with though, is a tower built from spaghetti that's serving actual users. I see this again and again and then people get surprised that the software doesn't scale.
        • JAlexoid 4y ago
          The most amazing kicker is that a lot of 5 YoE get senior status, while still making ridiculous mistakes.
        • selcuka 4y ago
          Agreed. This is a corollary of the Second-system effect [1]. By the time developers design their third system they have become relatively "old". [1]: https://en.wikipedia.org/wiki/Second-system_effect https://en.wikipedia.org/wiki/Second-system_effect
• goodoldneon 4y ago
  If the services need to be in separate data centers then how would a monolith be a solution? Monoliths can't span data centers
  • Ensorceled 4y ago
    I'm assuming that those services don't actually NEED to be in a separate data centre ...
• robertlagrant 4y ago
  Some of the services not being in the same datacentre seems orthoganal. If that solves a problem you have, wouldn't it still be an issue in a non-microservices design?
• LeonM 4y ago
  > one of the unexpected consequences is that we are now doing a lot of network calls between these microservices Not trying to be harsh here, but not expecting an increase of network call in a system where each component is tied together with... network calls sounds a bit naive. > We are now attempting to solve this with caches and doing batch requests So you have built a complex and bottle-necked application for the sake of scalability, then having to add caching and batching just to make it perform? That sounds like working backwards. Obviously, I have no clue on the scale of the project you are working on, but it sure sounds like you could have built it as a monolith in half of the time with orders of magnitude more performance. Scalability is a feature, you can always add it in the future.
  • ahmetnoid 4y ago
    Scalability is not a feature. If you "need" to scale but don't then you're not delivering to your target market, not delivering is not a lack of a feature it's a net loss to the organization. If you're Twitter/Instagram/Uber/whatever you cannot tell your users to not post or like or request a ride because "right now we don't have the scalability feature"
    • LeonM 4y ago
      I don't completely agree here. Yes, if you can't scale fast enough as you need to, it can hurt your business. Not being able to keep up with demand is a (luxury) problem that every business faces, not just in tech. They would often be called 'growing pains' in a business, and though they are bad, they rarely contribute to the failure of a company. Starting a startup/service/platform with microservices before you even understand the bottlenecks/market fit/customers is usually not a good idea. You can come a very, very long way with a monolith before you hit performance and scalability limits. And once you do, you can always start breaking things up into smaller services for scalabity. Obviously you need to make sure you are scaling on time to keep up with demand. 'Nail it, then scale it', and 'premature optimization is the mother of all f-ups' are popular sayings for a reason.
      • ahmetnoid 4y ago
• londons_explore 4y ago
  In which case, the next step for your org is a mandate that all microservices be available in all datacenters. Let each microservice owner figure out how to achieve their latency+reliability SLA's in every location - whether replicating datastores, caching, being stateless, or proxying requests to a master location.
• davej 4y ago
  Twitter has a similar issue according to Musk https://twitter.com/elonmusk/status/1592176202873085952 https://twitter.com/elonmusk/status/1592176202873085952
  • oblio 4y ago
    Musk last wrote code in 1995, most likely.
  • yellow_lead 4y ago
    The tweet is obviously wrong. > I was told ~1200 RPCs independently by several engineers at Twitter, which matches # of microservices. The ex-employee is wrong. > Same app in US takes ~2 secs to refresh (too long), but ~20 secs in India, due to bad batching/verbose comms. RPCs are on the server side. Why would they app take longer to refresh in India than in the US? Some more explanation:https://twitter.com/mjg59/status/1592380440346001408 https://twitter.com/mjg59/status/1592380440346001408
    • brasic 4y ago
      Yep, this is trivial to falsify, Musk jumped to an incorrect conclusion based on some anecdotes that he did not follow: https://twitter.com/Popeska/status/1592179502838435847 https://twitter.com/Popeska/status/1592179502838435847
    • davej 4y ago
      Musk may just be wrong but isn’t it also possible that the RPCs need to communicate with an edge node in India. Perhaps government regulation requires them to store/serve some user data from India if serving Indian customers? Or something like that?
    • bitL 4y ago
      They might be using cheaper caching in India and dedicate the more expensive ones to areas of the world they get profit from.
      • JAlexoid 4y ago
• wruza 4y ago
  Would be interesting to learn how your (or any other) team defines borders of a microservice. Iow, how “micro” they are and in which aspect. I guess without these details it will be hard to reason about it. At my last job we created a whole fleet of microservices instead of a single modular project/repo. Some of them required non-trivial dependencies. Some executed pretty long-running tasks or jobs for which network latency is insignificant and will remain so by design. Some were few pages long, some consisted of similar-purpose modules with shared parts factored out. But there was no or little processes like “ah, I’ll just ask M8 and it will ask M11 and it will check auth and refer to a database. I.e. no calls as trivial as foo(bar(baz())) but done all over the infra.
  • Pamar 4y ago
    Did you have a common, centralized data store or did each and every microservice manage their own instance? (Because this is at the same time one of the defining elements of this architecture... and the first one to be opted out when you actually start using it "for real").
    • wruza 4y ago
      Yes and no. The "central" part of data flowed naturally through services (i.e. passed in requests and webhooks, not in responses). Microservices maintained local state as well, though it was mostly small and disposable due to whole-intra crashonly design. For example, we didn't hesitate to shut something down or hot-fix it, except for bus-factor periods when there's only few of them. They could also go down by themselves or by upstream, and routers avoided them automatically.
• systems_glitch 4y ago
  I agree that monoliths are the way to start many times, especially if you're not actually sure you'll ever need to scale. One reason we do sometimes plan on microservices from the start with projects is separation of security concerns. Easier to lock a public-facing microservice down tight and have what's ostensibly a non public-facing monolith call out to it. Lots of ways to solve a problem, though.
  • zrail 4y ago
    Yeah, I think this is one of the very few places where splitting out something as a microservice makes sense. For example, you (mostly) never want to open/process/examine/glance at user-provided PDFs on a box with any sort of unfiltered network access. Ideally you do what you need to do within a sandbox that has _no_ network access, but that's really hard to do performantly. The primary reason for this is that PDFs can contain executable code and the common tools used to process them are full of unpatched CVEs.
• eternalban 4y ago
  Here is an analogy that can inform the implications of this (so-called, imo) architecture: Imagine if you are responsible, at runtime, for linking object files (.o) where each object is a just the compilation of a function. Now why would anyone think this is a good idea (as a general solution)? Because in software organizations, the “linker’s” job is supposed to be done by the (“unnecessary weight”) software architect. Microservices primarily serve as a patch for teams incapable of designing modular schemas. Because designing schemas is not entry level work and as we “know” s/e are “not as effective” after they pass 30 years of age. :) > monolith Unless monolith now means not-microservice, then be aware that there are a range of possible architectures between a monolith and microservices.
• strictfp 4y ago
  Yes. If you design a distributed system you need to consider the network traffic very carefully, and choose your segmentation in such a way that you minimize traffic and still achieve good scalability. For this reason, I've been trying to push for building a monolithic app first, then splitting into components, and introducing libs for common functionality. Only when this is all done, you think about the communication patterns and discuss how to scale the app. Most microservice shops I've been in have instead done the naïve thing; just come up with random functionally separate things and put them in different micro services; "voting service", "login service", "user service" etc. This can come with a very very high price. Not only in terms of network traffic, but also in debuggability, having a high amount of code duplication, and getting locked into the existing architecture, cementing the design and functionality.
  • c-fe 4y ago
    Everything you mention (network traffic cost, code duplication (we instead resorted to auto-generate some shared code between services based on a open-api spec), locked into the architecture) applies to our use case... And it is reassuring to hear that you seem to have success in avoiding these issues with a monolithic architecture, as I thought I was oldschool for starting to prefer monoliths again.
  • vaughan 4y ago
    > Only when this is all done, you think about the communication patterns and discuss how to scale the app. The main thing is that regardless of scaling, the app should always be able to run/debug/test locally in a monolithic thing. Once people scale they seem to abandon the need to debug locally at their peril. Scaling should just be a process of identifying hot function calls and when a flag is set, to execute a call as a network rpc instead.
    • mavelikara 4y ago
      The execution model of local calls are quite different from remote calls. Because of these differences, many efforts have been made, but "transparent remoting" is still not achieved [1]. [1] https://scholar.harvard.edu/waldo/publications/note-distributed-computing https://scholar.harvard.edu/waldo/publications/note-distribu...
• kyrra 4y ago
  May I ask why they are not even in the same data center?
• pjc50 4y ago
  No personal project has any business using microservices, unless it's specifically as a learning toy. Use a monolith. Monoliths can scale more than you can (provided you manage not to limit yourself to the single-thread single-process version of a monolith). Microservices are an organisational tool for when you need to release chunks independently. I first wrote a program which ran on a web server about 25 years ago. In that time, computers have experienced about ten doublings of Moore's law, i.e. are now over a thousand times faster. Computers are very fast if you let them be.
• omginternets 4y ago
  Unexpected?! Either way, one of my biggest pet peeves is the near-ubiquitous use of HTTP & JSON in microservice architectures. There's always going to be overhead in networked servies, but this is a place where binary protocols (especially ones like Cap'n Proto) really shine.
• codethief 4y ago
  > However one of the unexpected consequences is that we are now doing a lot of network calls between these microservices I wouldn't call that entirely unexpected. :-) It's a rather well-known issue: Microservices grug wonder why big brain take hardest problem, factoring system correctly, and introduce network call too seem very confusing to grug (from https://grugbrain.dev https://grugbrain.dev)