11 ms·

Domain-Oriented Microservice Architecture

130 points by amadk 6y ago

Twisol 6y ago
Depending on what angle you come at this from, you could say that DOMA groups services into clusters ("domains"), as Uber has done here, or that services always should have been domain-driven, and DOMA welcomes the networking layer inside the bounded context as well. I've spent a lot of time trying to understand when a microservices architecture makes sense, what the caveats are, and what philosophy one should take to building services. All the material I've read seems to point in the direction of services being ideally coupled to domain boundaries. It seems to me that Uber's services proliferated beyond the framing of bounded contexts, and DOMA is their attempt to reign it back in again. I think it's an excellent strategy, and arguably a very good approach for other companies who find themselves in this position. I don't think DOMA is a good place to stay at. The network should only be tolerated as long as it provides benefits that outweigh the costs. "Monolith" is not synonymous with "poor design". Seeing that these enclaves of services sitting within a domain depend on eachother in the way the OP describes, it really makes me think that they'd find further benefits by expelling the network from each domain.
- adamkl 6y ago
  > The network should only be tolerated as long as it provides benefits that outweigh the costs. I agree. There should be no need to use network calls to enforce interface boundaries if you have a cohesive bounded context. Or to be snarky about it, welcome to 2004 Uber! Eric Evans sends his regards!
  - gen220 6y ago
    Unfortunately, picking and enforcing bounded contexts is hard work in a big organization, even with strong code review processes. There will always be people who don’t want to respect where the boundaries are drawn (not in a constructive, “it could be better” kind of way, but in a “but this works, too” kind of way). If a group of such people get together, microservices compartmentalizes their capacity to drag down the ship, so to speak. I think this risk compartmentalization is a benefit that must be weighed against the costs (in terms of latency, maintenance of shared libs, opentracing, etc). These days the costs are vanishing, as tools are quite good and becoming easier to manage. All that said, if you’re a small team of senior engineers working with a shared mental model, a single binary with internally-bounded contexts works really well and I agree with you, having seen it done well.
    - etxm 6y ago
      It’s important to respect the boundaries, but also have the flexibility to change them as the business changes. I’ve seen many times where people were afraid to change boundaries because they assumed the first person got the architecture exactly right.
    - adamkl 6y ago
      I was lucky enough to be in the right place, at the right time, to lead a group that scaled this approach across half a dozen different development teams. Fortunately everyone bought into the architecture, and respected the boundaries. Not everyone was senior, and not all the code was great, but we adopted the viewpoint that so long as the bad code is in the right spot (and not talking to things it shouldn’t) everything would be ok in the end. And it was. Half a dozen teams working in one codebase was definitely pushing it though, and the need to scale much beyond that would have definitely required some service-level compartmentalization to keep the ship from sinking, as you said. Even then, it would still be a far cry from the “microservices should be small enough to re-write in 2 weeks” approach.
    - pjmlp 6y ago
      If the programming language is compiled and modules are distributed as binaries across teams, they have no means than to comply with modularity.
- blunderfel 6y ago
  I'm a big fan of the modular monolith pattern. I usually make domain modules as independent as possible and invert dependencies for web and persistence layers. If you design well you can break off domain-based services whenever the advantages warrant it.
  - gen220 6y ago
    If you ever have to start a big company from scratch, this is how to do it. There aren’t any major drawbacks to this model when a business is young (first couple years). The downsides appear when you have different parts of your application with very different load requirements. It also takes a lot of discipline to write code this way. Without strict code review and more experienced hands, the bounded contexts fall apart. One of the advantages of the microservices model is it limits the damage people can do. :) It forces a bounded context on a team of engineers and says “hey, play in this sandbox and follow these SLAs. If your internal designs are awful, good luck.”
    - fiddlerwoaroof 6y ago
      It mostly hides the damage: instead of a code base no one understands, you have network traffic no one understands.
      - ct520 6y ago
        I’ll take the latter any day with good patterns of aggregating to a grey log. Having to triage production issues in a multiple application saas environment, The latter has always been easier to me. Don’t get me started on trouble shooting someone else’s crazy event queues
        fiddlerwoaroof 6y ago
        This is really interesting to me: I’ve always found dealing with code preferable to dealing with network communication. It might be because the languages I work with (common lisp, Clojure and Scala/Java/Kotlin) all have excellent code navigation abilities.
- theptip 6y ago
  As far as I can tell from reading Evans' DDD, there's nothing forbidding network calls inside a service. For a trivial example, you make network calls from your API to your DB. And also to your Redis cache, and then if your service runs async/periodic tasks, to your task queue, say your RMQ & Celery instances. So to me the OP reads like "we're coming up with some new terminology for a bounded context, and also defining how those contexts should be allowed to layer in order to simplify/control failure modes". The layering stuff is more interesting than Uber's rediscovering bounded contexts, though it's definitely interesting that they have come into agreement with Evans (and the rest of the DDD community) on the "Service == Bounded Context" principle.
  - Twisol 6y ago
    > As far as I can tell from reading Evans' DDD, there's nothing forbidding network calls inside a service. You're quite right. > [...] defining how those contexts should be allowed to layer [...] This, however, seems to go against the spirit of things. There is a consistent "ubiquitous language" within a bounded context, where domain terms are concrete and unambiguous. (Or rather, the context disambiguates the language.) The concept of "layered contexts" seems to neuter the concept. Does each layer successively disambiguate the one above? Or does it add new terms that didn't exist? The layering here sounds much more technically-motivated than domain-oriented. And my argument is that the networking internal to their "domains" is largely an artifact of having build DOMA out of a plethora of disorganized microservices. Doubtless there will be some necessary networking remaining, as you remarked on, such as between processors and databases. But the origin here suggests most of it is left over from what came before.
    - theptip 6y ago
      Playing devil's advocate for a second, I'm wondering if, at Uber's scale (thousands of microservices, perhaps that means hundreds of bounded contexts after applying "DORA"?), the observations that make hexagonal/layered architectures a useful design within a single service become relevant at the system level. I'm not sure how many systems have been built using DDD with hundreds of interacting bounded contexts, but I suppose I could believe that _some_ structure would be beneficial. (If you know of any case studies here I'd love to hear of them, I've not actually seen anything published on this topic.) In general the concept of an "infrastructure bounded context" seemed a bit weird to me from my understanding of DDD, but then I though about Kubernetes, and you could make a case that it is an example of such a bounded context; it has its own ubiquitous language, etc. It would be weird for your infrastructure to have any understanding of the domain objects running on top of it, so a hierarchy makes sense. Likewise if you have BFFs for your different API clients; the domain services underneath them could be abstracted away from things like REST, if all your internal services use gRPC (for example). You could consider this the UI layer in DDD's layered architecture. I'm struggling to come up with more sensical layers than that though; in DDD there's the Application and Domain layers; I don't really see how you'd pull "Application" vs. "Domain" bounded context layers together in a way that made sense. > But the origin here suggests most of it is left over from what came before. I'd certainly agree with this -- it seems like lots of the intra-BC complexity is excessive compared to what you'd get if you built your services with a BC in mind from the start. I don't think I'd emulate their intra-BC structure, it's only the inter-BC organization that I think has any merit for other systems (and even then I'm not fully convinced yet).
recursivedoubts 6y ago
Seems to me that big progress will be made when we can decouple system boundaries from network calls.
- bob1029 6y ago
  I still think the approach of baking the entire enterprise into 1 gigantic binary is the best. Stack traces are so much easier to work with compared to distributed logs and side effects. We are already at a point where you could hypothetically do this for a reasonably-sized organization. A 64 core CPU can support a huge number of clients. Stuff like .NET Core scales really well if you want to build something complex like this. One big binary that occupies an entire physical host is an extremely compelling development model. Literally everything becomes a direct method invocation. You can also have type enforcement and atomic releases for the entire enterprise. Also makes a monorepo an obvious choice for source code management.
  - jzoch 6y ago
    This falls apart when you care a lot about availability and therefore want a particular service to be stateless. Then you add some feature that needs a tiny bit of state and suddenly you can't scale up and down as easily and need to start paying attention. This rolls forward into a big ball of mud - a monolith can be tricky too at scale
    - bob1029 6y ago
      One trick is to leverage the fact that you have 1 type domain, and to develop a common persistence abstraction for the entire enterprise. Then, all of your business entities can be pushed through it for a consolidated replication flow. This can be [a]synchronously replicated to additional node(s) as required, potentially with replication rulesets defined in application code as well. It sounds like a complex monster until you build it one time. Then you are basically done. The leverage you get when you have 1 way to do everything is extremely powerful. I do recognize there are scenarios where you cant force one persistence abstraction on all use cases, but there's no reason you couldn't have a TimeSeriesEntity (keyed by time) in addition to a typical BusinessEntity (keyed by a unique integer). Both could have unique replication implementations, but it would still be standardized and all nodes would be speaking the same protocol because they all derive from the same source. The monolith only remains a monolith in operational terms until the engineering team develops some imagination.
sdan 6y ago
Basically clumping microservices
- deltaeerie 6y ago
  I believe this is only the first point. It's also establishing a hierarchy of these "clumps" (layers), strict APIs representing a single clump for other clumps to consume (gateways), and a pattern for clumps to extend functionality of other clumps without polluting their models (extension architecture).
annaaix 6y ago
hi guys. my hot photos are here... https://mub.me/xxxxi https://mub.me/xxxxi my nickname anna1
dkersten 6y ago
Somewhat off topic but maybe someone can enlighten me: > Uber has grown to around 2,200 critical microservices Even thinking about the very largest systems I’ve worked on, I can’t think of what could possibly be split into 2000 separate individual services. What are these thousands of microservices and how micro are they?
- nevir 6y ago
  I'd be curious how that compares to Amazon's count of services (my guess is 10k+)
  - beastcoast 6y ago
    I’d say we’re closer to an order of magnitude larger When you consider all the different businesses and domains. My org alone has about 1000 services. But we have massive scale and the employees to support that number of services. You also won’t find one architecture; it runs the gamut from lambda to REST/RPC to pubsub, event bus and workflow architectures. Typically new orgs start out with a few larger services that get broken out over time, as the complexity and number of employees continues to grow.
- drol3 6y ago
  I am working for a company that has moved from 3 to 15 backend developers and 25 to 50 services over the last year. The amount of services is really quite meaninglessness compared to the amount of developers. As with anything the efficiency good down as the amount of people goes up. There is no way that 2000 people can agree on anything. So most likely there is lots and lots of overlap. You could probably refactor some of this duplication out, but by the time you would be done new duplication would have emerged. Keeping many people in sync is just difficult :) Anyway the point is that 2200 services is meaningless without telling how many employees are working on said services :)
- eikenberry 6y ago
  Remember that those aren't 2200 separate things going on, but 2200 instances of the services. I'm sure that includes very large clusters running many copies of the same service.
xcskier56 6y ago
While I work on much smaller codebases and so don’t have any particular insight into how DOMA will work for such large organizations, the piece that I really liked here is how the separate their different general layers of services. The key to any service being usable outside of the exact context it was first written in is to ensure no product specific business logic is added to it. By splitting out infrastructure from general business from product specific services, you can do a much better job of understanding, and therefore controlling where where product specific logic is allowed. This in turn will make your lower level services far less coupled to the exact context they are first used in. For smaller orgs, this is by far the more useful information, rather than how to deal with 2200 microservices.
nfoz 6y ago
I don't like that they define microservices as components communicating via RPC, because this precludes other models like pub-sub.
harel 6y ago
I'm not entirely sure why this is sold as innovative new approach. Over the last few years, I've seen and developed micro services that are indeed cut down to specific functionality, and I've done ones where the domain was the scope. The latter has proven to produce bigger services, but a more maintainable ecosystem. Also, the division for micro services should also be dictated by the context of the business, and not just the current whims of the devs.
kopos 6y ago
Feels like a lot of old knowledge was discounted / ignored and the mistakes are being rediscovered. Feels a bit like the Hexagonal Architecture being rediscovered. Micro service with half life of 1.5 years? Does that mean that enough planning is not being done? Or leadership failure at software planning level? Collaboration between teams at scale is very hard but that is what the leadership layer is for - to collaborate more not build more micro services.
- sixdimensional 6y ago
  Or how about plain old onion architecture... even older than hexagonal. This is literally scaled/distributed domain-driven design (DDD). I have felt strongly that folks got so caught up in the hype of yet another new thing they forgot how to extend what came before. It feels like a reinvention of existing ideas at larger scale, and a pattern we keep repeating. I'm not complaining, of course - new things are possible and being learned through this innovation. I do feel we should be more careful on the cutting edge, to see how it relates to where we came from. Put another way, reinventing the wheel is not pointless if you come out with a better thing, or better wheel. But don't forget what was good about the previous wheel before you throw it out?
  - staticassertion 6y ago
    I'm confused. You're saying that people are reinventing existing ideas at larger scale, but you also are saying that people don't know how to extend what came before? Those sound like the same exact thing.
    - sixdimensional 6y ago
      You're not confused, but there is a nuance. I'm trying to walk a fine line of not criticizing them for discovering this late into their process (maybe they knew all along but were busy inventing), but also questioning why they couldn't see this sooner. Trying to get to the heart of what took this innovation/recognition/learning to happen. I am making an observation - when each new "fad" or "hype cycle" tech starts, it seems as though the pattern knowledge of what came before is discarded, or, disregarded as "legacy" or possibly even just forgotten. It feels like a knowledge transfer is missing. It would be terrible if, we, as an industry aren't passing down knowledge and reinventing hard won pattern discoveries efficiently. Did this pattern Uber discovered come from studying onion architecture, DDD, etc first, finding the limitations, and then scaling them? Or did this arise from throwing away everything that came before (or not knowing about it), forging an undiscovered path, and then rediscovering the old patterns could be applied? If the latter, what can be learned to make this process of discovery and linkage to existing patterns more efficient? I think this article shows innovation is tricky, or, the risk (and potential reward) at the bleeding edge. Leaving behind design constraints of what came before might be necessary. Maybe I'm trying to say, as an industry we need to balance exploitation of previous knowledge and our attitudes about how we feel about "legacy", with the unquenchable thirst for the next new innovation?
levischoen 6y ago
Yup. So micro services of micro services. Clearly defined interfaces. Single entry point for other services/domains to interact with. seperation of concerns. The fundamental law will always be true : https://en.m.wikipedia.org/wiki/Fundamental_theorem_of_software_engineering https://en.m.wikipedia.org/wiki/Fundamental_theorem_of_softw...
etxm 6y ago
I’m honestly not trying to be snarky here but I didn’t know there was any other way to design microservices (with intention) that weren’t domain oriented.
- timclark 6y ago
  I’ve seen one service per database table, and have had architects seriously think that a table equals a bounded context. Then they try to work out how to do two-phase commit across micro services.
  - etxm 6y ago
    Oh that. I’ve had to merge services like that. People are always grabbing nouns and thinking they’re microservices.
- fma 6y ago
  I'm on the same boat. "Previously product teams would have to call numerous downstream services to leverage a domain; they now have to call just one....Furthermore, we were able to classify 2200 microservices into 70 domains." So they went from 2200 to 70 microservices? One extreme to the other. The answer is somewhere in the middle.
vmurthy 6y ago
I recently read Domain Modelling made Functional [0] and one of the under-recognised benefits of using Domain Driven or oriented designs is that (if done right), business workflows and processes become well explained for developers and minimises surprises from business closer to release dates. For e.g., an order processing system will have lesser assumptions by developers and tech leads because someone with enough knowledge of the business process from order processing will be able to guide you. [0]https://pragprog.com/titles/swdddf/ https://pragprog.com/titles/swdddf/
sagichmal 6y ago
> In other words, organizations adopt microservices for an operational benefit at the expense of performance. ...no, not at all. There are a couple of operational benefits, but vastly more drawbacks, and on balance microservices are phenomenally harder to operate than monoliths. Organizations adopt microservices when the logistical overhead of coordinating teams against a monolith becomes so large that it starts affecting product velocity. Microservices are a way to release that organizational/logistical friction, at great technical cost. With that said, the domain-oriented bounded context is indeed the right way to think about service delineation.
- rumanator 6y ago
  > Organizations adopt microservices when the logistical overhead of coordinating teams against a monolith becomes so large that it starts affecting product velocity. That's certainly one of the operational limits, but arguably not the most important one. You distribute your system so that parts of it may scale independently, for example. Traffic fluctuates along time and the only option you have to scale your system is adding nodes as you go.
  - sagichmal 6y ago
    The ability to scale services independently is not as impactful as many people believe, and changing your entire architecture for no other reason than to get that specific benefit is a mistake in the vast majority of cases. Microservices are a solution to organizational problems, not technical problems. On balance they create far more technical issues than they solve.
- scarface74 6y ago
  The last company I worked for was very much an “API first” company. We were a B2B company and our APIs were used by our relatively lightly used website. But they were also used for sporadic and large ETL jobs when files from customers came in and most importantly we sold access to our APIs that were used for our customer’s mobile apps and heavily trafficked website. We went microservices for the canonical reason you should - to be able to release and scale independently.
systematical 6y ago
I'm looking forward to the article about the problems DOMA introduced for them and what they came up with next. Wonder if we will see a full circle back to monoliths. I know I'm being snarky and I have used micro-services myself, but only when it was smacking me in the face as the best tool for the job. Is the fools-gold rush still on to do everything as a micro-service from the get-go? Side note, I can't wait to hear my boss use "DOMA" in a meeting in the coming months. FML.
- rumanator 6y ago
  > Wonder if we will see a full circle back to monoliths. For global-scale web applications? Obviously you won't. High-availability, low latency, resilience, scalability, performance. You don't get any of that by running your app on a single box. That ship has sailed two or three decades ago. Physics establishes all the limits, not software architects. Distributed system critics, where they fixate on trendy microservices architectures or poopoo other suggestions like DOMA, should take a step back and look at themselves and what they are actually complaining about. Yes, a solution with no moving parts is simpler than a solution with some moving parts. But have you really noticed what problems are being solved by adding these pieces?
  - inopinatus 6y ago
    Monoliths don't run on a single box. My entire current business runs on a single monolithic application, and it's running across multiple AZs with five different instance roles (consumer site, API endpoint, admin site, background jobs, and reporting), sized by memory demand and contention efficiency, then scaled horizontally according to workload demand. They are all, however, running exactly the same code, just in different configuration. I'd say it's roughly speaking 80% core libraries and the remaining 20% varies by role. I have a command-line control & observation tool that comes in bin/ of the same repository, and it is again wrapped around the same code besides. That's the modern monolith in production.
    - rumanator 6y ago
      It seems you should revise your concept of monolith because your description is anything but it. I mean, you yourself talk about "different instance roles". You should pay attention to your own claims: if you have a distributed deployment comprised of different nodes, and you have specialized nodes that you yourself state that are ran to handle limited and very specific responsibilities, then just because you decided, for any reason that only you can think about, to bundle everything in a single project... That doesn't make it a monolith, does it? Just to be absolutely clear, "monolith" is not a reference to how you chose to organize your source tree. Monolith is a software architecture concept that defines how your whole application is organized and deployed. A distributed system comprised of multiple specialized processeses running independently is not a monolith, even if you somehow believed it was a good idea to pick which role you run through configuration.
sroussey 6y ago
Uber thinks they only have that many services? Yikes, double, or triple the number... several years ago. Maybe they have fewer now?
sgt-serverless 6y ago
The last two systems I designed and built where microservices based. One manages 2 million transactions per day, the other is processing millions of dollars worth of transactions a month. While it might be slightly harder to grep microservices, if you do them right, I feel the benefits far outweigh the drawbacks of going monolith. I think the hardest part is trusting that your system can not be digested in just one bottlenecked app standing up. This really goes deeper than microservices as well, into event-driven programming. Too much time seems to be spent on people saying, well my monolith is just fine. It might be, but doing a microservice system works, you just have to be able to ride the wave.
deleted 6y ago
[deleted]
ianamartin 6y ago
I despise Uber as a company, but I've always loved them as a technology group ever since I used to go to their meetups in NYC. You can always count on Uber to throw a great party and also come up with the worst possible technology. If you want to know how not to do things, Uber is a very good place to look.
- hinkley 6y ago
  And all the people who worked there will consider the fact that they did 1) something and 2) were successful means that that 'something' had anything to do with it, and therefore we should keep doing it. Not that this is unique to Uber, mind. I just described half of my coworkers too.
exnuber 6y ago
They say they draw inspiration from DDD and CQRS, yet seem to miss one crucial factor. And no, I'm not talking about bounded contexts. They still focus in re-use of commodity rather than addressing why their microservice approach didn't work. Their strategy for defining how functionality moves down the stack is not bad, but it's not anchored towards the right thing. It's the consistency boundary (or aggregate root if you will). It's the circle you draw around your business rules and say "this group needs to remain consistently enforced". This defines your unit of scale, not re-use. So while they are headed in the right direction, I fear that they are still missing a fundamental piece of information to steer on. I was employed by Uber (I quit after a couple months), and the idea's that they now hint towards were largely rejected by the engineers, and that was less than a year ago. Uber is just in the position to throw a large sum of money at making wrong decisions and getting away with it, because it's not their money. It's VC funny money. DISCLAIMER: Yes, I did indeed create this account to be able to reply anonymously.
pgt 6y ago
Microservices are about scaling teams, not tech.
ExcavateGrandMa 6y ago
STOP WASTING UR TIME AND START HIRING REAL PROs based on their real SKILLZzzzZzzz! I would go this way... (am technical not commercial...) U keep your stuff in production while I study your environment & take notes... And then changes propositions happens! (I may offer to rewrite full or part of a stack for the purpose... and also technology change for independence...) Then conception, testing, validation... PRODUCTION in the long term without headaches and clear understanding! o/
keyraycheck 6y ago
> as Uber grew from 10s to 100s of engineers with multiple teams owning pieces of the tech stack, the monolithic architecture tied the fate of teams together and made it difficult to operate independently. I think this is key sentence. Microservices does not make a lot of sense for teams of 10s, while are a great tool for team of 100s.
kerng 6y ago
Very interesting and seems like a logical next step with things get a bit too messy with micro services. One thing that was not touched on - how is security done in DOMA? * Is auth happening at domain gateways or at each service? * Similarly for encryption, does it terminate at Domain GW or at each service?
- alpmeow 6y ago
  Hopefully both
SergeAx 6y ago
I felt bad for Uber engineering force while reading whole article. They swamped themselves into that microservices hell due to pressure for teams velocity and TTM, and created enormous tech debt in the process. Now they are trying to patch it up with ridiculous means like putting JSON inside of protobuf messages. I really hope this article could be a warning for management of growing startups for what could happen if you squeeze features out of your team, but managers usually speed-reading those articles just to order "do like Uber did".
stevenalowe 6y ago
Nice to see Uber rediscovering domain-driven design principles.
mbrodersen 6y ago
If you can't make a monolith work, then you will fail even harder trying to make a Microservice Architecture work.

130 points by amadk 6y ago

Depending on what angle you come at this from, you could say that DOMA groups services into clusters ("domains"), as Uber has done here, or that services always should have been domain-driven, and DOMA welcomes the networking layer inside the bounded context as well. I've spent a lot of time trying to understand when a microservices architecture makes sense, what the caveats are, and what philosophy one should take to building services. All the material I've read seems to point in the direction of services being ideally coupled to domain boundaries. It seems to me that Uber's services proliferated beyond the framing of bounded contexts, and DOMA is their attempt to reign it back in again. I think it's an excellent strategy, and arguably a very good approach for other companies who find themselves in this position. I don't think DOMA is a good place to stay at. The network should only be tolerated as long as it provides benefits that outweigh the costs. "Monolith" is not synonymous with "poor design". Seeing that these enclaves of services sitting within a domain depend on eachother in the way the OP describes, it really makes me think that they'd find further benefits by expelling the network from each domain.

adamkl 6y ago
> The network should only be tolerated as long as it provides benefits that outweigh the costs. I agree. There should be no need to use network calls to enforce interface boundaries if you have a cohesive bounded context. Or to be snarky about it, welcome to 2004 Uber! Eric Evans sends his regards!
- gen220 6y ago
  Unfortunately, picking and enforcing bounded contexts is hard work in a big organization, even with strong code review processes. There will always be people who don’t want to respect where the boundaries are drawn (not in a constructive, “it could be better” kind of way, but in a “but this works, too” kind of way). If a group of such people get together, microservices compartmentalizes their capacity to drag down the ship, so to speak. I think this risk compartmentalization is a benefit that must be weighed against the costs (in terms of latency, maintenance of shared libs, opentracing, etc). These days the costs are vanishing, as tools are quite good and becoming easier to manage. All that said, if you’re a small team of senior engineers working with a shared mental model, a single binary with internally-bounded contexts works really well and I agree with you, having seen it done well.
  - etxm 6y ago
    It’s important to respect the boundaries, but also have the flexibility to change them as the business changes. I’ve seen many times where people were afraid to change boundaries because they assumed the first person got the architecture exactly right.
  - adamkl 6y ago
    I was lucky enough to be in the right place, at the right time, to lead a group that scaled this approach across half a dozen different development teams. Fortunately everyone bought into the architecture, and respected the boundaries. Not everyone was senior, and not all the code was great, but we adopted the viewpoint that so long as the bad code is in the right spot (and not talking to things it shouldn’t) everything would be ok in the end. And it was. Half a dozen teams working in one codebase was definitely pushing it though, and the need to scale much beyond that would have definitely required some service-level compartmentalization to keep the ship from sinking, as you said. Even then, it would still be a far cry from the “microservices should be small enough to re-write in 2 weeks” approach.
  - pjmlp 6y ago
    If the programming language is compiled and modules are distributed as binaries across teams, they have no means than to comply with modularity.
blunderfel 6y ago
I'm a big fan of the modular monolith pattern. I usually make domain modules as independent as possible and invert dependencies for web and persistence layers. If you design well you can break off domain-based services whenever the advantages warrant it.
- gen220 6y ago
  If you ever have to start a big company from scratch, this is how to do it. There aren’t any major drawbacks to this model when a business is young (first couple years). The downsides appear when you have different parts of your application with very different load requirements. It also takes a lot of discipline to write code this way. Without strict code review and more experienced hands, the bounded contexts fall apart. One of the advantages of the microservices model is it limits the damage people can do. :) It forces a bounded context on a team of engineers and says “hey, play in this sandbox and follow these SLAs. If your internal designs are awful, good luck.”
  - fiddlerwoaroof 6y ago
    It mostly hides the damage: instead of a code base no one understands, you have network traffic no one understands.
    - ct520 6y ago
      I’ll take the latter any day with good patterns of aggregating to a grey log. Having to triage production issues in a multiple application saas environment, The latter has always been easier to me. Don’t get me started on trouble shooting someone else’s crazy event queues
      - fiddlerwoaroof 6y ago
        This is really interesting to me: I’ve always found dealing with code preferable to dealing with network communication. It might be because the languages I work with (common lisp, Clojure and Scala/Java/Kotlin) all have excellent code navigation abilities.
theptip 6y ago
As far as I can tell from reading Evans' DDD, there's nothing forbidding network calls inside a service. For a trivial example, you make network calls from your API to your DB. And also to your Redis cache, and then if your service runs async/periodic tasks, to your task queue, say your RMQ & Celery instances. So to me the OP reads like "we're coming up with some new terminology for a bounded context, and also defining how those contexts should be allowed to layer in order to simplify/control failure modes". The layering stuff is more interesting than Uber's rediscovering bounded contexts, though it's definitely interesting that they have come into agreement with Evans (and the rest of the DDD community) on the "Service == Bounded Context" principle.
- Twisol 6y ago
  > As far as I can tell from reading Evans' DDD, there's nothing forbidding network calls inside a service. You're quite right. > [...] defining how those contexts should be allowed to layer [...] This, however, seems to go against the spirit of things. There is a consistent "ubiquitous language" within a bounded context, where domain terms are concrete and unambiguous. (Or rather, the context disambiguates the language.) The concept of "layered contexts" seems to neuter the concept. Does each layer successively disambiguate the one above? Or does it add new terms that didn't exist? The layering here sounds much more technically-motivated than domain-oriented. And my argument is that the networking internal to their "domains" is largely an artifact of having build DOMA out of a plethora of disorganized microservices. Doubtless there will be some necessary networking remaining, as you remarked on, such as between processors and databases. But the origin here suggests most of it is left over from what came before.
  - theptip 6y ago
    Playing devil's advocate for a second, I'm wondering if, at Uber's scale (thousands of microservices, perhaps that means hundreds of bounded contexts after applying "DORA"?), the observations that make hexagonal/layered architectures a useful design within a single service become relevant at the system level. I'm not sure how many systems have been built using DDD with hundreds of interacting bounded contexts, but I suppose I could believe that _some_ structure would be beneficial. (If you know of any case studies here I'd love to hear of them, I've not actually seen anything published on this topic.) In general the concept of an "infrastructure bounded context" seemed a bit weird to me from my understanding of DDD, but then I though about Kubernetes, and you could make a case that it is an example of such a bounded context; it has its own ubiquitous language, etc. It would be weird for your infrastructure to have any understanding of the domain objects running on top of it, so a hierarchy makes sense. Likewise if you have BFFs for your different API clients; the domain services underneath them could be abstracted away from things like REST, if all your internal services use gRPC (for example). You could consider this the UI layer in DDD's layered architecture. I'm struggling to come up with more sensical layers than that though; in DDD there's the Application and Domain layers; I don't really see how you'd pull "Application" vs. "Domain" bounded context layers together in a way that made sense. > But the origin here suggests most of it is left over from what came before. I'd certainly agree with this -- it seems like lots of the intra-BC complexity is excessive compared to what you'd get if you built your services with a BC in mind from the start. I don't think I'd emulate their intra-BC structure, it's only the inter-BC organization that I think has any merit for other systems (and even then I'm not fully convinced yet).

recursivedoubts 6y ago

Seems to me that big progress will be made when we can decouple system boundaries from network calls.

bob1029 6y ago
I still think the approach of baking the entire enterprise into 1 gigantic binary is the best. Stack traces are so much easier to work with compared to distributed logs and side effects. We are already at a point where you could hypothetically do this for a reasonably-sized organization. A 64 core CPU can support a huge number of clients. Stuff like .NET Core scales really well if you want to build something complex like this. One big binary that occupies an entire physical host is an extremely compelling development model. Literally everything becomes a direct method invocation. You can also have type enforcement and atomic releases for the entire enterprise. Also makes a monorepo an obvious choice for source code management.
- jzoch 6y ago
  This falls apart when you care a lot about availability and therefore want a particular service to be stateless. Then you add some feature that needs a tiny bit of state and suddenly you can't scale up and down as easily and need to start paying attention. This rolls forward into a big ball of mud - a monolith can be tricky too at scale
  - bob1029 6y ago
    One trick is to leverage the fact that you have 1 type domain, and to develop a common persistence abstraction for the entire enterprise. Then, all of your business entities can be pushed through it for a consolidated replication flow. This can be [a]synchronously replicated to additional node(s) as required, potentially with replication rulesets defined in application code as well. It sounds like a complex monster until you build it one time. Then you are basically done. The leverage you get when you have 1 way to do everything is extremely powerful. I do recognize there are scenarios where you cant force one persistence abstraction on all use cases, but there's no reason you couldn't have a TimeSeriesEntity (keyed by time) in addition to a typical BusinessEntity (keyed by a unique integer). Both could have unique replication implementations, but it would still be standardized and all nodes would be speaking the same protocol because they all derive from the same source. The monolith only remains a monolith in operational terms until the engineering team develops some imagination.

sdan 6y ago

Basically clumping microservices

deltaeerie 6y ago
I believe this is only the first point. It's also establishing a hierarchy of these "clumps" (layers), strict APIs representing a single clump for other clumps to consume (gateways), and a pattern for clumps to extend functionality of other clumps without polluting their models (extension architecture).

annaaix 6y ago

hi guys. my hot photos are here... https://mub.me/xxxxi https://mub.me/xxxxi my nickname anna1

dkersten 6y ago

Somewhat off topic but maybe someone can enlighten me: > Uber has grown to around 2,200 critical microservices Even thinking about the very largest systems I’ve worked on, I can’t think of what could possibly be split into 2000 separate individual services. What are these thousands of microservices and how micro are they?

nevir 6y ago
I'd be curious how that compares to Amazon's count of services (my guess is 10k+)
- beastcoast 6y ago
  I’d say we’re closer to an order of magnitude larger When you consider all the different businesses and domains. My org alone has about 1000 services. But we have massive scale and the employees to support that number of services. You also won’t find one architecture; it runs the gamut from lambda to REST/RPC to pubsub, event bus and workflow architectures. Typically new orgs start out with a few larger services that get broken out over time, as the complexity and number of employees continues to grow.
drol3 6y ago
I am working for a company that has moved from 3 to 15 backend developers and 25 to 50 services over the last year. The amount of services is really quite meaninglessness compared to the amount of developers. As with anything the efficiency good down as the amount of people goes up. There is no way that 2000 people can agree on anything. So most likely there is lots and lots of overlap. You could probably refactor some of this duplication out, but by the time you would be done new duplication would have emerged. Keeping many people in sync is just difficult :) Anyway the point is that 2200 services is meaningless without telling how many employees are working on said services :)
eikenberry 6y ago
Remember that those aren't 2200 separate things going on, but 2200 instances of the services. I'm sure that includes very large clusters running many copies of the same service.

xcskier56 6y ago

While I work on much smaller codebases and so don’t have any particular insight into how DOMA will work for such large organizations, the piece that I really liked here is how the separate their different general layers of services. The key to any service being usable outside of the exact context it was first written in is to ensure no product specific business logic is added to it. By splitting out infrastructure from general business from product specific services, you can do a much better job of understanding, and therefore controlling where where product specific logic is allowed. This in turn will make your lower level services far less coupled to the exact context they are first used in. For smaller orgs, this is by far the more useful information, rather than how to deal with 2200 microservices.

nfoz 6y ago

I don't like that they define microservices as components communicating via RPC, because this precludes other models like pub-sub.

harel 6y ago

I'm not entirely sure why this is sold as innovative new approach. Over the last few years, I've seen and developed micro services that are indeed cut down to specific functionality, and I've done ones where the domain was the scope. The latter has proven to produce bigger services, but a more maintainable ecosystem. Also, the division for micro services should also be dictated by the context of the business, and not just the current whims of the devs.

kopos 6y ago

Feels like a lot of old knowledge was discounted / ignored and the mistakes are being rediscovered. Feels a bit like the Hexagonal Architecture being rediscovered. Micro service with half life of 1.5 years? Does that mean that enough planning is not being done? Or leadership failure at software planning level? Collaboration between teams at scale is very hard but that is what the leadership layer is for - to collaborate more not build more micro services.

sixdimensional 6y ago
Or how about plain old onion architecture... even older than hexagonal. This is literally scaled/distributed domain-driven design (DDD). I have felt strongly that folks got so caught up in the hype of yet another new thing they forgot how to extend what came before. It feels like a reinvention of existing ideas at larger scale, and a pattern we keep repeating. I'm not complaining, of course - new things are possible and being learned through this innovation. I do feel we should be more careful on the cutting edge, to see how it relates to where we came from. Put another way, reinventing the wheel is not pointless if you come out with a better thing, or better wheel. But don't forget what was good about the previous wheel before you throw it out?
- staticassertion 6y ago
  I'm confused. You're saying that people are reinventing existing ideas at larger scale, but you also are saying that people don't know how to extend what came before? Those sound like the same exact thing.
  - sixdimensional 6y ago
    You're not confused, but there is a nuance. I'm trying to walk a fine line of not criticizing them for discovering this late into their process (maybe they knew all along but were busy inventing), but also questioning why they couldn't see this sooner. Trying to get to the heart of what took this innovation/recognition/learning to happen. I am making an observation - when each new "fad" or "hype cycle" tech starts, it seems as though the pattern knowledge of what came before is discarded, or, disregarded as "legacy" or possibly even just forgotten. It feels like a knowledge transfer is missing. It would be terrible if, we, as an industry aren't passing down knowledge and reinventing hard won pattern discoveries efficiently. Did this pattern Uber discovered come from studying onion architecture, DDD, etc first, finding the limitations, and then scaling them? Or did this arise from throwing away everything that came before (or not knowing about it), forging an undiscovered path, and then rediscovering the old patterns could be applied? If the latter, what can be learned to make this process of discovery and linkage to existing patterns more efficient? I think this article shows innovation is tricky, or, the risk (and potential reward) at the bleeding edge. Leaving behind design constraints of what came before might be necessary. Maybe I'm trying to say, as an industry we need to balance exploitation of previous knowledge and our attitudes about how we feel about "legacy", with the unquenchable thirst for the next new innovation?

levischoen 6y ago

Yup. So micro services of micro services. Clearly defined interfaces. Single entry point for other services/domains to interact with. seperation of concerns. The fundamental law will always be true : https://en.m.wikipedia.org/wiki/Fundamental_theorem_of_software_engineering https://en.m.wikipedia.org/wiki/Fundamental_theorem_of_softw...

etxm 6y ago

I’m honestly not trying to be snarky here but I didn’t know there was any other way to design microservices (with intention) that weren’t domain oriented.

timclark 6y ago
I’ve seen one service per database table, and have had architects seriously think that a table equals a bounded context. Then they try to work out how to do two-phase commit across micro services.
- etxm 6y ago
  Oh that. I’ve had to merge services like that. People are always grabbing nouns and thinking they’re microservices.
fma 6y ago
I'm on the same boat. "Previously product teams would have to call numerous downstream services to leverage a domain; they now have to call just one....Furthermore, we were able to classify 2200 microservices into 70 domains." So they went from 2200 to 70 microservices? One extreme to the other. The answer is somewhere in the middle.

vmurthy 6y ago

I recently read Domain Modelling made Functional [0] and one of the under-recognised benefits of using Domain Driven or oriented designs is that (if done right), business workflows and processes become well explained for developers and minimises surprises from business closer to release dates. For e.g., an order processing system will have lesser assumptions by developers and tech leads because someone with enough knowledge of the business process from order processing will be able to guide you. [0]https://pragprog.com/titles/swdddf/ https://pragprog.com/titles/swdddf/

sagichmal 6y ago

> In other words, organizations adopt microservices for an operational benefit at the expense of performance. ...no, not at all. There are a couple of operational benefits, but vastly more drawbacks, and on balance microservices are phenomenally harder to operate than monoliths. Organizations adopt microservices when the logistical overhead of coordinating teams against a monolith becomes so large that it starts affecting product velocity. Microservices are a way to release that organizational/logistical friction, at great technical cost. With that said, the domain-oriented bounded context is indeed the right way to think about service delineation.

rumanator 6y ago
> Organizations adopt microservices when the logistical overhead of coordinating teams against a monolith becomes so large that it starts affecting product velocity. That's certainly one of the operational limits, but arguably not the most important one. You distribute your system so that parts of it may scale independently, for example. Traffic fluctuates along time and the only option you have to scale your system is adding nodes as you go.
- sagichmal 6y ago
  The ability to scale services independently is not as impactful as many people believe, and changing your entire architecture for no other reason than to get that specific benefit is a mistake in the vast majority of cases. Microservices are a solution to organizational problems, not technical problems. On balance they create far more technical issues than they solve.
scarface74 6y ago
The last company I worked for was very much an “API first” company. We were a B2B company and our APIs were used by our relatively lightly used website. But they were also used for sporadic and large ETL jobs when files from customers came in and most importantly we sold access to our APIs that were used for our customer’s mobile apps and heavily trafficked website. We went microservices for the canonical reason you should - to be able to release and scale independently.

systematical 6y ago

I'm looking forward to the article about the problems DOMA introduced for them and what they came up with next. Wonder if we will see a full circle back to monoliths. I know I'm being snarky and I have used micro-services myself, but only when it was smacking me in the face as the best tool for the job. Is the fools-gold rush still on to do everything as a micro-service from the get-go? Side note, I can't wait to hear my boss use "DOMA" in a meeting in the coming months. FML.

rumanator 6y ago
> Wonder if we will see a full circle back to monoliths. For global-scale web applications? Obviously you won't. High-availability, low latency, resilience, scalability, performance. You don't get any of that by running your app on a single box. That ship has sailed two or three decades ago. Physics establishes all the limits, not software architects. Distributed system critics, where they fixate on trendy microservices architectures or poopoo other suggestions like DOMA, should take a step back and look at themselves and what they are actually complaining about. Yes, a solution with no moving parts is simpler than a solution with some moving parts. But have you really noticed what problems are being solved by adding these pieces?
- inopinatus 6y ago
  Monoliths don't run on a single box. My entire current business runs on a single monolithic application, and it's running across multiple AZs with five different instance roles (consumer site, API endpoint, admin site, background jobs, and reporting), sized by memory demand and contention efficiency, then scaled horizontally according to workload demand. They are all, however, running exactly the same code, just in different configuration. I'd say it's roughly speaking 80% core libraries and the remaining 20% varies by role. I have a command-line control & observation tool that comes in bin/ of the same repository, and it is again wrapped around the same code besides. That's the modern monolith in production.
  - rumanator 6y ago
    It seems you should revise your concept of monolith because your description is anything but it. I mean, you yourself talk about "different instance roles". You should pay attention to your own claims: if you have a distributed deployment comprised of different nodes, and you have specialized nodes that you yourself state that are ran to handle limited and very specific responsibilities, then just because you decided, for any reason that only you can think about, to bundle everything in a single project... That doesn't make it a monolith, does it? Just to be absolutely clear, "monolith" is not a reference to how you chose to organize your source tree. Monolith is a software architecture concept that defines how your whole application is organized and deployed. A distributed system comprised of multiple specialized processeses running independently is not a monolith, even if you somehow believed it was a good idea to pick which role you run through configuration.

sroussey 6y ago

Uber thinks they only have that many services? Yikes, double, or triple the number... several years ago. Maybe they have fewer now?

sgt-serverless 6y ago

The last two systems I designed and built where microservices based. One manages 2 million transactions per day, the other is processing millions of dollars worth of transactions a month. While it might be slightly harder to grep microservices, if you do them right, I feel the benefits far outweigh the drawbacks of going monolith. I think the hardest part is trusting that your system can not be digested in just one bottlenecked app standing up. This really goes deeper than microservices as well, into event-driven programming. Too much time seems to be spent on people saying, well my monolith is just fine. It might be, but doing a microservice system works, you just have to be able to ride the wave.

deleted 6y ago

[deleted]

ianamartin 6y ago

I despise Uber as a company, but I've always loved them as a technology group ever since I used to go to their meetups in NYC. You can always count on Uber to throw a great party and also come up with the worst possible technology. If you want to know how not to do things, Uber is a very good place to look.

hinkley 6y ago
And all the people who worked there will consider the fact that they did 1) something and 2) were successful means that that 'something' had anything to do with it, and therefore we should keep doing it. Not that this is unique to Uber, mind. I just described half of my coworkers too.

exnuber 6y ago

They say they draw inspiration from DDD and CQRS, yet seem to miss one crucial factor. And no, I'm not talking about bounded contexts. They still focus in re-use of commodity rather than addressing why their microservice approach didn't work. Their strategy for defining how functionality moves down the stack is not bad, but it's not anchored towards the right thing. It's the consistency boundary (or aggregate root if you will). It's the circle you draw around your business rules and say "this group needs to remain consistently enforced". This defines your unit of scale, not re-use. So while they are headed in the right direction, I fear that they are still missing a fundamental piece of information to steer on. I was employed by Uber (I quit after a couple months), and the idea's that they now hint towards were largely rejected by the engineers, and that was less than a year ago. Uber is just in the position to throw a large sum of money at making wrong decisions and getting away with it, because it's not their money. It's VC funny money. DISCLAIMER: Yes, I did indeed create this account to be able to reply anonymously.

pgt 6y ago

Microservices are about scaling teams, not tech.

ExcavateGrandMa 6y ago

STOP WASTING UR TIME AND START HIRING REAL PROs based on their real SKILLZzzzZzzz! I would go this way... (am technical not commercial...) U keep your stuff in production while I study your environment & take notes... And then changes propositions happens! (I may offer to rewrite full or part of a stack for the purpose... and also technology change for independence...) Then conception, testing, validation... PRODUCTION in the long term without headaches and clear understanding! o/

keyraycheck 6y ago

> as Uber grew from 10s to 100s of engineers with multiple teams owning pieces of the tech stack, the monolithic architecture tied the fate of teams together and made it difficult to operate independently. I think this is key sentence. Microservices does not make a lot of sense for teams of 10s, while are a great tool for team of 100s.

kerng 6y ago

Very interesting and seems like a logical next step with things get a bit too messy with micro services. One thing that was not touched on - how is security done in DOMA? * Is auth happening at domain gateways or at each service? * Similarly for encryption, does it terminate at Domain GW or at each service?

alpmeow 6y ago
Hopefully both

SergeAx 6y ago

I felt bad for Uber engineering force while reading whole article. They swamped themselves into that microservices hell due to pressure for teams velocity and TTM, and created enormous tech debt in the process. Now they are trying to patch it up with ridiculous means like putting JSON inside of protobuf messages. I really hope this article could be a warning for management of growing startups for what could happen if you squeeze features out of your team, but managers usually speed-reading those articles just to order "do like Uber did".

stevenalowe 6y ago

Nice to see Uber rediscovering domain-driven design principles.

mbrodersen 6y ago

If you can't make a monolith work, then you will fail even harder trying to make a Microservice Architecture work.