14 ms·

Some thoughts on microservices

146 points by filipn 5y ago

cjfd 5y ago
One should probably not even think of microservices if one employs fewer than 100 programmers. In many cases microservices are introduced because of fashion and/or resume driven design not because it actually makes sense. I prefer refactoring a poorly structured monolith to refactoring poorly structured microservices in every case.
- solmag 5y ago
  I like them for modularity, message queues and asynchronous events. But you are correct, esp. last one.
  - cjfd 5y ago
    Things like message queues and asynchronous events are available for your favorite programming language for in-executable use. You can then do things like running them on a thread pool that has the same number of threads as your machine has cores.
    - jayd16 5y ago
      So you're not only suggesting monolith, but single instance? In process and non-shared queues only? No load balancing across monoliths?
      - imwillofficial 5y ago
        I didn’t read any of that. I read it as more tools in the toolbox.
        jayd16 5y ago
        Without single instance, an in process queue doesn't provide the same kind of functionality that a shared queue does and would probably introduce split brain kind of problems, no?
        imwillofficial 5y ago
        Right, you’re taking it as a suggestion, as in a positive nudge to single instance monolithism. However, I read it as merely making GP aware of other options that are on the table.
      - cjfd 5y ago
        Not necessarily. That one can have these tools in-application doesn't mean one cannot combine this with some form of RPC as well. Especially whether one needs load balancing or not seem to be a completely orthogonal issue. On the other hand, why not a monolith and single instance? If one uses a performant language one would be amazed how much can run on a single machine. If one expects growth one should have some plan to scale, sure, but if one does not go for shenanigans like running queues and the like between applications using some form of RCP one might get amazed how much your single monolith can actually do. All this extra networks stuff and so on is not exactly free.
        jayd16 5y ago
        >On the other hand, why not a monolith and single instance? Single points of failure are bad. I agree its orthogonal, which is why I inquired further about the, imo, odd suggestion.
        ratww 5y ago
        Running a single instance doesn’t mean not having a backup.
  - wongarsu 5y ago
    Plenty of languages offer modularity, message queues and asynchronous events within a monolith. They are quite idiomatic in go, rust and elixir
    - foobarian 5y ago
      Splitting a system into microservices can help individual teams be better stewards of their part of the system. They can release on their own schedule, they can use their own linting rules, heck even a different language, and they can have better control of incoming code changes. With a monolith any random developer can go and flip some private method to public, import it way across the modules, and presto you are now building a ball of mud. Need an out of cycle release? Hopefully you have CI/CD or now you have to beg the SRE in charge to do it for you.
      - Sebb767 5y ago
        > Splitting a system into microservices can help individual teams be better stewards of their part of the system. Team being the keyword here. If you have 3-5 developers per microservice, you're absolutely okay. If you have 3-5 microservices per developer, that's when it gets ugly.
      - yakshaving_jgt 5y ago
        Given your negative characterisation of the "ball of mud", I'm guessing you haven't actually read the original paper. > heck even a different language There's nothing about writing software in different languages that necessitates separating functionality with HTTP calls.
        foobarian 5y ago
        > There's nothing about writing software in different languages that necessitates separating functionality with HTTP calls I mean it's only computers and the only limit to what we can make them do is our imagination. In this case though for the sake of argument what options would I have if I, say, needed to let a remote team add some functionality to my, say, Spring backend but they really prefer to write C# and have their own CI/CD system. I'm not sure how I would accomplish this in a monolith.
- BoorishBears 5y ago
  To me microservices should not be an architecture. You can have pieces of functionality in a monolith that make sense to scale independently, and those should not be micro, they should be meaningful pieces of functionality that justify the overhead of spinning them out. In a way your comment reflects this, a lot of the places that justified microservices were at a scale where their "microservice" was serving more requests than the average company's entire codebase. It's "big data" with 10 GBs of logs all over again.
- belter 5y ago
  The whole problem starts with calling them microservices. Just call it a service oriented architecture please. The whole micro thing is already an indication of the kind of trouble you are getting yourself into. More respect to the monolith please.
  - cjfd 5y ago
    Well, a service-oriented architecture is a more general category then microservices and has been around much longer. In many cases it is much more justifiable on technical grounds. Just to give one example. You have an API that is not the most stable thing in the world and when it misbehaves you do not want it to take the whole application down with it. So you put it in a separate service that can easily be restarted. That is a good technical reason for a separation of executables. Problems arise when people introduce these separations for no good reason. Then one gets all of the problems of RPC and none of the benefits. I.e., one is making things more complicated for no good reason.
    - jjoonathan 5y ago
      > and has been around much longer There's your problem. People understood the tradeoffs, so it needed a rebranding before it could go through another hype cycle.
  - UglyToad 5y ago
    I'm going to build a lucrative personal brand / consulting business by hyping mesoservices, neither micro nor macro, just right.
    - belter 5y ago
      Reality and Fiction mingle together... "Mesoservices: Architecture for optimal trade-off between engineering and operations efficiency" https://www.codemotion.com/talks/mesoservices-architecture-for-optimal-trade-off-between-engineering-and-operations-efficiency-5774 https://www.codemotion.com/talks/mesoservices-architecture-f...
    - 5y ago
- afarrell 5y ago
  It is worth thinking about them before you need them because by the time you gain the skill and infra needed to support them, you’ll have scaled so large that your ability to coordinate and communicate will be overwhelmed.
  - papito 5y ago
    How many startups out there have run into the problem of "oh no, we got too big too fast"?
    - afarrell 5y ago
      A fraction of the ones that are successful. I suppose what I really mean is: If your leadership subordinates (especially new joiners) are starting to struggle to coordinate, consider the possibility that you might already be at that point yet emotionally attached to the notion of staying small as you double or quadruple in population.
      - papito 5y ago
        I can guarantee you that most systems out there could serve production traffic on an old laptop, with well-written code and lean database queries. I have done things in MySQL v3 in 2006 that should not be done with a modern database even today (a taxonomy search engine 3 levels deep. Never again). In the age when database trips and network trips are treated as "free", we somehow arrived at MORE complicated solutions, like microservices. A "hello, world" problem in 99% of the cases is a "hello, world" problem. At Google - it is not. It's a scale problem. Everything is a scale problem at Google first, and a business logic problem second. The FAANG alumni has convinced the new generation of developers that everything is a scale problem.
- m0llusk 5y ago
  This doesn't match my experience at all. In particular, in the case where a small team is working on a series of small projects it can make a lot of sense to split up the components so that any one of the small projects can be built with one or two existing generalized components along with some custom machinery and some glue code to bring it all together. That way development of all of the small products can contribute to some well honed components that get shared by multiple products and development cycles. It seems like this is not coming up because the most common context of application is one or more teams working on a single large project probably forced to grow as fast as possible because of the funding structure involved. Increasingly, though, there are companies doing software development without the need for focus and scale that is so common to venture capital powered groups.
  - mason55 5y ago
    The major difference is whether the small projects are independent applications that use some shared libraries (what I think you mean by "existing generalized components") or whether the small projects all talk to each other to create a single large application. If it's the former then you're just talking about refactoring functionality into a shared library, but at the end of the day you're still just building little monoliths. You don't have to worry about most of the problems that come up with microservices.
- codyb 5y ago
  A coworker at a previous four person startup was always advocating microservices. Having to push back on that over and over was frustrating. We didn’t even have a real devops guy, or a vpc with properly partitioned CIDR blocks to segregate our databases from the public web and we’re going to start adding the complexity of a microservice architecture? For what! We didn’t even have _users_ yet. But try to get folk to dogfood our application since we had no actual users besides the founder and it was like pulling teeth. Totally backwards to me.
BoumTAC 5y ago
I watch the other day, the Lex Fridman podcast with Kevin Systrom, founder of Instagram. He talks about the scaling issue; they were using Django at that time and scale up to 50 millions users with a small team of developers. I do not think a majority of companies have more than 50 millions users and absolutely need to go full microservices. https://www.youtube.com/watch?v=3pvpNKUPbIY https://www.youtube.com/watch?v=3pvpNKUPbIY
- zarkov99 5y ago
  Micro-services are meant to scale the number of developers not the number of users. As the article points out they are meant to address organizational issues and they do - at a significant technical cost.
  - BoumTAC 5y ago
    As I point out, they were a small team of developers before being bought by Facebook. And they still scale up to 50 million users.
    - zarkov99 5y ago
      Sorry, I misinterpreted what you originally wrote.
- hbn 5y ago
  I don't think the number of users nor number of developers is really the deciding factor. Instagram, pre-Facebook acquisition, was a VERY simple application. It was literally just a chronological feed of (strictly square) photos with captions, you followed your friends, could explore hashtags, and not much more. Videos wouldn't even come for a few more years, let alone all the crazy stuff Facebook has hamfisted in there since. For the scope of that app, it would have been absurd to use microservices. And I think most people who are in favor of microservices would say the same thing. To me, what microservices help with is when you're building an entire platform, rather than a single product. Not even necessarily on the scope of Facebook or Google, but I've worked at companies where one team might work on an app for managing social media accounts, and another app helps you optimize the SEO of your website. Neither of those things really want to own the concept of a user they both share, or deal with account creation and whatnot. So that's handled by a dedicated microservice. Now, when you get to a size where you're building a platform, you're likely going to have lots of developers and users, but I don't think whether you use microservices is a function of either of those numbers, and they're just a side effect of the thing you've built.
orefalo 5y ago
Conways law is not about structuring... it's about domain knowledge. ie. if you work alone on a problem, you can only solve for the issues you know.
orefalo 5y ago
I do not agree with many of your points. I also agree with some of them.
alexchamberlain 5y ago
An interesting list, but is this specific to Microservices, or just Service Oriented Architecture in general? For me, other than the obvious size difference, the difference between microservices and "large" (?) services is that a single team breaks down their domain into sensible layers, abstractions etc.
- scottious 5y ago
  A single team can break down their domain into sensible layers and abstractions within a monolith. Similarly, A microservices/SOA architecture can fail to break down their domain into good boundaries and abstractions. I've seen this happen a lot. It's a lot harder to fix bad microservices than to fix a bad monolith
  - alexchamberlain 5y ago
    It's very hard to isolate load from different use cases, use a mixture of different technologies or combine batch, event driven and request/response paradigms within a monolith though. I think certain things are easier to change in a monolith, where as other things are easier to change in a service based design. Depends what mistakes you've made along the way or how the spec/environment changes.
KronisLV 5y ago
> Never start with a microservice architecture if you have a single team. This is probably a good point, however isn't the entirety of the story. Personally, i agree that most teams shouldn't start out with microservices, monoliths can be entirely sufficient and are easier to run and reason about. Otherwise you might end up with so much operational complexity that you don't have much capacity left to actually develop the software and make sure that it's actually good. However, you also need to think about the coupling within your monolith, so that if the need arises, it can be broken up easily. I actually wrote more about this in my blog, in an article called "Moduliths: because we need to scale, but we also cannot afford microservices": https://blog.kronis.dev/articles/modulith-because-we-need-to-scale-but-we-also-cannot-afford-micro-services https://blog.kronis.dev/articles/modulith-because-we-need-to... Where this goes wrong, is that no one actually thinks about this because their code works at that point in time, so they make their PDF report generation logic be tightly coupled to the rest of the codebase, same as with their file upload and handling logic, same with serving the static assets etc., so when suddenly you need to separate the front end from the back end, or extract one of the components because it's blocking updating to newer tech (for example, Java 8 to Java 11, everything else works, that one component breaks, so it would be more logical to keep it on the old/stable version for a bit, instead for it to block everything else), you just can't. Sooner or later, containers also have to be brought up, since they can be a way to do a multitude of applications in a manageable way, but at the same time it's easy to do them wrong, perhaps due to not understanding the tech or some of the potential concerns. Many out there think that "doing containers" involves taking their legacy monolith, putting it inside of a container and calling it a day. It isn't so, and you'll still have plenty of operational challenges if you do that. To do containers "properly", you'd need to actually look into how the application is configured, how it handles logging, external services, and how it handles persistent data. And it's not the "No true Scotsman" fallacy either, there are attempts to collect some of the more useful suggestions in actionable steps, for example: https://12factor.net/ https://12factor.net/ (though those suggestions aren't related directly to containers alone, they can work wonderfully on their own, outside of container deployments) Lastly, i've also seen Kubernetes be used as almost something synonymous to containers - in some environments, you can't have a conversation about containers without it being mentioned. I've also seen projects essentially fail because people chose it due to its popularity and couldn't cope with the complexity it introduced ("Oh hey, now we also need Istio, Kiali, Helm, oh and a Nexus instance to store Helm charts in, and we'll need to write them all, and then also have a service mesh and some key value store for the services"), when something simpler, like Docker Swarm or Hashicorp Nomad would have sufficed. I actually have yet another blog topic on the subject, "Docker Swarm over Kubernetes": https://blog.kronis.dev/articles/docker-swarm-over-kubernetes https://blog.kronis.dev/articles/docker-swarm-over-kubernete... (honestly, this also applies outside of the context of containers, for example, picking something like Apache Kafka over RabbitMQ, and then being stuck dealing with its complexity) In conclusion, lots of consideration should be given when choosing both the architecture for any piece of software, as well as the tech to use to get it right. In some ways, this is slower and more cumbersome than just pushing some files to an FTP server that has PHP running there, but it can also be safer and more productive in the long term (configuration drift and environment rot). Sadly, if the wrong choices are made early, the bad design decisions will compound with time.
FinanceAnon 5y ago
Fundamentally, I don't really see much difference between monoliths and microservices. In a monolith, you just call another function/class, but in microservices that function is a http call. I guess the benefit of microservices is the ability to independently scale different microservices, being able to choose different languages for different microservices and less conflicts in the repo as more people work on it, but you still have to deal with backwards compatibility and versioning of endpoints. I think lambdas are interesting when you look at it this way. A microservice is essentially a set of functions which is constantly deployed as one unit. But with Lambdas, each function is a single unit that can scale independently.
- KronisLV 5y ago
  > Fundamentally, I don't really see much difference between monoliths and microservices. > but in microservices that function is a http call I think that maybe you're understating the complexity that distributed systems may involve. For example, see all of the following: - https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing - https://blog.erratasec.com/2012/06/falsehoods-programmers-believe-about.html - https://medium.com/@kenbantoft/falsehoods-programmers-believe-about-networks-30a328c25c50 Plus, in the current day and age, we still don't have that many convenient ways to make two systems interact over a network. REST and things like GraphQL don't map well to actions, whereas RPC solutions like gRPC also involve a certain amount of boilerplate code and you still need to think about how the concerns above.
  - mikro2nd 5y ago
    And not forgetting "A Note On Distributed Computing" https://www.cc.gatech.edu/classes/AY2010/cs4210_fall/papers/smli_tr-94-29.pdf https://www.cc.gatech.edu/classes/AY2010/cs4210_fall/papers/...
- trixie_ 5y ago
  There’s a huge difference between http and function calls. When I call a function there is no chance it fails because the function ‘cannot be reached’. With services there are endless reasons why one service is unable to communicate with another. It’s a huge amount of overhead to build a system to handle inter-service communication and failure that doesn’t exist when only calling a function in the same binary.
wayoutthere 5y ago
I have found one other use case: limiting the blast radius of ops and deployment issues. As an example, I once was working with a large telecom client who had their AAA web service (yes, don’t roll your own auth, but this client was big enough and had the right expertise on staff to do so) in the same monolith as their account management web service. The account management service saw active, frequent development to support new functionality while the AAA code only got updated a couple times a year. Why touch a critical web service relied upon by literally every product you have when you don’t have to? Your business still functions if account management is offline; but not when authentication is offline. Even short outages to auth were unacceptable (millions of customers) and any updates had to be performed in constrained windows due to the criticality of the service. So we cleaved off the mission-critical parts, stuck them in their own repos to be versioned independently, which let us move faster on the account management work since we could confidently deploy code that wasn’t 100% working because we didn’t need to wait for a maintenance window.
acjohnson55 5y ago
Microservices are not a solution, they're a capability. It's powerful for a team to be able to deploy a tiny service with the absolute minimum amount of explicit plumbing to meet operational requirements. Whether they should break their system up this way is a case by case judgment. Every place I've been, the costs of microservices get overlooked in favor of the illusion of decoupling. Microservices will absolutely make simple features fully contained within a service easier, but as soon as a feature spans services or even impacts the contract of a service, you're in for more pain than in a monolithic architecture. Microservices sieze local simplicity at the cost of integration complexity. Microservices, as a philosophy, is encoding your org design at the networking layer. I hope you get it the factoring right the first time, because it's going to be painful to change. I'm all for small services where they make sense. But just like the advice that "all functions should be small" microservices has been myopically adopted as a design principle, rather than a debatable decision.
- amelius 5y ago
  Your first question in a job interview should be: "do you use microservices?" If the answer is yes, you can save yourself a lot of time.
  - mumblemumble 5y ago
    I know microservices have passed the excitement phase of the hype cycle, but they're not useless. One of the most enjoyable systems I've worked on was a microservice architecture. That said, I'm going make some wild inferences about what you were getting at in ordet to say that I agree that a microservice architecture is probably a solution in search of a problem in most cases. And, even in the cases where it is a good option, I can see all sorts of ways to mess up the implementation. The article is right; the trickiest things to get right about microservices are actually organizational issues, not technical ones. My hot take is that dev teams who are considering adopting microservices should take a serious look at how much ability they have to influence the org chart and inter-team and inter-departmental communication. If management is strictly something that happens to them, I would not give them stellar odds of achieving sustainable success with microservices. Perhaps some other form of SOA, but not actual microservices.
recursivedoubts 5y ago
Microservices means just that: tiny, single purpose services that deploy and scale on their own. I often see appeals to Conway's Law when discussing microservices, but teams don't organize themselves this way. Instead, teams work on a macro services: the email delivery team, or the monitoring team, or whatever. In most cases these macroservices would be best implemented and deployed as a monolith, and then presented to the outside world over a reasonable API.
- throwaway984393 5y ago
  ^ This. Microservices are introduced because it seems like they'll be able to decouple and scale well, and then they make mistakes that make everything even harder than if it'd stayed a monolith. Usually they don't plan for how they'll coordinate their work, and that leaves gaps in the design, and puts more risk on the business. Team A: ↑ Email product: __|_____________ ↑ Service C <---------- / How do we \ | Service D <-- \ | work together | --------------------- \ ------\-------------- | on overall | | How do we manage | | | How do we | | system design? | | stakeholder risk? | | | coordinate changes? | \_______________/ --------------------- | \--------/----------- / | | | | ↓ | \ | Team B: ↓ | | "Data" team: | / Service A <---|------- Service B <--/ On top of that, they don't even make a true microservice. They start directly calling into each others' data rather than interface at an API layer, they make assumptions about how each other works, they don't do load testing or set limits or quotas... and because none of them understand the rest of the system, they don't see that their mutual lack of understanding is the cause of their problems. Even with multiple teams, if they're forced to work inside a monolith, there's a much better chance they will by accident come to understand the rest of the system.
  - mLuby 5y ago
    Total tangent: very nice ASCII diagram, especially for a throwaway account. It's so unusually nice that it'd probably help identify the author if they ever made ASCII diagrams on their non-throwaway account (and who could resist?).
samwillis 5y ago
“The longer I’m in tech, the more I’m convinced microservices were a technical solution to an organizational problem.” @carnage4life on Twitter https://mobile.twitter.com/carnage4life/status/1311702322024644608 https://mobile.twitter.com/carnage4life/status/1311702322024... Now, I’ve not had the pleasure of working in an organisation using Microservices and so I’m only informed anecdotally. But I always assumed they where best used as an API boundary between teams, rather than adding more complexity to a teams work.
- deleted 5y ago
  [deleted]
- bestcoder69 5y ago
  Imagine if all of AWS was a single executable file, but had to handle all the same scale, complexity, and release velocity. The S3 team would have to patiently coordinate with the… like… SageMaker team to figure out how big of servers they need, along with every other single possible shared concern. I picked a ridiculous example there. But just trying to show that if you need services, it should be easy to argue for them on the merits, because it’s gonna be the least bad option.
bluGill 5y ago
I'm looking at micro services for a different reasons: security. I have given up on the idea that our code will ever be completely secure. However micro services means if someone breaks into one service they can't see data belonging to a different service. (that is run each service as a different user, and so OS protections means file commands cannot open such data) This only protects against some threats related to insecure code, but layers of protection is the key to threats and it is useful for the parts it does help.
- Nextgrid 5y ago
  I'd still be concerned about kernel-level exploits in this case. I'd run every service in its own VM.
  - bluGill 5y ago
    There have been examples of people breaking out of VMs. My ultimate wish is to run everything on separate CPUs.
- daxfohl 5y ago
  I could maybe see that for some specialized case, but for the general case it seems like the more independent, distributed things you're juggling, the more likely you are to end up with security holes in the first place. The time you have to spend on security would have to be spread too broad and thin.
  - bluGill 5y ago
    The cost varies. Some of the code I work with is safety critical - people can die if it isn't working. If someone breaks into our system and gets private data that "only" costs us a lot of money, but if they break into our system and take over the safety critical parts people die.
    - daxfohl 5y ago
      I see, yeah sounds like "control plane" vs "data plane", which is a good place to split things.
deleted 5y ago
[deleted]
tomrod 5y ago
Hey! I enjoyed this post a lot. I agree with almost all your points raised except for single-team == no microservices. Your prose makes your points easy to understand. I have a couple layout comments: First, I love that you have a high contract text-to-background. That is really helpful for me. There was/is a trend to have light gray on white backgrounds for blogs; this is absolutely a terrible pattern. I appreciate that you did not go this route. Second: serif fonts are difficult to read when the font size is relatively small. Something like Jura or similar could maintain the "terminal" feel without getting bogged down in serifs. Third: I have a really hard time reading content when it uses smaller fonts and uses a minor fraction of the screen. This is what I see: https://imgur.com/a/NPCBkHJ https://imgur.com/a/NPCBkHJ -- I am getting older, and reading smaller fonts is increasingly difficult for me. I tend to keep my zoom at 150%, something about this page forced it back to 100%. I am not well versed in responsive design, so I don't know the technical details for it, but having zoom maintain or using a larger font would save some cognitive cost to older users like myself needing to zoom in. Thanks for your thoughts on microservices!
nunez 5y ago
OP makes a really good point in that dev teams also need to own their infrastructure when everything is a microservice. Asking "DevOps" to change an infra component prior to a release just shifts the monolith under the rug while also completely defeating the point of DevOps. Lots of devs don't know infrastructure that well, though this is changing with the adoption of Kubernetes. Additionally, most devs don't want to go on-call when their app crashes unexpectedly.
roberthahn 5y ago
I find people’s reactions to microservices to be fascinating. I have always looked at them with the same framing as Unix userland tools - many small, focused apps that do something really well, coupled with a super generic IPC mechanism - in Unix’s case, using pipes or message queues. But Unix isn’t all small tools. We have servers for the heavy work- like databases. The challenge then becomes; how do you design that IPC mechanism? Maybe it exists! I don’t know the answer yet. But it’s something I think about a lot and I haven’t seen compelling evidence for “microservices are always bad, no exceptions”
- cjfd 5y ago
  Well, pipes are very nice and useful for small processing tasks. "I want to know how often the word 'Parameter' occurs in the source files inside this repository." That is great! pipes are your friend. They are the fastest way known to mankind to solve this problem. But they are also fragile. You may not have thought about the fact that the word NoParameter also occurs in the code base and you did not want to count that. Now, when writing something big and complicated pipes simply cannot keep up. Nobody wants a browser that is actually 100 small executables that communicate over pipes. It will be horrible. The IPC mechanism that you are looking for and that can actually handle what is needed for a browser is called 'function call' and it was already quite popular when C was introduced.... The interaction mechanisms between services, e.g. REST calls, is somewhere in between pipes and function calls. They are a bit more reliable than pipes and less reliable than function calls. They can be used for more complicated task than what pipes are used for but should not be used for task that are so complicated that they need function calls.
papito 5y ago
First of all, we called these "distributed systems". We reserved those for rare cases where scale was THE requirement, and you jumped into it fully expecting to have no life. The world did not finally "crack" distributed systems. Most companies created multiple microservices, putting their small dev team underwater because "this is the way at the FAANG". All of your DRY principles are out the window, and now you have to debug in production - the new word for that is OBSERVABILITY. Not to mention that the actual reason for distributed systems is to scale multiple parts of the system independently, but you need to KNOW what has to scale individually before you do it. What I see is that the topology really reflects the company structure, of course. It's not "what has to scale separately", it's "team Y is working on X, and team Y does not want to talk to team Z, so they will create a service to make sure they don't have to talk to people". Except that this is a giant self-own. We all still have to talk to each other, like, a LOT, because things just keep breaking all the time and no one knows why. Dropbox, Instagram, StackOverflow - these companies are largely monoliths to this day. You thinking that your small outfit needs to be like Google is highly arrogant. And don't get me started on the amount of money, people, CPU cycles, and CO2 emissions wasted on this solving of the problem most people don't have.
daxfohl 5y ago
For me it's the repetition in horizontal tasks. Need to update TLS version? Need to move to a new region? Need to add i18n? You have to do these things 20 times instead of one.
Sohcahtoa82 5y ago
My thoughts on microservices... They're fine. But what's NOT fine are nanoservices. I did security on a project once where it seemed that every function was its own microservice. User registration, user login, and password resetting were each a separate microservice. It was an utter nightmare.
- coding123 5y ago
  Did they do this? user-registration.api.dev.my-company.com user-login.api.dev.my-company.com password-reset.api.dev.my-company.com or paths by some huge k8s nginx ingress?
  - Sohcahtoa82 5y ago
    They started with the latter, but started shifting towards the former.
- jeffbee 5y ago
  Shoot, that's practically a monolith compared to the nightmare from which I resigned earlier this year. No way would they have allowed the entire password reset to reside in a single service. It would have been in the password-db-reader service, the password-db-mutator service, and the password-checker-service, among others. Absolutely terrifying and unmanageable.
  - spmurrayzzz 5y ago
    This is the flavor of reductio ad absurdum example I always give when people hype up the "micro" part of microservices. Only meant to be illustrative— I had no idea that any org was actually doing that sort of nonsense, yikes.
- zkldi 5y ago
  Reminds me of this classic: https://www.youtube.com/watch?v=y8OnoxKotPQ https://www.youtube.com/watch?v=y8OnoxKotPQ
choeger 5y ago
If I was CTO of a company, microservices would give me nightmares. How do you do due diligence on used free software (licenses and security updates)? How do you plan the resource usage of your whole setup if every developer can add a new autoscaling service? Who is actually keeping track on deployments so we don't accidentally overload the system? How do you refactor a cross-service feature consistently? And the worst part: Who keeps track of the n*n contracts between the services? I mean yes, I know that each of these problems can be solved, sometimes in a relatively straightforward manner. But who really has all these aspects covered and doesn't run some services that started to smell weirdly a couple of months ago?
- clintonb 5y ago
  > How do you do due diligence on used free software (licenses and security updates)? Use the same process you would use if you had a monolith. The rest of your issues can be solved by planning out your services, rather than giving everyone free reign to make a new service. Switching to services doesn't magically mean your teams stop talking and designing together.
  - choeger 5y ago
    Yeah and the inter-service specification lives where? How is it monitored, tested, enforced? I have the feeling that with the (quite possible!) addition of an inter-service codebase we would end up with a distributed monolith, i.e., a program that doesn't target a single computer but a particular substrate. I don't know whether that's a good design, though. Benefits: The program becomes more transparent and resilient to nonfunctional problems. It is also much easier to replace parts of the program. Downsides: Executing on a developer's workstation (critical for productivity and quality, IMO) might become harder. Efficiency gets reduced by orders of magnitude in certain spots.
maxdo 5y ago
Yet another “based on my experience” opinion. Based on my experience our service would fail due to performance because we’ll , nodejs is still single thread. Given this we should either duplicate deployment of big service by roles that gives you same level of orchestrating complexity or rewrite in different languages, means hello microservices again :) ps our product was initially written by non tech cofounders that used heroku and microservices from day one. They used a swarm of small services to stay on free tier. So microservices in this use case are cheaper. And yes, it's simpler for non experience developer to get up to speed with your backend if it's a simple service.
- SomeCallMeTim 5y ago
  Node can run multi-process with the Cluster module. Node being written as a monolith but running it in 100 instances is also an option. Roles can be implemented in software. There's much less "orchestrating complexity" when you're deploying a single service. So Node being single-threaded is not itself a reason to use microservices. I tend toward writing a monolith for the core API of a service, but then break out microservices for tasks that need to scale independently (or that need to run on high-memory/high-performance instances, for example). So I'm not totally against using microservices. But we should choose to use them when they're to our advantage to use them, not just "because they're already written that way."
  - maxdo 5y ago
    If you deploy same binary under different roles, it's same issue with complexity. With modern tooling deployment and managed storages is not a problem at all you use templates or buildbacks or even lambda combined with gitlab github CI abilities. Recent progress allows you to embrace zero-ops and microservices. In my team we don't have a dedicated ops person, and deployment from dev to prod is running via git by developer. Node itself has a very bad profiling tooling compares to more "adult" languages. if you run a microservice it's much easier to spot a problem in CPU or especially memory leak. And vise versa if you doing something in scala, or java, microservices benefits are minor. It's also so much easier later to rewrite some of the services in a more performant language.
daxfohl 5y ago
I was going to say that one nice aspect is that rollbacks are less frequent because the size of each deployment is smaller. But I've worked on million line code bases that still do weekly CICD pretty well, so I don't know if that advantage really holds water.
bob1029 5y ago
Start with a monolith and only break it up if you are actually forced to by the computer (i.e. the process won't fit in ram anymore or eats all the CPU/IO). The second you break up your monolith, you lose its most powerful feature - The direct method invocation. The amount of time I see developers spending on JSON wire protocols, CORS problems, API endpoint designs, et. al. really is starting to concern me. I sometimes wonder if anyone wants to do any actual work or if this is just a big game to some people. I did the full trip on this microservices rollercoaster. Monolith => uServices => Monolith. I used to vehemently advocate for using microservices because of how easy it would be to segment all the concerns into happy little buckets individuals could own. We used to sell our product to our customers as having a "microservices oriented architecture" as if that was magically going to solve all of our problems or was otherwise some inherent feature that our customers would be expected to care about. All this stuff really did for us is cause all of our current customers to walk away and force a re-evaluation of our desire to do business in this market. All the fun/shiny technology conversations and ideas instantly evaporated into a cloud of reality. We are back on the right track. Hardcore monorepo/monolith design zealotry has recovered our ship. We are focused on the business and customers again. The sense of relief as we deprecated our final service-to-service JSON API/controllers was immense. No more checking 10 different piles of logs or pulling out wireshark to figure out what the fuck is happening in-between 2 different code piles at arbitrary versions.
- jspash 5y ago
  Here! here! I'm within spitting distance of the end of a project of collapsing a service-oriented system back into a Majestic Monolith. Every step of the way has reduced the lines of code, fixed bugs, saved money, saved time. It's been such a joy that I'm considering doing only this as a side hustle. "Saving" companies who were sold an over-complicated dream.
  - bob1029 5y ago
    > It's been such a joy that I'm considering doing only this as a side hustle. "Saving" companies who were sold an over-complicated dream. This has crossed my mind a lot lately. I think we are looking directly at one of the largest emerging markets in technology. What do we think the TAM is going to be for undoing webscale monstrosities by 2025? Not every business will fail due to their poor technology choices and will be able to pay some serious consulting fees... I've practically got a system for doing this now. It mostly starts with domain modeling in excel and all the business stakeholders being in the loop at the same time until everyone agrees. I find if you get this part right it doesn't really matter if you use C# vs python, or AWS vs on-prem to build the actual product. Hard to get opinionated and locked in when your deploy to prod involves a 100 meg zip file and 3 lines of powershell ran against a single box.
kakoni 5y ago
"microservices is a psyop by big tech to make deploying & maintaining software so insanely difficult that future potential competitors are too tied up trying to keep the cloud equivalent of "hello world" afloat to present any real threat" https://twitter.com/htmx_org/status/1462636513578135556 https://twitter.com/htmx_org/status/1462636513578135556

146 points by filipn 5y ago

cjfd 5y ago

One should probably not even think of microservices if one employs fewer than 100 programmers. In many cases microservices are introduced because of fashion and/or resume driven design not because it actually makes sense. I prefer refactoring a poorly structured monolith to refactoring poorly structured microservices in every case.

solmag 5y ago
I like them for modularity, message queues and asynchronous events. But you are correct, esp. last one.
- cjfd 5y ago
  Things like message queues and asynchronous events are available for your favorite programming language for in-executable use. You can then do things like running them on a thread pool that has the same number of threads as your machine has cores.
  - jayd16 5y ago
    So you're not only suggesting monolith, but single instance? In process and non-shared queues only? No load balancing across monoliths?
    - imwillofficial 5y ago
      I didn’t read any of that. I read it as more tools in the toolbox.
      - jayd16 5y ago
        Without single instance, an in process queue doesn't provide the same kind of functionality that a shared queue does and would probably introduce split brain kind of problems, no?
        imwillofficial 5y ago
        Right, you’re taking it as a suggestion, as in a positive nudge to single instance monolithism. However, I read it as merely making GP aware of other options that are on the table.
    - cjfd 5y ago
      Not necessarily. That one can have these tools in-application doesn't mean one cannot combine this with some form of RPC as well. Especially whether one needs load balancing or not seem to be a completely orthogonal issue. On the other hand, why not a monolith and single instance? If one uses a performant language one would be amazed how much can run on a single machine. If one expects growth one should have some plan to scale, sure, but if one does not go for shenanigans like running queues and the like between applications using some form of RCP one might get amazed how much your single monolith can actually do. All this extra networks stuff and so on is not exactly free.
      - jayd16 5y ago
        >On the other hand, why not a monolith and single instance? Single points of failure are bad. I agree its orthogonal, which is why I inquired further about the, imo, odd suggestion.
        ratww 5y ago
        Running a single instance doesn’t mean not having a backup.
- wongarsu 5y ago
  Plenty of languages offer modularity, message queues and asynchronous events within a monolith. They are quite idiomatic in go, rust and elixir
  - foobarian 5y ago
    Splitting a system into microservices can help individual teams be better stewards of their part of the system. They can release on their own schedule, they can use their own linting rules, heck even a different language, and they can have better control of incoming code changes. With a monolith any random developer can go and flip some private method to public, import it way across the modules, and presto you are now building a ball of mud. Need an out of cycle release? Hopefully you have CI/CD or now you have to beg the SRE in charge to do it for you.
    - Sebb767 5y ago
      > Splitting a system into microservices can help individual teams be better stewards of their part of the system. Team being the keyword here. If you have 3-5 developers per microservice, you're absolutely okay. If you have 3-5 microservices per developer, that's when it gets ugly.
    - yakshaving_jgt 5y ago
      Given your negative characterisation of the "ball of mud", I'm guessing you haven't actually read the original paper. > heck even a different language There's nothing about writing software in different languages that necessitates separating functionality with HTTP calls.
      - foobarian 5y ago
        > There's nothing about writing software in different languages that necessitates separating functionality with HTTP calls I mean it's only computers and the only limit to what we can make them do is our imagination. In this case though for the sake of argument what options would I have if I, say, needed to let a remote team add some functionality to my, say, Spring backend but they really prefer to write C# and have their own CI/CD system. I'm not sure how I would accomplish this in a monolith.
BoorishBears 5y ago
To me microservices should not be an architecture. You can have pieces of functionality in a monolith that make sense to scale independently, and those should not be micro, they should be meaningful pieces of functionality that justify the overhead of spinning them out. In a way your comment reflects this, a lot of the places that justified microservices were at a scale where their "microservice" was serving more requests than the average company's entire codebase. It's "big data" with 10 GBs of logs all over again.
belter 5y ago
The whole problem starts with calling them microservices. Just call it a service oriented architecture please. The whole micro thing is already an indication of the kind of trouble you are getting yourself into. More respect to the monolith please.
- cjfd 5y ago
  Well, a service-oriented architecture is a more general category then microservices and has been around much longer. In many cases it is much more justifiable on technical grounds. Just to give one example. You have an API that is not the most stable thing in the world and when it misbehaves you do not want it to take the whole application down with it. So you put it in a separate service that can easily be restarted. That is a good technical reason for a separation of executables. Problems arise when people introduce these separations for no good reason. Then one gets all of the problems of RPC and none of the benefits. I.e., one is making things more complicated for no good reason.
  - jjoonathan 5y ago
    > and has been around much longer There's your problem. People understood the tradeoffs, so it needed a rebranding before it could go through another hype cycle.
- UglyToad 5y ago
  I'm going to build a lucrative personal brand / consulting business by hyping mesoservices, neither micro nor macro, just right.
  - belter 5y ago
    Reality and Fiction mingle together... "Mesoservices: Architecture for optimal trade-off between engineering and operations efficiency" https://www.codemotion.com/talks/mesoservices-architecture-for-optimal-trade-off-between-engineering-and-operations-efficiency-5774 https://www.codemotion.com/talks/mesoservices-architecture-f...
  - 5y ago
afarrell 5y ago
It is worth thinking about them before you need them because by the time you gain the skill and infra needed to support them, you’ll have scaled so large that your ability to coordinate and communicate will be overwhelmed.
- papito 5y ago
  How many startups out there have run into the problem of "oh no, we got too big too fast"?
  - afarrell 5y ago
    A fraction of the ones that are successful. I suppose what I really mean is: If your leadership subordinates (especially new joiners) are starting to struggle to coordinate, consider the possibility that you might already be at that point yet emotionally attached to the notion of staying small as you double or quadruple in population.
    - papito 5y ago
      I can guarantee you that most systems out there could serve production traffic on an old laptop, with well-written code and lean database queries. I have done things in MySQL v3 in 2006 that should not be done with a modern database even today (a taxonomy search engine 3 levels deep. Never again). In the age when database trips and network trips are treated as "free", we somehow arrived at MORE complicated solutions, like microservices. A "hello, world" problem in 99% of the cases is a "hello, world" problem. At Google - it is not. It's a scale problem. Everything is a scale problem at Google first, and a business logic problem second. The FAANG alumni has convinced the new generation of developers that everything is a scale problem.
m0llusk 5y ago
This doesn't match my experience at all. In particular, in the case where a small team is working on a series of small projects it can make a lot of sense to split up the components so that any one of the small projects can be built with one or two existing generalized components along with some custom machinery and some glue code to bring it all together. That way development of all of the small products can contribute to some well honed components that get shared by multiple products and development cycles. It seems like this is not coming up because the most common context of application is one or more teams working on a single large project probably forced to grow as fast as possible because of the funding structure involved. Increasingly, though, there are companies doing software development without the need for focus and scale that is so common to venture capital powered groups.
- mason55 5y ago
  The major difference is whether the small projects are independent applications that use some shared libraries (what I think you mean by "existing generalized components") or whether the small projects all talk to each other to create a single large application. If it's the former then you're just talking about refactoring functionality into a shared library, but at the end of the day you're still just building little monoliths. You don't have to worry about most of the problems that come up with microservices.
codyb 5y ago
A coworker at a previous four person startup was always advocating microservices. Having to push back on that over and over was frustrating. We didn’t even have a real devops guy, or a vpc with properly partitioned CIDR blocks to segregate our databases from the public web and we’re going to start adding the complexity of a microservice architecture? For what! We didn’t even have _users_ yet. But try to get folk to dogfood our application since we had no actual users besides the founder and it was like pulling teeth. Totally backwards to me.

BoumTAC 5y ago

I watch the other day, the Lex Fridman podcast with Kevin Systrom, founder of Instagram. He talks about the scaling issue; they were using Django at that time and scale up to 50 millions users with a small team of developers. I do not think a majority of companies have more than 50 millions users and absolutely need to go full microservices. https://www.youtube.com/watch?v=3pvpNKUPbIY https://www.youtube.com/watch?v=3pvpNKUPbIY

zarkov99 5y ago
Micro-services are meant to scale the number of developers not the number of users. As the article points out they are meant to address organizational issues and they do - at a significant technical cost.
- BoumTAC 5y ago
  As I point out, they were a small team of developers before being bought by Facebook. And they still scale up to 50 million users.
  - zarkov99 5y ago
    Sorry, I misinterpreted what you originally wrote.
hbn 5y ago
I don't think the number of users nor number of developers is really the deciding factor. Instagram, pre-Facebook acquisition, was a VERY simple application. It was literally just a chronological feed of (strictly square) photos with captions, you followed your friends, could explore hashtags, and not much more. Videos wouldn't even come for a few more years, let alone all the crazy stuff Facebook has hamfisted in there since. For the scope of that app, it would have been absurd to use microservices. And I think most people who are in favor of microservices would say the same thing. To me, what microservices help with is when you're building an entire platform, rather than a single product. Not even necessarily on the scope of Facebook or Google, but I've worked at companies where one team might work on an app for managing social media accounts, and another app helps you optimize the SEO of your website. Neither of those things really want to own the concept of a user they both share, or deal with account creation and whatnot. So that's handled by a dedicated microservice. Now, when you get to a size where you're building a platform, you're likely going to have lots of developers and users, but I don't think whether you use microservices is a function of either of those numbers, and they're just a side effect of the thing you've built.

orefalo 5y ago

Conways law is not about structuring... it's about domain knowledge. ie. if you work alone on a problem, you can only solve for the issues you know.

orefalo 5y ago

I do not agree with many of your points. I also agree with some of them.

alexchamberlain 5y ago

An interesting list, but is this specific to Microservices, or just Service Oriented Architecture in general? For me, other than the obvious size difference, the difference between microservices and "large" (?) services is that a single team breaks down their domain into sensible layers, abstractions etc.

scottious 5y ago
A single team can break down their domain into sensible layers and abstractions within a monolith. Similarly, A microservices/SOA architecture can fail to break down their domain into good boundaries and abstractions. I've seen this happen a lot. It's a lot harder to fix bad microservices than to fix a bad monolith
- alexchamberlain 5y ago
  It's very hard to isolate load from different use cases, use a mixture of different technologies or combine batch, event driven and request/response paradigms within a monolith though. I think certain things are easier to change in a monolith, where as other things are easier to change in a service based design. Depends what mistakes you've made along the way or how the spec/environment changes.

KronisLV 5y ago

> Never start with a microservice architecture if you have a single team. This is probably a good point, however isn't the entirety of the story. Personally, i agree that most teams shouldn't start out with microservices, monoliths can be entirely sufficient and are easier to run and reason about. Otherwise you might end up with so much operational complexity that you don't have much capacity left to actually develop the software and make sure that it's actually good. However, you also need to think about the coupling within your monolith, so that if the need arises, it can be broken up easily. I actually wrote more about this in my blog, in an article called "Moduliths: because we need to scale, but we also cannot afford microservices": https://blog.kronis.dev/articles/modulith-because-we-need-to-scale-but-we-also-cannot-afford-micro-services https://blog.kronis.dev/articles/modulith-because-we-need-to... Where this goes wrong, is that no one actually thinks about this because their code works at that point in time, so they make their PDF report generation logic be tightly coupled to the rest of the codebase, same as with their file upload and handling logic, same with serving the static assets etc., so when suddenly you need to separate the front end from the back end, or extract one of the components because it's blocking updating to newer tech (for example, Java 8 to Java 11, everything else works, that one component breaks, so it would be more logical to keep it on the old/stable version for a bit, instead for it to block everything else), you just can't. Sooner or later, containers also have to be brought up, since they can be a way to do a multitude of applications in a manageable way, but at the same time it's easy to do them wrong, perhaps due to not understanding the tech or some of the potential concerns. Many out there think that "doing containers" involves taking their legacy monolith, putting it inside of a container and calling it a day. It isn't so, and you'll still have plenty of operational challenges if you do that. To do containers "properly", you'd need to actually look into how the application is configured, how it handles logging, external services, and how it handles persistent data. And it's not the "No true Scotsman" fallacy either, there are attempts to collect some of the more useful suggestions in actionable steps, for example: https://12factor.net/ https://12factor.net/ (though those suggestions aren't related directly to containers alone, they can work wonderfully on their own, outside of container deployments) Lastly, i've also seen Kubernetes be used as almost something synonymous to containers - in some environments, you can't have a conversation about containers without it being mentioned. I've also seen projects essentially fail because people chose it due to its popularity and couldn't cope with the complexity it introduced ("Oh hey, now we also need Istio, Kiali, Helm, oh and a Nexus instance to store Helm charts in, and we'll need to write them all, and then also have a service mesh and some key value store for the services"), when something simpler, like Docker Swarm or Hashicorp Nomad would have sufficed. I actually have yet another blog topic on the subject, "Docker Swarm over Kubernetes": https://blog.kronis.dev/articles/docker-swarm-over-kubernetes https://blog.kronis.dev/articles/docker-swarm-over-kubernete... (honestly, this also applies outside of the context of containers, for example, picking something like Apache Kafka over RabbitMQ, and then being stuck dealing with its complexity) In conclusion, lots of consideration should be given when choosing both the architecture for any piece of software, as well as the tech to use to get it right. In some ways, this is slower and more cumbersome than just pushing some files to an FTP server that has PHP running there, but it can also be safer and more productive in the long term (configuration drift and environment rot). Sadly, if the wrong choices are made early, the bad design decisions will compound with time.

FinanceAnon 5y ago

Fundamentally, I don't really see much difference between monoliths and microservices. In a monolith, you just call another function/class, but in microservices that function is a http call. I guess the benefit of microservices is the ability to independently scale different microservices, being able to choose different languages for different microservices and less conflicts in the repo as more people work on it, but you still have to deal with backwards compatibility and versioning of endpoints. I think lambdas are interesting when you look at it this way. A microservice is essentially a set of functions which is constantly deployed as one unit. But with Lambdas, each function is a single unit that can scale independently.

KronisLV 5y ago
> Fundamentally, I don't really see much difference between monoliths and microservices. > but in microservices that function is a http call I think that maybe you're understating the complexity that distributed systems may involve. For example, see all of the following: - https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing - https://blog.erratasec.com/2012/06/falsehoods-programmers-believe-about.html - https://medium.com/@kenbantoft/falsehoods-programmers-believe-about-networks-30a328c25c50 Plus, in the current day and age, we still don't have that many convenient ways to make two systems interact over a network. REST and things like GraphQL don't map well to actions, whereas RPC solutions like gRPC also involve a certain amount of boilerplate code and you still need to think about how the concerns above.
- mikro2nd 5y ago
  And not forgetting "A Note On Distributed Computing" https://www.cc.gatech.edu/classes/AY2010/cs4210_fall/papers/smli_tr-94-29.pdf https://www.cc.gatech.edu/classes/AY2010/cs4210_fall/papers/...
trixie_ 5y ago
There’s a huge difference between http and function calls. When I call a function there is no chance it fails because the function ‘cannot be reached’. With services there are endless reasons why one service is unable to communicate with another. It’s a huge amount of overhead to build a system to handle inter-service communication and failure that doesn’t exist when only calling a function in the same binary.

wayoutthere 5y ago

I have found one other use case: limiting the blast radius of ops and deployment issues. As an example, I once was working with a large telecom client who had their AAA web service (yes, don’t roll your own auth, but this client was big enough and had the right expertise on staff to do so) in the same monolith as their account management web service. The account management service saw active, frequent development to support new functionality while the AAA code only got updated a couple times a year. Why touch a critical web service relied upon by literally every product you have when you don’t have to? Your business still functions if account management is offline; but not when authentication is offline. Even short outages to auth were unacceptable (millions of customers) and any updates had to be performed in constrained windows due to the criticality of the service. So we cleaved off the mission-critical parts, stuck them in their own repos to be versioned independently, which let us move faster on the account management work since we could confidently deploy code that wasn’t 100% working because we didn’t need to wait for a maintenance window.

acjohnson55 5y ago

Microservices are not a solution, they're a capability. It's powerful for a team to be able to deploy a tiny service with the absolute minimum amount of explicit plumbing to meet operational requirements. Whether they should break their system up this way is a case by case judgment. Every place I've been, the costs of microservices get overlooked in favor of the illusion of decoupling. Microservices will absolutely make simple features fully contained within a service easier, but as soon as a feature spans services or even impacts the contract of a service, you're in for more pain than in a monolithic architecture. Microservices sieze local simplicity at the cost of integration complexity. Microservices, as a philosophy, is encoding your org design at the networking layer. I hope you get it the factoring right the first time, because it's going to be painful to change. I'm all for small services where they make sense. But just like the advice that "all functions should be small" microservices has been myopically adopted as a design principle, rather than a debatable decision.

amelius 5y ago
Your first question in a job interview should be: "do you use microservices?" If the answer is yes, you can save yourself a lot of time.
- mumblemumble 5y ago
  I know microservices have passed the excitement phase of the hype cycle, but they're not useless. One of the most enjoyable systems I've worked on was a microservice architecture. That said, I'm going make some wild inferences about what you were getting at in ordet to say that I agree that a microservice architecture is probably a solution in search of a problem in most cases. And, even in the cases where it is a good option, I can see all sorts of ways to mess up the implementation. The article is right; the trickiest things to get right about microservices are actually organizational issues, not technical ones. My hot take is that dev teams who are considering adopting microservices should take a serious look at how much ability they have to influence the org chart and inter-team and inter-departmental communication. If management is strictly something that happens to them, I would not give them stellar odds of achieving sustainable success with microservices. Perhaps some other form of SOA, but not actual microservices.

recursivedoubts 5y ago

Microservices means just that: tiny, single purpose services that deploy and scale on their own. I often see appeals to Conway's Law when discussing microservices, but teams don't organize themselves this way. Instead, teams work on a macro services: the email delivery team, or the monitoring team, or whatever. In most cases these macroservices would be best implemented and deployed as a monolith, and then presented to the outside world over a reasonable API.

throwaway984393 5y ago
^ This. Microservices are introduced because it seems like they'll be able to decouple and scale well, and then they make mistakes that make everything even harder than if it'd stayed a monolith. Usually they don't plan for how they'll coordinate their work, and that leaves gaps in the design, and puts more risk on the business. Team A: ↑ Email product: __|_____________ ↑ Service C <---------- / How do we \ | Service D <-- \ | work together | --------------------- \ ------\-------------- | on overall | | How do we manage | | | How do we | | system design? | | stakeholder risk? | | | coordinate changes? | \_______________/ --------------------- | \--------/----------- / | | | | ↓ | \ | Team B: ↓ | | "Data" team: | / Service A <---|------- Service B <--/ On top of that, they don't even make a true microservice. They start directly calling into each others' data rather than interface at an API layer, they make assumptions about how each other works, they don't do load testing or set limits or quotas... and because none of them understand the rest of the system, they don't see that their mutual lack of understanding is the cause of their problems. Even with multiple teams, if they're forced to work inside a monolith, there's a much better chance they will by accident come to understand the rest of the system.
- mLuby 5y ago
  Total tangent: very nice ASCII diagram, especially for a throwaway account. It's so unusually nice that it'd probably help identify the author if they ever made ASCII diagrams on their non-throwaway account (and who could resist?).

samwillis 5y ago

“The longer I’m in tech, the more I’m convinced microservices were a technical solution to an organizational problem.” @carnage4life on Twitter https://mobile.twitter.com/carnage4life/status/1311702322024644608 https://mobile.twitter.com/carnage4life/status/1311702322024... Now, I’ve not had the pleasure of working in an organisation using Microservices and so I’m only informed anecdotally. But I always assumed they where best used as an API boundary between teams, rather than adding more complexity to a teams work.

deleted 5y ago
[deleted]
bestcoder69 5y ago
Imagine if all of AWS was a single executable file, but had to handle all the same scale, complexity, and release velocity. The S3 team would have to patiently coordinate with the… like… SageMaker team to figure out how big of servers they need, along with every other single possible shared concern. I picked a ridiculous example there. But just trying to show that if you need services, it should be easy to argue for them on the merits, because it’s gonna be the least bad option.

bluGill 5y ago

I'm looking at micro services for a different reasons: security. I have given up on the idea that our code will ever be completely secure. However micro services means if someone breaks into one service they can't see data belonging to a different service. (that is run each service as a different user, and so OS protections means file commands cannot open such data) This only protects against some threats related to insecure code, but layers of protection is the key to threats and it is useful for the parts it does help.

Nextgrid 5y ago
I'd still be concerned about kernel-level exploits in this case. I'd run every service in its own VM.
- bluGill 5y ago
  There have been examples of people breaking out of VMs. My ultimate wish is to run everything on separate CPUs.
daxfohl 5y ago
I could maybe see that for some specialized case, but for the general case it seems like the more independent, distributed things you're juggling, the more likely you are to end up with security holes in the first place. The time you have to spend on security would have to be spread too broad and thin.
- bluGill 5y ago
  The cost varies. Some of the code I work with is safety critical - people can die if it isn't working. If someone breaks into our system and gets private data that "only" costs us a lot of money, but if they break into our system and take over the safety critical parts people die.
  - daxfohl 5y ago
    I see, yeah sounds like "control plane" vs "data plane", which is a good place to split things.

deleted 5y ago

[deleted]

tomrod 5y ago

Hey! I enjoyed this post a lot. I agree with almost all your points raised except for single-team == no microservices. Your prose makes your points easy to understand. I have a couple layout comments: First, I love that you have a high contract text-to-background. That is really helpful for me. There was/is a trend to have light gray on white backgrounds for blogs; this is absolutely a terrible pattern. I appreciate that you did not go this route. Second: serif fonts are difficult to read when the font size is relatively small. Something like Jura or similar could maintain the "terminal" feel without getting bogged down in serifs. Third: I have a really hard time reading content when it uses smaller fonts and uses a minor fraction of the screen. This is what I see: https://imgur.com/a/NPCBkHJ https://imgur.com/a/NPCBkHJ -- I am getting older, and reading smaller fonts is increasingly difficult for me. I tend to keep my zoom at 150%, something about this page forced it back to 100%. I am not well versed in responsive design, so I don't know the technical details for it, but having zoom maintain or using a larger font would save some cognitive cost to older users like myself needing to zoom in. Thanks for your thoughts on microservices!

nunez 5y ago

OP makes a really good point in that dev teams also need to own their infrastructure when everything is a microservice. Asking "DevOps" to change an infra component prior to a release just shifts the monolith under the rug while also completely defeating the point of DevOps. Lots of devs don't know infrastructure that well, though this is changing with the adoption of Kubernetes. Additionally, most devs don't want to go on-call when their app crashes unexpectedly.

roberthahn 5y ago

I find people’s reactions to microservices to be fascinating. I have always looked at them with the same framing as Unix userland tools - many small, focused apps that do something really well, coupled with a super generic IPC mechanism - in Unix’s case, using pipes or message queues. But Unix isn’t all small tools. We have servers for the heavy work- like databases. The challenge then becomes; how do you design that IPC mechanism? Maybe it exists! I don’t know the answer yet. But it’s something I think about a lot and I haven’t seen compelling evidence for “microservices are always bad, no exceptions”

cjfd 5y ago
Well, pipes are very nice and useful for small processing tasks. "I want to know how often the word 'Parameter' occurs in the source files inside this repository." That is great! pipes are your friend. They are the fastest way known to mankind to solve this problem. But they are also fragile. You may not have thought about the fact that the word NoParameter also occurs in the code base and you did not want to count that. Now, when writing something big and complicated pipes simply cannot keep up. Nobody wants a browser that is actually 100 small executables that communicate over pipes. It will be horrible. The IPC mechanism that you are looking for and that can actually handle what is needed for a browser is called 'function call' and it was already quite popular when C was introduced.... The interaction mechanisms between services, e.g. REST calls, is somewhere in between pipes and function calls. They are a bit more reliable than pipes and less reliable than function calls. They can be used for more complicated task than what pipes are used for but should not be used for task that are so complicated that they need function calls.

papito 5y ago

First of all, we called these "distributed systems". We reserved those for rare cases where scale was THE requirement, and you jumped into it fully expecting to have no life. The world did not finally "crack" distributed systems. Most companies created multiple microservices, putting their small dev team underwater because "this is the way at the FAANG". All of your DRY principles are out the window, and now you have to debug in production - the new word for that is OBSERVABILITY. Not to mention that the actual reason for distributed systems is to scale multiple parts of the system independently, but you need to KNOW what has to scale individually before you do it. What I see is that the topology really reflects the company structure, of course. It's not "what has to scale separately", it's "team Y is working on X, and team Y does not want to talk to team Z, so they will create a service to make sure they don't have to talk to people". Except that this is a giant self-own. We all still have to talk to each other, like, a LOT, because things just keep breaking all the time and no one knows why. Dropbox, Instagram, StackOverflow - these companies are largely monoliths to this day. You thinking that your small outfit needs to be like Google is highly arrogant. And don't get me started on the amount of money, people, CPU cycles, and CO2 emissions wasted on this solving of the problem most people don't have.

daxfohl 5y ago

For me it's the repetition in horizontal tasks. Need to update TLS version? Need to move to a new region? Need to add i18n? You have to do these things 20 times instead of one.

Sohcahtoa82 5y ago

My thoughts on microservices... They're fine. But what's NOT fine are nanoservices. I did security on a project once where it seemed that every function was its own microservice. User registration, user login, and password resetting were each a separate microservice. It was an utter nightmare.

coding123 5y ago
Did they do this? user-registration.api.dev.my-company.com user-login.api.dev.my-company.com password-reset.api.dev.my-company.com or paths by some huge k8s nginx ingress?
- Sohcahtoa82 5y ago
  They started with the latter, but started shifting towards the former.
jeffbee 5y ago
Shoot, that's practically a monolith compared to the nightmare from which I resigned earlier this year. No way would they have allowed the entire password reset to reside in a single service. It would have been in the password-db-reader service, the password-db-mutator service, and the password-checker-service, among others. Absolutely terrifying and unmanageable.
- spmurrayzzz 5y ago
  This is the flavor of reductio ad absurdum example I always give when people hype up the "micro" part of microservices. Only meant to be illustrative— I had no idea that any org was actually doing that sort of nonsense, yikes.
zkldi 5y ago
Reminds me of this classic: https://www.youtube.com/watch?v=y8OnoxKotPQ https://www.youtube.com/watch?v=y8OnoxKotPQ

choeger 5y ago

If I was CTO of a company, microservices would give me nightmares. How do you do due diligence on used free software (licenses and security updates)? How do you plan the resource usage of your whole setup if every developer can add a new autoscaling service? Who is actually keeping track on deployments so we don't accidentally overload the system? How do you refactor a cross-service feature consistently? And the worst part: Who keeps track of the n*n contracts between the services? I mean yes, I know that each of these problems can be solved, sometimes in a relatively straightforward manner. But who really has all these aspects covered and doesn't run some services that started to smell weirdly a couple of months ago?

clintonb 5y ago
> How do you do due diligence on used free software (licenses and security updates)? Use the same process you would use if you had a monolith. The rest of your issues can be solved by planning out your services, rather than giving everyone free reign to make a new service. Switching to services doesn't magically mean your teams stop talking and designing together.
- choeger 5y ago
  Yeah and the inter-service specification lives where? How is it monitored, tested, enforced? I have the feeling that with the (quite possible!) addition of an inter-service codebase we would end up with a distributed monolith, i.e., a program that doesn't target a single computer but a particular substrate. I don't know whether that's a good design, though. Benefits: The program becomes more transparent and resilient to nonfunctional problems. It is also much easier to replace parts of the program. Downsides: Executing on a developer's workstation (critical for productivity and quality, IMO) might become harder. Efficiency gets reduced by orders of magnitude in certain spots.

maxdo 5y ago

Yet another “based on my experience” opinion. Based on my experience our service would fail due to performance because we’ll , nodejs is still single thread. Given this we should either duplicate deployment of big service by roles that gives you same level of orchestrating complexity or rewrite in different languages, means hello microservices again :) ps our product was initially written by non tech cofounders that used heroku and microservices from day one. They used a swarm of small services to stay on free tier. So microservices in this use case are cheaper. And yes, it's simpler for non experience developer to get up to speed with your backend if it's a simple service.

SomeCallMeTim 5y ago
Node can run multi-process with the Cluster module. Node being written as a monolith but running it in 100 instances is also an option. Roles can be implemented in software. There's much less "orchestrating complexity" when you're deploying a single service. So Node being single-threaded is not itself a reason to use microservices. I tend toward writing a monolith for the core API of a service, but then break out microservices for tasks that need to scale independently (or that need to run on high-memory/high-performance instances, for example). So I'm not totally against using microservices. But we should choose to use them when they're to our advantage to use them, not just "because they're already written that way."
- maxdo 5y ago
  If you deploy same binary under different roles, it's same issue with complexity. With modern tooling deployment and managed storages is not a problem at all you use templates or buildbacks or even lambda combined with gitlab github CI abilities. Recent progress allows you to embrace zero-ops and microservices. In my team we don't have a dedicated ops person, and deployment from dev to prod is running via git by developer. Node itself has a very bad profiling tooling compares to more "adult" languages. if you run a microservice it's much easier to spot a problem in CPU or especially memory leak. And vise versa if you doing something in scala, or java, microservices benefits are minor. It's also so much easier later to rewrite some of the services in a more performant language.

daxfohl 5y ago

I was going to say that one nice aspect is that rollbacks are less frequent because the size of each deployment is smaller. But I've worked on million line code bases that still do weekly CICD pretty well, so I don't know if that advantage really holds water.

bob1029 5y ago

Start with a monolith and only break it up if you are actually forced to by the computer (i.e. the process won't fit in ram anymore or eats all the CPU/IO). The second you break up your monolith, you lose its most powerful feature - The direct method invocation. The amount of time I see developers spending on JSON wire protocols, CORS problems, API endpoint designs, et. al. really is starting to concern me. I sometimes wonder if anyone wants to do any actual work or if this is just a big game to some people. I did the full trip on this microservices rollercoaster. Monolith => uServices => Monolith. I used to vehemently advocate for using microservices because of how easy it would be to segment all the concerns into happy little buckets individuals could own. We used to sell our product to our customers as having a "microservices oriented architecture" as if that was magically going to solve all of our problems or was otherwise some inherent feature that our customers would be expected to care about. All this stuff really did for us is cause all of our current customers to walk away and force a re-evaluation of our desire to do business in this market. All the fun/shiny technology conversations and ideas instantly evaporated into a cloud of reality. We are back on the right track. Hardcore monorepo/monolith design zealotry has recovered our ship. We are focused on the business and customers again. The sense of relief as we deprecated our final service-to-service JSON API/controllers was immense. No more checking 10 different piles of logs or pulling out wireshark to figure out what the fuck is happening in-between 2 different code piles at arbitrary versions.

jspash 5y ago
Here! here! I'm within spitting distance of the end of a project of collapsing a service-oriented system back into a Majestic Monolith. Every step of the way has reduced the lines of code, fixed bugs, saved money, saved time. It's been such a joy that I'm considering doing only this as a side hustle. "Saving" companies who were sold an over-complicated dream.
- bob1029 5y ago
  > It's been such a joy that I'm considering doing only this as a side hustle. "Saving" companies who were sold an over-complicated dream. This has crossed my mind a lot lately. I think we are looking directly at one of the largest emerging markets in technology. What do we think the TAM is going to be for undoing webscale monstrosities by 2025? Not every business will fail due to their poor technology choices and will be able to pay some serious consulting fees... I've practically got a system for doing this now. It mostly starts with domain modeling in excel and all the business stakeholders being in the loop at the same time until everyone agrees. I find if you get this part right it doesn't really matter if you use C# vs python, or AWS vs on-prem to build the actual product. Hard to get opinionated and locked in when your deploy to prod involves a 100 meg zip file and 3 lines of powershell ran against a single box.

kakoni 5y ago

"microservices is a psyop by big tech to make deploying & maintaining software so insanely difficult that future potential competitors are too tied up trying to keep the cloud equivalent of "hello world" afloat to present any real threat" https://twitter.com/htmx_org/status/1462636513578135556 https://twitter.com/htmx_org/status/1462636513578135556