33 ms·
Goodbye Microservices: From 100s of problem children to 1 superstar
- iamleppert 8y agoMicroservices are an organizational and design choice that is intended to mirror the structure of the teams or engineers, the actual human beings doing this stuff. It seems like Segment didn’t really understand this at all, and instead decided to have seemingly arbitrary and rediculous service boundaries that had no relationship to the real world. See also people who create poor abstractions in their code and other sources of technical debt. This isn’t an article about how microservices architecture is somehow bad, it’s an article about how bad Segment’s engineering team is. You can make the same argument for anything. At the end of the day some architecture or process or programming language or tech can’t replace real thinking about your problem and correct application.
- halayli 8y agoNothing you said addresses the issues they mentioned: > In early 2017 we reached a tipping point with a core piece of Segment’s product. It seemed as if we were falling from the microservices tree, hitting every branch on the way down. Instead of enabling us to move faster, the small team found themselves mired in exploding complexity. Essential benefits of this architecture became burdens. As our velocity plummeted, our defect rate exploded.
- WaxProlix 8y agoThat doesn't really say anything though: "Things were bad, and it felt out of control, because of microservices. They didn't work, and actually made things worse and more complex. The things that were supposed to be good were bad. We didn't do good work, we did bad work, and slowly." How do you address something like that coherently? I sort of agree with the OP, it kinda sounds like they just didn't know what they were doing. Or maybe that quote wasn't really the meat of their complaint?
- fastball 8y agoiamleppert does address that. As they said, microservices are supposed to mirror the structure of your team(s). A single team should never be maintaining 140 different microservices. That's not a failing of the microservices architecture, that is a failing of massively abusing the microservices architecture. Arguably, a small team should never be managing more than 10 services (totally arbitrary number, but it seems like the upper limit to what a human can focus on).
- sulam 8y agoThe important part here, that the parent addressed, is the statement "small team", as an organizational structure distinct from, say, a "large team" which has the opportunity to experience Conway's law. It sounds like Segment very much prematurely optimized. Microservices is probably not a great approach below 20 developers, and becomes necessary past 100, at least in my experience.
- maccam94 8y agoThey used microservices to paper over poor testing hygiene and software architecture. Forking one service into a bunch of different flavors of the same service multiplied the maintenance complexity. Services/microservices that are tightly coupled to shared libraries with lots of surface area are an anti-pattern. Instead, I would make the shared libraries into a service, and make a standard interface for the destinations as plugins.
- noobtubepython 8y agohttps://en.wikipedia.org/wiki/List_of_burn_centers_in_the_United_States https://en.wikipedia.org/wiki/List_of_burn_centers_in_the_Un...
- some_account 8y agoI don't they think are worse than anyone else. It takes courage to write about failures and micro services are a complex thing to get right.
- tclancy 8y agoIt would be fascinating to see this posted 6 months from now but with a man's name and picture on the byline and see if the comments are so negative.
- quickthrower2 8y agoI doubt it would make any difference.
- otterley 8y agoAs a principal engineer at Segment with over 20 years’ experience at successful companies large and small, I can personally attest to the fact that our team is incredibly talented and thoughtful. We make our engineering decisions very carefully and are highly evidence driven. Our choices are highly pragmatic and we investigate and analyze the trade-offs whenever we make a significant change such as this. I’m proud of what we’re doing. You can quibble with our implementations, but at the end of the day, the proof of the engineering is in the working. Our system works even better than before and our customers appreciate it!
- deleted 8y ago[deleted]
- stanmancan 8y agoOP can correct me if I'm wrong, but I believe the comment around Segements engineering team was most likely referencing the original decision to create hundreds of microservices in the first place.
- collinvandyck76 8y ago> it’s an article about how bad Segment’s engineering team is That's totally unnecessary and reflects poorly on you for having said so.
- fipple 8y agoYes, this awful engineering team built an incredibly successful company/product. I wish I was as awful as them.
- stanmancan 8y agoNot here to comment on Segments engineering prowness, I have no clue one way or another. However; I think an overwhelming number of engineers and developers are mislead in believing that you have to have the perfectly crafted program/service/app/whatever in order for it to be successful. It doesn't! Your tech stack has next to no impact on the success of your product. Making something useful is really all that matters. Is the service running? Great, that's all the end user cares about. The back end could be a bunch of code copy-pasted off old form posts from 1998 and string together with scotch tape as long as it works. The end user doesn't care about what language you use. They don't care if you use redis, mysql, postgres, or mssql 2000. They don't care if the HTML or CSS is messy. They won't judge you if you don't use SASS or LESS or whatever is popular today. The most successful side project of mine runs on a $20/m VPS and has been re-written 3 times and the code has been pretty terrible every time because every re-write was done in a new language I learnt over a weekend and never touched again. The first was some horrible PHP, then some less horrible PHP, then GO, and recently someone re-write it (properly) in elixir. The end user doesn't know, the end user doesn't care. The site gets about 250K uniques a month and the API handles about 350 million requests a month. The crappy GO I wrote in a weekend worked just the same for the end user as the fancy elixir. Anyways, sorry for the rant. I've just seen so much wasted effort go on projects that never launch because people are too concerned about stuff that really doesn't matter in the long run. If your stuff runs like crap because it found a market fit and you have people signing up and your crappy code can't handle it, then that is a FANTASTIC problem you now get to solve.
- modarts 8y ago> This isn’t an article about how microservices architecture is somehow bad, it’s an article about how bad Segment’s engineering team is. Wow. This is the level of discourse we've reached here now. Why the ad hominem? If you have an actual point to make on technical merit, by all means, make it. Were you passed on in a job interview at Segment? Why the hostility? (honest question)
- alexrbarlow 8y agoI'm not sure what they've been left with is a monolith after all. I would say they just have a new service, which is the size of what they should have originally attempted before splitting. In particular, as to their original problem, the shared library seems to be the main source of pain and that isn't technically solved by a monolith, along with not following the basic rule of services "put together first, split later". I feel prematurely splitting services like that is bound to have issues unless they have 100 developers for 100 services. The claim of "1 superstar" is misleading too, this service doesn't include the logic for their API, Admin, Billing, User storage etc etc, it's still a service, one of a few that make up Segment in totality.
- athenot 8y agoAgreed. Reading about their setup and comparing with some truly large scale services I work with, I'm left with the idea that Segment's service is roughly the size of one microservice on our end. Perhaps the takeaway is don't go overboard with fragmenting services when they conceptually fulfill the same business role. And regardless of the architecture of the system, there are hard state problems to deal with in association with service availability.
- tomnipotent 8y ago> Segment's service is roughly the size of one microservice on our end. This gave me a good chuckle. Dunning–Kruger in full effect.
- testplzignore 8y agoThe most telling fact is that it "took milliseconds to complete running the tests for all 140+ of our destinations". I've never worked on a single service whose tests ran that fast, given that the time spent by the overhead of the test framework and any other one-time initialization can take a few seconds just itself. It's great to have tests that run fast, but that's a bit ridiculous. Some rules of thumb I just came up with: Number of repos should not exceed number of developers. Number of tests divided by number of developers should be at least 100. Number of lines of code divided by number of repos should be at least 5000. Your tests should not run faster than the time it takes to read this sentence. A single person should not be able to memorize the entire contents of a single repo, unless that person is Rain Man.
- thermodynthrway 8y agoI've always said if the Linux kernel can be a giant monolith, in C no less, than there's maybe 100 web applications in the world that need to be split into multiple services. I've worked with microservices a lot. It's a never-ending nightmare. You push data consistency concerns out of the database and between service boundaries. Fanning out one big service in parallel with a matching scalable DB is by far the most sane way to build things.
- madrox 8y agoMicroservices aren't some magic bullet for scaling. If anything, they conform to Conway's Law [1]. I'd agree, though, if a single engineer is singularly responsible for 2+ microservices that are only supporting a single product...you're doing it wrong. [1] https://en.wikipedia.org/wiki/Conway%27s_law https://en.wikipedia.org/wiki/Conway%27s_law
- Marazan 8y agoI think that's possibly wrong as a key advantage of micro service over monolith is scaling. We have a 'single' product built as 2 micro services which is basically an external facing orchestration service that calls the compute heavy backend and it allows the external facing service to serve all the incoming traffic out of (on average) 4 containers whilst the backend computes with 32 containers and scales independently - very small changes to the incoming traffic can have large effects on the volume of traffic going to the compute service.
- davidkellis 8y ago> You push data consistency concerns out of the database and between service boundaries. Sing that from the rooftops. That is exactly my observation as well. All the vanilla "track some resource"-style webapps I've worked on were never designed to cope with a consistency boundary that spans across service boundaries. Turning a monolith into distributed services is hard for that reason - you have to redesign your data access to ensure that consistency boundaries don't span across multiple services. If you don't do that, then you have to learn to cope with eventual consistency; in my experience, most people just don't think that way. I know I have trouble with it. Surely I'm not the only one.
- justinzollars 8y agoI knew this day would come. We have come full circle. Also I really like this comment: > You push data consistency concerns out of the database and between service boundaries.
- topicseed 8y agoI think that with the rise in popularity of functions as a service (lambda, gcf, azure), we are heading more and more towards nanoservices. Small services are easier to develop with several teams, in my opinion. Each team knows what to input, and output. They can do whatever in between as long as these two contracts are respected. But the overlooking of all these moving pieces changing at different paces is tough. And the smaller the services get, the harder it will become. Not a size fits all, clearly!
- deleted 8y ago[deleted]
- xj9 8y ago> Reusable Reproducible Composable Software https://github.com/fractalide/fractalide/ https://github.com/fractalide/fractalide/
- madamelic 8y ago>I think that with the rise in popularity of functions as a service (lambda, gcf, azure), we are heading more and more towards nanoservices. There are some pretty big asterisks next to running "nanoservices". Mostly how expensive they actually are to run at large scale and the weird caveats that can happen due to them not always being up. And I wouldn't advocate for "always-up nanoservices". The basic answer to both "nanoservices" and microservices is do what you think is right but don't go too far. There are good reasons to make a nanoservice and good reasons not to, same with microservices.
- kraftman 8y agoSo they split everything apart because their tests were failing and they didn't want to spend time fixing them, and they they merged it back together by spending time fixing and improving their tests? It seems like the problem here was bad testing and micro repos, not microservices.
- dgritsko 8y agoI agree, and their conclusion even features this salient bit: >However, we weren’t set up to scale. We lacked the proper tooling for testing and deploying the microservices when bulk updates were needed. As a result, our developer productivity quickly declined. My impression after reading this post was that microservices were symptoms of problems in how their organization wasn't set up to implement them effectively, rather than the actual cause of those problems.
- deleted 8y ago[deleted]
- matwood 8y agoYeah, multiple services are going to be challenging if an organization has not already setup fully automated CI/CD pipeline(s).
- _5meq 8y agoAgreed. It's amazing to me how people still "meh" away testing as a secondary concern, and then regret it later. Over and over again. WRITING software is easy, anyone can do it. CHANGING software is extremely difficult. THAT is why we have tests. Also, if you are smart about it, you can get documentation out of the deal for relatively little additional cost. My go to example is on-boarding new developers: New dev: "OK, i'm here! How do I start?" with tests: Clone the repo, install dependencies, and run the test suite. As you develop new features, be sure to write additional test. They are up and going in a matter of minutes. without tests: Clone the repo, install deps, download testing database, achieve homeostasis with your dev environment, learn the entire system, build up the state you require to write your feature, iterate on it by hand over and over again.
- cnlwsu 8y agoThey took right approach. Don't treat it like a religion and do what works best for your team and project. There is no single perfect model so be willing to change if it makes sense.
- framebit 8y ago"2020 prediction: Monolithic applications will be back in style after people discover the drawbacks of distributed monolithic applications." -Kelsey Hightower on Twitter https://twitter.com/kelseyhightower/status/940259898331238402?lang=en https://twitter.com/kelseyhightower/status/94025989833123840...
- jspash 8y agoI'm waiting for the day that this becomes a standard refrain: "Javascript. What were we thinking?!"
- draw_down 8y agoLove when you guys manage to work in a dig at JavaScript while discussing a completely unrelated topic!!! Bet you think PHP is sometimes the best tool for the job and that people shouldn't be so quick to dismiss it, too. ;)
- chrisco255 8y agoBeen using JS for years and still loving it. The only thing I would move towards from here is some kind of ML like Elm, Reason, Haskell, etc. Certainly wouldn't go back to Java.
- romanovcode 8y agoCan't wait to write sane monolith with libraries instead of microservices.
- dpark 8y agoAm I understanding correctly that they had 3 engineers and >140 microservices? Microservices definitely have their own costs and tradeoffs, but 140 services and 3 engineers sounds like just a terrible engineering choice.
- ljm 8y agoI feel that when it comes to making any decision about separating things out, you really do have to consider the size of the team you've got before you commit to it. If you've got three devs and they all work across all of the microservices, then what on earth do they add except a hell of a lot of overhead? If you had ten teams of seven and they managed two services each... it's easier to see how that architecture could actually help. Same as if you have a two person team building a web-app and they go for client-server architecture rather than a basic full stack web framework. If you're both working the frontend and backend at the same time, save yourself the extra ops effort.
- paublyrne 8y agoI don't think so, the article says there were 3 engineers working just to keep the system up and running. I'm sure there were many more engineers than that working on and with those services.
- hobls 8y agoAgreed. "Micro" is a terribly defined term, and it sounds like this team went absolutely nuts in one direction. (And then in response to their problems, went as far back as possible to a single monolith.) This suggests a bit of a lack of nuance in their decision making process to me. I'm not interested in figuring out exactly what the right marketing term for it is, but I've had good experiences with teams of 6-10 engineers owning something like 2-5 services with a larger ecosystem of dozens to hundreds of services. Of course, I've been working at very large companies with extremely high traffic for several years now, so my experience is skewed in that direction. If I had three engineers on my team I'd be unlikely to end up with more than a small handful of services. Half the benefit of splitting up your services has to do with keeping your independent teams actually independent -- if it's just one team then that isn't a problem in the first place.
- badasstronaut 8y agoPersonally, I don’t like the terms ‘microservices’ or ‘nanoservices.’ What’s the value add in describing the relative size of the service? The _domain_ should drive what becomes its own service. Every service should handle the business logic within a particular domain. It’s definitely a goldilocks problem, though, in that there’s a too-small and too-large, and we’re looking for the just-right fit!
- jacquesm 8y agoThat's exactly it. But when you start doing 'microservices' you get the architecture astronauts who go and see into how many silly little services a monolith can be broken up. The end results are as predictable as the original monolith, both end up as an unmaintainable mess in a couple of years. I predict the same will happen to the 'superstar', it just isn't old enough yet (and at least it was built with some badly needed domain knowledge).
- l1ambda 8y agoMicroservices typically have two goals, performance and modularity. However, porting a typical webapp to a fast, modular compiled language will typically achieve at least one (often two) orders of magnitude performance improvement over a typical interpreted language. We are seeing this to be true more often than not, and a large performance gain off the bat like that may even obviate a lot of the desire to move to microservices. Furthermore, if one uses modules (as one should), one can arbitrarily and somewhat trivially run those modules either in-process (compiled in) or out-of-process (via REST, gRPC, Cap’n Proto, or another RPC system), e.g., in a separate service/microservice/whatever you want to call it. This gives you a best of both worlds approach where code can be arbitrarily run in a monolith or a separate service as-needed. This changes the thought dynamics from a rigid "monolith vs. microservice" decision to a more fluid process where things can be rather easily changed on a whim. When modularity is the goal, then services become something of a secondary concern. Microservices are used as something of a sledgehammer to force modularity and performance in languages that lack proper modularity and/or are innately slow or otherwise inefficient, while suffering orchestration costs and the performance penalties of copying data across multiple processes and networks as well as making it harder to derive a single-source of truth in some cases. Probably a good approach for a typical webappp looking to improve performance would be to first port core logic to a modern, fast, compiled language with modules, then evaluate the performance from there, and then determine if any modules should be split out into separate processes or services. Like NoSQL, microservices can be (but not always are) a case of the cure being worse than the disease; however, they can also be useful in certain situations or architectures. Like anything in engineering, there are tradeoffs and it depends on your situation.
- alexanderscott 8y agoThis misses the mark completely. A separate repo and service simply for different serialization and routing? Each performing the same basic function. That’s not a legitimate use-case for microservice separation and was doomed from the start.
- taeric 8y agoNot that I disagree with microservices easily going awry, the problem here seems to be traced to shared library code. Each microservice should be as standalone as possible. Your contract with that service is the service contract. Not some shared library. As soon as you have shared library, you now have coordinated deployments. And that is just not fun and will cause problems. The trick here is that this does mean you will duplicate things in different spots. But that duplication is there for a reason. It is literally done in two places. When you update the service, you have to do it in a backwards compatible way. And then you can follow with updates to the callers. This makes it obvious you will have a split fleet at some point, but it also means you can easily control it.
- XorNot 8y agoThis is deployment tooling related though - the ops part of devops really. If your build and test process doesn't actually exercise your deployment pipeline across versions, then it's not testing anything. I don't think shared libraries are a problem at all - they're probably a good idea - the problem is when they're used as an excuse to not worry about testing your upgrade and rollback scenarios. I'd pan that criticism a bit wider too: it's not just about having a testing process either - it's about making sure your devs are able to easily use it and watch it work as part of their regular cycle. This is in fact something I'm about to start working on at my new job for a new project - pushing the development of each service down so someone can write `make test` and not only run tests, but see if what they've done can upgrade between the currently deployed version.
- taeric 8y agoFair. But don't get to where you expect code to go out as a single deployment if it can't. Advanced tooling helps. So does a more transparent codebase.
- ec109685 8y agoThere is commonality across those 140 services that they extracted into a shared library so they weren’t copying that commonality 140 times.
- tracer4201 8y agoToo many smart folks that I've worked with, for some reason, just stop thinking critically when it comes to certain ideas. I was a product manager on a team of really rockstar developers. They all earn at least $200k a year. Instead of demanding more ambitious projects, you could keep about 99% of them happy by just letting them use the new framework of the week to build their next web app. Their excitement when they were green-lighted to use react was mind-boggling. Building another dumb website with the new framework, yay. Mindlessly applying microservice architecture is the same issue at heart.
- pythonaut_16 8y agoSounds like you really don't understand development if you think getting to use React isn't a big deal. React (and others like it, e.g. Vue) really is a huge win and solves a ton of pain points common to front end development. It still has its own pain points, but it's hard to overstate how much of an improvement it is over other older approaches like jQuery.
- basejumping 8y agoWhat does react have to do with jquery?
- pythonaut_16 8y agoThey're both tools used to build interactive websites. The parent didn't understand why developers would be happy that React got the greenlight. My response was that React (and Vue) really is a much better tool for building interactive websites than many older tools like jQuery.
- mockingbirdy 8y agoI think he doesn't have a problem with that. > Building another dumb website with the new framework, yay. He's just astonished how excited they are although what they build is "another dumb website" in his opinion. Many developers love the "how?" and don't care about the "why?". I don't judge it, but it can explain some of the excitement. Angular, React and Vue are great for development. But Ember, Backbone and others were also good back in their times. What he essentially says is "We use specific technology so they get excited although the problems we solve are boring af"
- the_arun 8y agoMicro Services solve for logical smaller teams, speed, deployment isolation & ton of other problems. Every solution comes with a trade off. There is no perfect solution. It is up to us to decide whether we need a monolith or micro service for our need & use case instead of comparing them.
- cortesoft 8y agoNot only that, but there is a lot of room between a 'monolith' and 'micro services'. How about medium services? You break somethings up and leave other things combined.
- sbov 8y agoSome people call this... microservices! Common advice is that a microservice should align with a bounded context in domain driven design, which can involve a LOT of code. Many large companies have millions of LOC behind their microservices. Your average startup probably doesn't.
- deleted 8y ago[deleted]
- ascendantlogic 8y agoWhat's old is new again. The inevitable rewinding of the 2012 architecture "revolution" starts now. I look forward to people in 2026 saying "monoliths are bad, split everything out to services"
- cirgue 8y ago"You may not be interested in the dialectic, but the dialectic is interested in you."
- jacquesm 8y ago100's of problem children sounds like a step too far. A few services > monolith monolith > 100's of services. The big trick with any technology is to apply it properly rather than dogmatically and if you are breaking up your monolith into a 100's(!) of microservices you are clearly not in control of your domain. That's a spaghetti of processes and connections between them rather than a spaghetti of code. Just as bad, just in a different way.
- matwood 8y agoGreat point. I like separate services, but would cringe at 100s of services in any system I have seen.
- spyspy 8y agoI can't believe this made it out of any planning meetings. Engineer 1: "We'll have one repo per downstream service." Engineer 2: "But we have hundreds of those, so now we have to manage hundreds of github repos???" Anyone sane: "That doesn't sound right, we should rethink this..."
- catchmeifyoucan 8y agoI wonder what language they're using? > if a bug is introduced in one destination that causes the service to crash, the service will crash for all destinations. This sounds dangerous. If I was a destination provider, and returned some garbage, could that take system down? Also, reading up on centrifuge, instead of breaking up by source, destination; would it be more spatially conservant to create queues by response type, which are finite. The entire time I was wondering why bad requests are put back into the same queue they came from. Shouldn't they be isolated? You could then treat those isolated requests as one unit, so in a destination outage, your not backlogged on your actual compute resources for good requests. Nonetheless, seems like just another day in an engineers workday :)
- guu 8y agoThe examples are written in JavaScript but it also looks like they use Go. Not sure which language this service is using specifically.
- woahthere123 8y agoThis particular destination service is written in JavaScript.
- SE4L 8y agoSeems to me the problem is the shared libraries. Yes, without sharing it means you have to repeat a fair amount of code, but in most cases the representation that each service cares about is not necessarily the same, which reduces the value of these shared libraries. It seems that they would have solved a lot of the really critical issues by simply not sharing as much code.
- hinkley 8y agoI maintained the shared library for five teams trying to move data around. The biggest challenge is making the shared library forward and backward compatible with itself for at least a few releases in either direction, because not everyone will redeploy at the exact same moment. If you can't solve that problem everything gets painful. Doing that right was the second hardest part of that job (meetings were the hardest). The job title (securing the data interchange) came in third place.
- artpepper2 8y agoOr use versioning and a package manager. You should be able to introduce vNext and then have each service update at its own pace, with a deprecation strategy so that service owners are responsible to get off the old version by such-and-such time. But as others mentioned, part of the problem seems to be too many services given the size of the team.
- e67f70028a46fba 8y agoAs with J2EE EJBs, microservices conflates two things: - a strong API between components - network calls The former is a very good idea that should be implemented widely in most code bases, especially as they mature, using techniques like modules and interfaces. The latter is incredibly powerful in some cases but comes at a huge cost in system complexity, performance and comprehensibility. It should be used sparingly.
- aniketpant 8y agoI probably don't understand their architecture well enough since they don't try and explain it in detail. The blame is also put on microservices whereas it's the architecture that can be fixed to improve performance and the cost of deployment. A shared library works up until a certain point. If every service uses a shared library then you are already getting to a world of a monorepo. A monorepo for different services should work fine if the overall architecture is feasible.
- adrianmsmith 8y agoSurely this is "no true scotsman"? They've tried microservices and it didn't work for them, you're saying that if only they'd "fixed" their micro-services architecture it would have worked?
- softwarefounder 8y agoI once scoffed at microservices. Then I was tasked with building a scalable platform around a message broker system. Then the switch clicked. I get it now.
- spyspy 8y ago> While our systems would automatically scale in response to increased load, the sudden increase in queue depth would outpace our ability to scale up, resulting in delays for the newest events. This strikes me as the core of their problem, and every step taken was a way to bandaid this limitation. Would the cost of moving to faster-scaling infrastructure been so high as rearchitecting the entire system? > When we wanted to deploy a change, we had to spend time fixing the broken test even if the changes had nothing to do with the initial change. This seems like a separate and even larger problem. Changes are breaking tests for unrelated code areas? Is the code too tightly coupled? Sounds like it. The unit tests are doing exactly what they're designed to do. Hard to feel sympathy for the person who's breaking them and then trying to figure out a way to sidestep them rather than fix the underlying issues.
- dasil003 8y agoThe job of these services is to transform their internal event format to 140 different output formats. You can imagine that there is a lot of duplication in the functionality that these services need to do. Are you suggesting that they avoid any shared libraries and just rewrite the same code over and over hundreds of times and update them independently?
- spyspy 8y agoIf making a change to a shared library breaks half the services, should it have been shared in the first place? It still smells odd to me there’s so much interdependence amongst the individual services break at will.
- dasil003 8y agoMaybe, but it’s conceivable to me that with 140 endpoints growing organically it would be very hard for an individual engineer to know for sure what should be abstracted or not. I think it’s a fundamentally hard problem even though on the surface it seems like it should be simple. Adding arbitrary new unaffiliated services is exactly the kind of thing that leads to irreducible complexity and a moving target that is very difficult to design for.
- Sir_Cmpwn 8y agoMicroservices aren't a magic bullet and won't save you from a poorly designed application. You already need to have very good separation of concerns within your application to make it out of microservices, and at that point an API that communicates over HTTP isn't much different from one that communicates over the application stack.
- ChicagoDave 8y agoI liked the rundown and agree there are tactical benefits for building monolithic API's, but one of the core tenants of microservice architecture coming out of the domain driven design space is that monolithic systems hide business logic. I'm pretty sure by moving everything back to a monolith, you've abandon DDD entirely. From a strategic perspective, that's bad for your business. If you get your DevOps ducks in order, a lot of the issues you have with standing up and maintaining individual microservices should be manageable. I certainly understand the pain of dependency management, but that's also a part of good architecture. I'm willing to hear more of these stories though. We can always learn about edge cases or even new paradigms that come out of current thinking.
- grosjona 8y agoIt's refreshing to read an article which challenges common wisdom. I've endured a lot of suffering at the hands of the microservices fan club. It's good to see reason finally prevail over rhetoric. It would have been nice if people had written articles like this 2 years ago but unfortunately, people with such good reasoning abilities would probably not have been able to find work back then. Software development rhetoric is like religion. If you're not on board you will be burned at the stake. So many times during technical discussions, I had to keep my mouth shut in the name of self-preservation.
- orcasauce 8y ago> So many times during technical discussions, I had to keep my mouth shut in the name of self-preservation. This sounds like an issue with being able to articulate why something is or isn't going to net the expected benefits or being able to foresee unexpected risks. Keeping silent is better than throwing out silly hyperbole risks, but not bring up real risks because "they don't want to hear it" is completely bogus. Any solid engineer will bite at another potential risk to ensure they don't find themselves engineered into a corner 65% through a project. Your comment also makes it out like the notion the article is making, monolith over microserves, is gospel for every situation; that in no condition would it ever make sense to use microserves and that only naive zealots would espouse the wisdom (dogma) to use them. You can use any piece of technology poorly, that doesn't mean the core concept is flawed, just that your problem space is different than what that software is trying to solve. Consider using HDFS as a primary data store in place of MySQL where it doesn't make sense and you might cry the wisdom of wishing someone had told you HDFS is terrible and to just use the tried and true MySQL of olden days.
- grosjona 8y ago>> This sounds like an issue with being able to articulate why something is... The issue is not articulation of ideas; the issue is that when all the books, all the articles and all people believe that something is true, there is no amount of articulation which will be able to convince them otherwise. You have to wait for the hype to go away before even considering bringing up the argument.
- FollowSteph3 8y agoMy view is that microservives is more of a workaround to company culctures and silos than a software solution. At least in most cases.
- jb3689 8y agoUse Erlang/Elixir. You get the benefits of the monolith and the benefits of the microservices
- atombender 8y agoPeople might read this article as an argument against microservices. It's not, in my opinion. It's an argument against impractical design. Their system should never have been 50 separate repos in the first place; in terms of design, it's all one app that would benefit hugely from being a single coordinated system. Microservices work when they are separate, independent, single-concern systems that coordinate using APIs. People often go overboard in splitting apps up into small pieces, even when those pieces logically belong to a single system. Start with figuring out the subsystems, then considering whether they are worth splitting in the first place. It's worth pointing out that microservices don't mean separate repos or even codebase separation. What matters the most is encapsulation. Monoliths grow horrible because they end up being balls of spaghetti, and forcing modularization at the service level is a way to avoid such messes by reducing the individual parts manageable sizes, allowing a part to be replaced without being concerned about its tendrils having grown through the whole system. For me, the biggest value of microservices is composition, of thinking abou modules as off-the-shelf components that you use as parts to build something bigger. Using a complete enough set of microservices, I can build a frontend or client that has zero app-specific backend code. For example, if I have a generic data layer (think Firebase), a user database layer with OAuth/OIDC, and a way to store images, then I can build Instagram from scratch with no backend development at all. That's very powerful. But once I need some specialized, app-specific stuff ("business rules"), such as rating of photos, commenting, moderation, etc., then those probably wouldn't be microservices! The concerns are unified there for the most part, and disentangling them would just lead to annoying fragmentation. A single use-case specific service ("monolith") that would exist at the center of it all. On the other hand, composition is mostly useful if your pieces are going to be reusable. If I intend to build more than one Instagram, or maybe a Facebook (which also needs data storage, and logins, and photos, etc.), then the individual pieces would be reusable and could just be shared between the apps. But if I'm just building Instagram for 5 years and I'm not building a series of apps for different use cases, reusability has zero importance, and I might as well just move everything into a single monorepo and forget about making anything general-purpose. (Each piece should be general-purpose enough, but they usually don't need to be so generic that you could open-source it for everyone.) I never liked the word "microservice", and I think we'd be better off if we called them, say, modules or subsystems.
- abalone 8y ago> Recall that the original motivation for separating each destination codebase into its own repo was to isolate test failures. This seems like a weird reason to adopt microservices. Can't you isolate tests within the same repo using folders?
- Diederich 8y agoIt's quite possible to do a monolithic app well and correctly. It's also quite possible to do a big pile of microservices as a cohesive app well and correctly. In my professional experience, where I've run into numerous examples of good and bad examples of both, microservices tend to win because it's a lot easier to unwind the badly implemented microservices compared to the badly implemented monolithic service. Of course, this is just another small set of anecdata.
- antishatter 8y agoSeems like an application of sturgeon's law, there is a tendency to do what is new because it is new. A well designed microservice architecture can be a tremendous boon for a product. A well designed monolith can be a tremendous boon for a product. About 90% of both are poorly done. It's interesting how often people favor ANY motion vs motion in the correct direction.
- scarface74 8y agoI don’t see any reason you shouldn’t always design a system as domain specific micro services. Now those micro services shouldn’t always be out of process modules that communicate over HTTP/queues, etc. A microservice can just as easily be separately compiled modules within a monolithic solution with different namespaces, public versus private classes, and communicate with each other in process. Then if you see that you need to share a “service” across teams/projects, or a module needs to be separately, deployed, scaled, it’s quite easy to separate out the service into versioned packages or a separate out of process service.
- gfodor 8y agoBefore you go down the path of splitting your app up into microservices you should grok erlang/BEAM/OTP. Lots of thinking went into its creation that leads to highly reliable real time systems and at the very least some of the ideas can be lifted in informing how to best design things. (But really, you should probably just use it instead.)
- qaq 8y agoOr just use Erlang/Elixir etc and you will have your microsrevices platform without k8s and most of the pain :)
- dexterdog 8y ago"microservices is the architecture du jour" ? There is nothing new about microservices. They've been a hot topic for far longer than Segment has been an idea.
- lostcolony 8y agoI'm hardly a microservice apologist, but sharing code or data across services is a major smell. It means these things are related, and should likely be bundled together. Don't just break a service apart because it's de rigueur.
- riknos314 8y agoWorking at a large tech company, we tend to approach microservice division as information authority division. If the data doesn't need to exist in the same database, put it in a new service with a separate database (or at least completely independent tables). Ideally there is no shared code between services (and there shouldn't need to be, because each service owns completely disparate data), so the only coupling is the API definitions. Each service is free to use whatever internal architecture they see fit as long as they honor the API definitions they provide to their dependent services. In the case outlined in the article, the fact that each microservice had similar enough concerns to all use some common libraries and the same database makes me doubt that this should ever have been built as microservices to begin with.
- acroback 8y agoThis is a classic case of not understanding micro services and trying to fit a problem around a tool. At work, we have close to ~50 services(no one calls them microservices), but they do not suffer from this brittleness. We segregate our services based on languages. So, all C services go under coco/ , all Java services go under jumanji/ , all go services go under goat/ , all JS services go under js/. This means, everytime you touch something under a repo, it affects everyone. You are forced to use existing code or improve it, or you risk breaking code for everyone else. What does this solve? This solves the fundamental problem a lot of leetcode/hackerrank monkeys miss, programming is a Social activity it is not a go into a cave and come out with a perfect solution in a month activity. More interaction among developers means Engineers are forced to account for trade offs. Software Engineering in its entirety is all about trade offs, unlike theoretical Comp Science. Anyway, this helps because as Engineers we must respect and account for other Engineers decisions. This methods helps tremendously to do this. No one complains, everyone who wants 1000 more microservices usually turns out to be a code monkey entangled in new fad, or who doesn't want to work with other Engineers. You want to use rust? There is a repo named fe2O3/, go on. Accountability and responsibility is on your shoulders now. If you think about it, an Engineer is tied to his tools, why not segregate repos at language level instead of some arbitrary boundary no one knows about in a dynamic ecosystem?
- jiveturkey 8y ago> This means, everytime you touch something under a repo, it affects everyone. This is horrible at scale. > This is a classic case of not understanding micro services and trying to fit a problem around a tool. That much I agree with. TFA even acknowledges that, in the conclusion. Not in so many words, but they basically admit they did it wrong.
- stickfigure 8y ago>> This means, everytime you touch something under a repo, it affects everyone. >This is horrible at scale. I just want to reiterate this. In the early 2000s I worked in the online platform group at EA. The list of things done poorly there was long, but picture: * 40+ engineers * Monorepo with hundreds of thousands of classes; all code deployed to all servers. * Hundreds of different services running across thousands of servers. * Communication based on Java serialization, so all code had to be deployed to all servers at the same time. * Deployments (and thus downtime) sometimes lasted up to a hour. Worldwide audience; it was always in the middle of someone's day. * Rational Clearcase for version control. It took nearly an hour to sync to tip. Pretty much every morning you'd come in, spend an hour syncing, find that someone broke the build, hunt them down, and resync for another hour. Generally speaking the first few hours of every day were wasted for 40 engineers. This was a very poor platform. Sometimes I wonder how it's going there these days.
- abc_lisper 8y agoAhh.. The sweet gong of the software hype pendulum
- dbt00 8y agoReading this post-mortem was very useful, and I appreciate the segment engineering team sharing it. It seems like the primary problem causes were flaky/unreliable tests, and difficulty making coordinated changes across many small repositories. Having worked on similar projects before (and currently), with a small team driving microservices oriented projects, I would probably recommend: 1) single repository to allow easy coordinated changes. 2) a build system that only runs tests that are downstream of the change you made (Bazel is my favorite here, but others exist). This means all services use the HEAD version of libraries, and you find out if a library change broke one of the services you didn't think about. This also allows for faster performance. 3) Emphasis on making tests reliable. Mock out external services, or if you must reach out to dependencies use conditional test execution, like golang's Skip or junit's Assume if you can't verify a working connection. If you still can't build a reliable service with those choices, then it's time to think about changing the architecture.
- ben509 8y agoA number of comments touched on this, that microservices are also an organizational strategy: you have a team that manages that particular service. My company is probably typical in that we implicitly use microservices because we have consume dozens of microservices provided by other companies. We only have a handful of services we maintain, each with dedicated engineers. And what becomes a service? You want to answer two questions: 1. Does it have a concrete business justification? 2. Does it have clear functional requirements? If it has a business justification, it will get people assigned to keeping it running. If it has clear functional requirements, it will make sense to the people working on it which service does what. That's still pretty vague, so you want to look at who has done it well, and why it worked for them. Companies like AWS have been extremely successful in using microservices because every last one service has a business rationale, it's either directly making money (EC2, S3, SQS) or it supports the needs of customers who are using a service that makes money (VPC, Cloudformation, all their internal auth, billing, provisioning, security and such). The caveat there is that the big B2B service providers are not a great model for companies that have a lot of business logic.
- trhway 8y ago>The first item on the list was to consolidate the now over 140 services into a single service. software engineering walks in circles. NoSQL people seem to be growing up and learning about consistency and transactionality. I'm waiting for somebody to discover threads and shared variables back as a revolutionary way to improve performance and greatly simplify the implementation of a system of "actors". Joking aside, i think greatest reason for Segment's microservices failure was that they didn't use Kubernetes.
- teilo 8y agoOne of the things that I like about the Erlang/OTP ecosystem is that it encourages you to design systems that are very modular, but does not force you to run every service as an independent entity. By designing around supervision trees of dependent services, one gains the advantages of a monolithic environment without losing the ability to split services out onto discrete nodes. But that's an "all in" environment. You are Elixir/Erlang/OTP, or you are out. That is, understandably, not an option for many use cases.
- StavrosK 8y agoLet me write a meta technology hype roadmap, so we can place these sorts of articles: * Old technology is deemed by people too troublesome or restrictive. * They come up with a new technology that has great long-term disadvantages, but is either easy to get started with short-term, or plays to people's ego about long-term prospects. * Everyone adopts this new technology and raves about how great it is now that they have just adopted it. * Some people warn that the technology is not supposed to be mainstream, but only for very specific use cases. They are labeled backwards dinosaurs, and they don't help their case by mentioning how they already tried that technology in the 60s and abandoned it. * Five years pass, people realize that the new technology wasn't actually great, as it either led to huge problems down the line that nobody could have foreseen (except the people who were yelling about them), or it ended up not being necessary as the company failed to become one of the ten largest in the world. * The people who used the technology start writing articles about how it's not actually not that great in the long term, and the hype abates. * Some proponents of the technology post about how "they used it wrong", which is everyone's entire damn point. * Everyone slowly goes back to the old technology, forgetting the new technology. * Now that everyone forgot why the new technology was bad, we're free to begin the cycle again.
- curun1r 8y agoWith hype always comes a throng of people who aren't capable of doing the necessary analytical thinking to solve their problem and who think they can substitute a magical pattern for designing their application instead of truly understanding their problem domain and building a real solution to that problem. Does that make it the fault of the technology/pattern? I don't think so. I think it just means that there's no magic bullets in tech and people who don't know what they're doing will always cause problems no matter what models they follow.
- JustSomeNobody 8y agoGreat summary! I wonder if you could fit Resume Driven Design in there somewhere. This should be the top post on articles like this. Just to keep it fresh always in everyone's mind.
- 8y ago
- tutfbhuf 8y agoIf you have 140 somewhat similar entities that all share common code, then they don't fullfill the very important microservice criterion of being independent. In your case, I would recommend to use a plugin based system. Do it the other way around, have 1 application that contains the common code (previously shared library code) and create 140 plugins. This way you can update the single application, load all plugins, execute the tests, check if everything is fine and deploy the application. Every plugin can live in its own repository and can be versioned separately, but a new version can only be deployed if it works with the latest version of the application.
- kyberias 8y agoThis is the voice of an adult developer talking.
- r00tanon 8y agoThis coincides with my own experiences in the financial sector. Distributed computing is undoubtedly the way to scale, but the trick is making the distributed nature of the system completely invisible (or as much as possible) to the developers, the applications and the supporting staff. I have seen this phenomena of thinking that a large system, broken down into tiny parts, is somehow easier to manage time and time again over 30 years of development. In every case, the one thing the central thinkers fail to realize is that complexity is like conservation of energy - it can be transformed, but it cannot be destroyed. Also, when it comes to large teams I have seen one thing work when it comes to sharing a resource(s) critical to a larger system - shared pain. If the central/reusable code/service breaks everyone's stuff, then everyone forms a team to immediately address the problem before continuing on. The solution is almost never "find a way to let the other teams continue while something important is on fire." It seems like a major motivation for a microservices architecture seeks to avoid the pain - which perhaps is not the best reason to use microservices. I like the idea of unseen, but indispensable, complexity. For instance, the human brain is probably the most complex thing in the world, but the interface is fairly simple :)
- deleted 8y ago[deleted]
- verletx64 8y agoWhat's wrong with a circuit breaker?
- perlgeek 8y agoTwo things come to mind: * splitting everything up into separate repos and services seems like a pretty radical move. You could start with separate queues that are handled by a single service, potentially with many instances, or you could try to break out a few parts that change often or have a very high load * A big chunk of the complexity seems to be in transforming one message format to another. That is something that should be very easy to write tests for. So you need a CI that tests all the services when you change the base libraries, and then it's pretty easy to find out if a change to a base library is backwards compatible or not. And for libraries that are shared between many services, you should mostly stick to backwards compatible changes.
- jaequery 8y agoSo they went from managing 100's of little parts to a just a single one and are loving it. Sounds like a no-brainer to me.
- twblalock 8y ago> Our initial microservice architecture worked for a time, solving the immediate performance issues in our pipeline by isolating the destinations from each other. However, we weren’t set up to scale. We lacked the proper tooling for testing and deploying the microservices when bulk updates were needed. As a result, our developer productivity quickly declined. Well, I'm not surprised. Microservices are made possible by advances in automated testing, CI, and deployment. You should have those things anyway, even if you have a monolith -- but to go to microservices without them is a pretty bad decision.
- acd 8y agoI have had thinkings on how efficient Micro services are compared to a Monolith. In a micro service architecture, the request bounces through different service layers, json serialization, network transfers. In a monthlith normal application, especially if its on one machine, the request touches main memory and then CPU cache. Latency Numbers Every Programmer Should Know, I especially think of the Send 1K bytes over 1 Gbps network vs cache latency of microservices/monolith. https://gist.github.com/jboner/2841832 https://gist.github.com/jboner/2841832 Ie instead of micro services 10000 ns for a network transfer you could fetch within 10ns from CPU cache in a monolith, that is 1000 times more efficient. Are we beyond the peak of inflated expectations on Micro services and towards the Plateau of productivity? https://en.wikipedia.org/wiki/Hype_cycle https://en.wikipedia.org/wiki/Hype_cycle
- dpeck 8y agoMicroservice based architecture will continue to be used quite a bit as many weaker engineering leads have latched onto it as a way to solve interpersonal problems on their team and letting everyone have their own little fiefdoms. It creates a terrible scenario and a very dysfunctional team, but the hype cycle and a few high scale orgs mentioning it means its part of the "mainstream" now and something that ill-informed leads will be pushing for the next decade, whether it makes sense for the specific context (it doesn't) or not.
- jpswade 8y agoHow on earth did you end up with 100s of services? I can't imagine you have applied Conway's law. I think there's some serious confusion between FaaS and microservice architecture.
- man2525 8y agoI'm thinking it's more of a software business life cycle. Microservices are somewhat easier to deploy, which makes it easy for developers to add capabilities to be used by UX developers. Once the eye candy and rough functionality is in place, the sales team signs up new clients. The company draws down investment in developers and puts more money into infrastructure. The product moves from developer land to system land which is monolithic and can handle heavier loads.
- stcredzero 8y agoWith every destination living in one service, our developer productivity substantially improved. We no longer had to deploy 140+ services for a change to one of the shared libraries. I say this as someone who worked in an environment widely castigated for its "monolithic" nature: It sounds like trying to have "modularity" hurt you, because you didn't actually have it in the first place. When you stopped pretending, the pain went away.
- kyberias 8y agoIt's as if these people just take a buzz-word and start applying without ANY consideration about basic stuff about distributed systems. Are there any adults around in these firms or what is going on?
- heme 8y ago... step back and embraced an approach that aligned well with our product requirements and needs of the team. This ^^^. Everyone has an opinion on the Internet... frontend, backend, no-end. It's probably not going to align perfectly with your needs (are you Google/Facebook?) . There are a lot of tools & specs out there. If your blindly following someone else's opinion without truly understanding your users, product, & team you'll probably accrue debt. Also, devs often forget the team. If your choice of tech cuts the team's velocity in half then it probably wasn't a good choice... even if it had some other technical benefits.
- ris 8y agoI'm not sure how worthwhile it is writing any more "microservices are dumb" articles - all the people who have spent the last 5 years leaving microservice messes in their wake appear to have moved on to creating "serverless" messes of lambda functions which people like you and me are going to be going around tidying up in about 5 years from now.
- jarfil 8y agoThe whole article reads like BS to me. So the initial problem was a single queue? Well, then split the queue, no need to go all crazy splitting all the code. Switching to 100+ microservices? There is no need to switch to 100+ repos too, runtime services don't need to have one repo per service, just use a modular approach, or even feature flags. 100+ microservices, some of them with much lower load than others? Then consolidate the lower load ones, no need to consolidate "all" of the microservices at once. Library inconsistencies between services? No, just no, always use the same library version for all services. Automate importing/updating the libraries if you need to. A single change breaks tests in a way you need to fix unrelated code? WTF, don't you have unit tests to ensure service boundary consistency and API contracts? Little motivation to clean up failing tests? Yeah... you're doing it wrong. Only then you figure out to record traffic for the tests? HUGE FACEPALM, that's the FIRST thing you should do when dealing with remote services!
- web007 8y ago> Library inconsistencies between services? No, just no, always use the same library version for all services. Automate importing/updating the libraries if you need to. This answer drove me crazy in the article. "We had trouble keeping the libraries up to date and fixing breakages, so our solution was... to update them all and fix the breakages." And per your second point, tests can go both ways - if libX is used by serviceY, write a libX integration test for serviceY instead of / in addition to a serviceY integration test for libX.
- thomasfromcdnjs 8y agoI must agree. Seems like every step could have been solved without jumping to the conclusion that it was because we didn't use a monolith. I've never understood the false dichotomy of microservices vs monolith... just split things when it makes sense. ¯\_(ツ)_/¯
- Cthulhu_ 8y ago"when it makes sense" isn't a science though; for them it made sense at the time, like how now it makes sense to move back to a monolith.
- ninjakeyboard 8y agoI've heard Jamie Alan talk about copying shared code into the services. I think shared libraries are an antipattern.
- lagadu 8y agoInteresting read, my biggest complaint is: confusing unit testing with integration and system testing. On every build you only run unit testing, you don't rely on any external dependencies. If your test suite relies on querying external servers, you're not testing your software, you're testing your server, your internet connection, your credentials, their internet connection and their servers. That's insane to run on every build! That kind of system testing should only happen whenever you're preparing to deploy a release which isn't something that happens everyday because of how hard it is to properly do system testing. edit: also it seems they're missing a solid release cycle and manager. For every release before going to production there's a TON of testing that needs to be conducted, they're mentioned over 40 improvements this year, that's two releases a week. It's not possible that you're properly testing each of those releases. Had they done a single, well tested release with multiple fixes every 1-2 months, the burden would be significantly reduced.
- dasil003 8y agoTerminology aside, why would you not automate all your testing so you can release more frequently? We're not talking about a martian lander here; for a SaaS startup velocity is much more important than perfect releases, you will have more success by optimizing and de-risking for very frequent releases.
- grizzles 8y ago+ -js -java -monolith.jar {guava, netty, etc } -svc1 -svc2 -python -go For biz code, I've seen this kind of lib-ifying architecture provide a nice microservices workflow. The unifying heuristic is: Any dependency goes into the lib project. Anything unique to the service that requires no dep should usually go into the service project. The nice thing about this is it modular with fast compiles and preserves optionality. Since everyone is using the same deps, the code can live in the svc project or the lib project. A bit of namespacing convention makes it's trivial to shuttle the code between the two projects to wherever it's most natural to have it.
- theptip 8y agoI can't help but think that this is an experience report covering "microservices-done-wrong considered harmful". There are definitely pitfalls to the microservice approach, but there are canonical answers to all of these issues. That said, the right approach is the one that fits with your team's disposition, skill-set, and experience level, so ditching microservices might well have been the correct choice in this case. > Eventually, all of them were using different versions of these shared libraries. We could’ve built tools to automate rolling out changes, but at this point, not only was developer productivity suffering but we began to encounter other issues with the microservice architecture. In the extreme case where you have 150 microservices and 3 devs, I think that spending a few days to build a tool that auto-updates your common deps and re-runs your tests would be a good investment. Or you could pay someone else to do this with a service like https://www.dependencies.io/ https://www.dependencies.io/. Handling common code is one of the known pain points in microservices, so it's worth tackling head-on. (Last I saw Netflix handles this by the rule "no services can share code unless it's by an open source library", which encourages common code to be thoughtfully packaged and released.) > The additional problem is that each service had a distinct load pattern. Some services would handle a handful of events per day while others handled thousands of events per second. For destinations that handled a small number of events, an operator would have to manually scale the service up to meet demand whenever there was an unexpected spike in load. I can see this being tricky to tune, and am hesitant to opine without knowing the details, but if you can fix the problem by bundling all the services into a single monolith (i.e. aggregating all load into N nodes), then you should also be able to fix the problem by using a cluster scheduler like k8s with equivalently sized nodes. As long as you don't have bursts that are an integer factor of your baseline system load, both approaches should work equivalently (to the first order). It sounds like they were running individual instance(s) per microservice, which isn't a good fit for very bursty services. And as a bonus, with a cluster scheduler you get a number of primitives to do resource reservation, which you don't get for free if you're merging all of your services back inside a single monolith. This means the problem of back-pressure from a single misbehaving endpoint -- which was one of the reasons they moved to microservices in the first place -- will probably come up in some form down the road. > Recall that the original motivation for separating each destination codebase into its own repo was to isolate test failures. However, it turned out this was a false advantage. Tests that made HTTP requests were still failing with some frequency. With destinations separated into their own repos, there was little motivation to clean up failing tests. This poor hygiene led to a constant source of frustrating technical debt. I don't have anything to say here except... don't do this? If your tests are failing you should be fixing your tests (or removing them if they aren't adding value), not adding new integrations. Consistently-failing tests are a big warning sign that your CI/CD process is not in good shape, and a healthy CI/CD process is a strict precursor to doing microservices successfully. > The outbound HTTP requests to destination endpoints during the test run was the primary cause of failing tests. Unrelated issues like expired credentials shouldn’t fail tests. Your UTs shouldn't be hitting your external dependencies; the correct solution here is the one that they eventually landed on, i.e. to either record/replay real HTTP requests, or to mock out the HTTP responses manually. You still need real integration and smoke tests in a production-like environment to make sure that you've not missed a change in the remote API schema. IME this is one of the biggest challenges of working with external APIs, and I don't envy the task of maintaining 150 integrations, however this issue seems unrelated to microservices.
- kungfooguru 8y ago"With everything running in a monolith, if a bug is introduced in one destination that causes the service to crash, the service will crash for all destinations." The real reason people should investigate Erlang/Elixir.
- crimsonalucard 8y agoWe should bring Categorical concepts into microservices. Maintain compose-ability between entities rather than forming a graph of intercommunicating objects similar to OOP. I don't know if adhering to these principles will make microservices better.
- lafar6502 8y agokudos to people paying for all this mumbo-jumbo
- staticassertion 8y agoDidn't take long for "we used shared libraries, and then found out we couldn't deploy independently". Sounds like you weren't quite doing microservices?
- caoilte 8y agoquite.
- jskaggz 8y agoApril fools trolling experiment: recast something like perl5/cgi as the new backend web hotness with an edgy name and flashy tutorials. But does perl5 even need hype? Soon it'll age into retro-chic status, ala lisp. :)
- SerjEpatoff 8y ago>>Unless you’ve been living under a rock... Monolithic Scylla DB built on top of venerable Seastar framework is very good example of under-rock-living competitive advantages.
- sealthedeal 8y agosounds like a lack of Devops
- exabrial 8y agoThe thing people failed to realize is that microservices are a way of structuring people in teams in an organization, not a way of structuring a product architecture. To use words that are mine, microservices are a hack on Conway's law. One team should be responsible for 2-3 microservices and should have a lot of autonomy.
- sciurus 8y agoI like that they highlight the tradeoffs between isolating faults and optimizing resource utilization. However, you don't need microservices to achieve isolation. You can have different worker pools of the monolith configured to handle different destinations. The monolith actually gives you more flexibility in how you approach the tradeoff. With microservices you're forced into one pool of workers per destination, but with the monolith you can choose any mapping of pools to destinations that makes sense.
- akamaozu 8y agoGetting the feeling Segment didn't really stop using microservices. Sounds like they just redrew their service boundary, from Integration APIs to Business Function. Centrifuge sounds like a new service deals with connecting to integration APIs, so they've replaced 140+ services with one. Another service they've spun up is Traffic Recorder, and its responsibility is to eliminate the need for http requests when testing integrations. Feels like the biggest change is going from so many repos to a monorepo.
- woahthere123 8y agoSegment is very much built around microservices. They just consolidated 100+ of these destination services into a single one. That's it.
- akamaozu 8y agoGlad you think so too. The title and opening paragraphs gave me the impression they felt they were moving away from microservices, but maybe I didn't those bits correctly.
- exclusiv 8y agoI found the frustrations with microservices too setting up a pretty substantial ETL system using Amazon Lambda among other services. Originally you couldn't trigger Lambda functions from SQS (seemingly the most obvious integration). You could use Kinesis but small print says Lambda concurrency is restricted to the Kinesis streams which gets very expensive. Visibility/monitoring into most microservices is not good (Iron.io is quite nice but any concurrency is really expensive). I don't like the workflow for deployment and testing either. So I shifted to single EC2 instance with my application and Beanstalkd w/ my own configurable workers. Way cheaper, easier to manage, normal programming workflow, etc. For some use-cases Lambda and other services are really nice and efficient but there's usually a lot of hidden limitations so be sure to spend a lot of time evaluating before committing. You often spend way more time fighting the microservice drawbacks than the benefits are worth.
- nikon 8y ago> Once the code for all destinations lived in a single repo, they could be merged into a single service. With every destination living in one service, our developer productivity substantially improved. We no longer had to deploy 140+ services for a change to one of the shared libraries. One engineer can deploy the service in a matter of minutes. ... 140 tightly coupled services wasn't the problem then?
- cestith 8y agoIf you're using microservices why are you using shared libraries? Wouldn't it make sense to break the common portion into a separate service and define an API for it? I swear more people writing microservices need to read about flow-based programming and perhaps the actor model.
- devquixote 8y ago"We no longer had to deploy 140+ services for a change to one of the shared libraries." I would be curious as to what the focus and scope of these shared libraries were such that they required frequent updates with cascading side-effects requiring everything that leverages them to have to be updated.
- f2prateek 8y agoMost of these libraries are open source. One we had the most friction updating was was our library that wraps common logic for dealing with user generated events - https://github.com/segmentio/facade https://github.com/segmentio/facade (this is the example that was referenced in the blog post).
- tybit 8y ago“Recall that the original motivation for separating each destination codebase into its own repo was to isolate test failures. However, it turned out this was a false advantage. ” At least they finally got to the right conclusion, they were on the wrong path to begin with. People seem to be blaming Microservices when they weren’t even close to understanding what they were doing and why. I’d be much more interested in an article about issues faced with Microservices where they actually tried to slice their functionality based on their domain.
- rvr_ 8y agoA 3-person team cannot manage 100+ repos with 100+ micro services. Sounds like a big impedance mismatch.
- khalilravanna 8y agoThis might come off as being snide, but I'm genuinely curious: was their solution really just having all the services in one repository? That doesn't seem like a problem with microservices at all but more of a devops problem. To be clear, I'm not arguing for microservices, I'm just trying to understand if this was really a problem with splitting off multiple repos. Maybe I'm just really dense and someone can set me straight. I've actually had experiences with seemingly this same problem at a previous startup. Once we started spinning off individual repositories for small pieces of business logic stuff started to go downhill as the logistics of communicating and sharing one another's code became more and more complex.
- avarun 8y agoIt seems like splitting off small repos for everything is a solution looking for a problem. Some of the most successful software companies out there have monolithic repos, but not monolithic services. The real solution is microservices in monorepos.
- bennettlp 8y agoThis debate makes me want to punch myself in the face, right approach for the right problems.
- eximius 8y ago> The shared libraries made building new destinations quick. The familiarity brought by a uniform set of shared functionality made maintenance less of a headache. > However, a new problem began to arise. Testing and deploying changes to these shared libraries impacted all of our destinations. It began to require considerable time and effort to maintain. Making changes to improve our libraries, knowing we’d have to test and deploy dozens of services, was a risky proposition. When pressed for time, engineers would only include the updated versions of these libraries on a single destination’s codebase. When pressed for time, engineers would only include the updated versions of these libraries on a single destination’s codebase. I think I see the problem and it wasn't with microservices.
- ahallock 8y agoI don't think you reverted to a monolith. I think you built a singular service with more cohesion than when it was split into 140 repos. And it sounds like the split wasn't because of the microservices architecture but because you didn't have a good solution for the queue problem.
- wedgeantilles 8y agoEverything is cyclical.
- skyisblue 8y agoMicroservices benefits large teams. There’s just too much operational overhead for a small team to manage microservices. Large teams with monolith applications benefit from microservices as the communication issue of large teams is reduced by splitting the team into cross functional teams.
- philippz 8y agoThis is what we did at STOMT at day one. For those who are interested in how to accomplish this with PHP / Laravel / Lumen: https://www.stomt.com/blog/shared-components-across-multiple-laravel-lumen-micro-services/ https://www.stomt.com/blog/shared-components-across-multiple...
- mnm1 8y ago3 full time engineers. No wonder they crashed and burned with microservices. Even a slightly distributed system, what I call miniservices, with an api and login server and half a dozen regular apps on top is way too much for such a small team. In a similar, but much smaller setup, I estimate at one point our two person team was spending as much as 25% of their time on cross-service (between the app and api / login server) concerns that would simply not exist with a monolith or something not distributed. That's 3 months out of every year of development time wasted on concerns that shouldn't even exist.
- imajes 8y ago- All teams will henceforth expose their data and functionality through service interfaces. - Teams must communicate with each other through these interfaces. - There will be no other form of inter-process communication allowed: no direct linking, no direct reads of another team’s data store, no shared-memory model, no back-doors whatsoever. The only communication allowed is via service interface calls over the network. - It doesn’t matter what technology they use. - All service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world. No exceptions. - Anyone who doesn’t do this will be fired. Thank you; have a nice day! still kinda works.
- luord 8y agoProbably off-topic, even if it's about the article: they should've never relied on doing requests for the tests, specially unit tests. Isn't the recommended practice to treat third-party services and libraries like a black box? That's what I do, and that's how I figured what their solution was (roughly) before I read it. Felt a bit proud of myself.
- sequoia 8y agoIt seems like their initial problems were: 1. Tests hitting 3rd party APIs are flakey & slow 2. The job queuing mechanism can cause all jobs to be slowed by a single 3rd party API outage/slowdown Eventually they arrived at: 1. Replay responses to speed up HTTP based tests 2. Create a smarter queuing mechanism in house I'm not sure what microservices has to do with any of this. Anyway, kudos to them for having the courage openness to share their learning from mistakes!
- mdekkers 8y agoIt appears to me that this isn't so much a case of "microservices don't work" as it is the case of several poor architectural choices. It is always really easy to be part of the crowd in the peanut gallery and say "hah, you are doing it wrong", but when actually at the coalface the pressures and perspectives are different. Having said that, the microservices story isn't half as interesting as the Centrifuge story they link to in the article. Their approach (queues for everything) didn't work for them at their scale, so they invented something different (Centrifuge) that does. Poorly thought-out choices around microservices has little to do with the issue they solved.
- Floegipoky 8y agoIt seems like splitting into separate repos was a rash response to low-value automated tests. If tests don't actually increase confidence in the correctness of the code they're negative value. Maybe they should have deleted or rewritten a bunch of tests instead. Which is what they did in the end anyway. >> A huge point of frustration was that a single broken test caused tests to fail across all destinations. When we wanted to deploy a change, we had to spend time fixing the broken test even if the changes had nothing to do with the initial change. In response to this problem, it was decided to break out the code for each destination into their own repos They also introduced tech debt and did not responsibly address it. The result was entirely predictable, and they ended up paying back this debt anyway when they switched back to a monorepo. >> When pressed for time, engineers would only include the updated versions of these libraries on a single destination’s codebase... Eventually, all of them were using different versions of these shared libraries. To summarize, it seems like they made some mistakes, microed their services in a knee-jerk attempt to alleviate the symptoms of the mistakes, realized microservices didn't fix their mistakes, finally addressed the mistakes, then wrote a blog post about microservices.
- stickfigure 8y agoit seems like they made some mistakes, microed their services in a knee-jerk attempt to alleviate the symptoms of the mistakes, realized microservices didn't fix their mistakes, finally addressed the mistakes, then wrote a blog post about microservices. That seems... appropriate? This is the general problem with the microservices bandwagon: Most of the people touting it have no idea when or why it's appropriate. I once had a newly hired director of engineering, two weeks into a very complicated codebase (which he spent nearly zero time looking at), ask me "Hey there's products here! How about a products microservice?" He was an idiot that didn't last another two months, but not before I (and the rest of the senior eng staff) quit. I'm fully prepared to upvote more stories with the outline of Microservices were sold as the answer! But they weren't.
- aidenn0 8y agoX was sold as the answer but wasn't is true for pretty much all X that have ever been sold as The Answer. It's almost like engineering techniques aren't magic pixie dust that you can sprinkle over your project and get amazing results...
- aimatt 8y agoI imagine that your development velocity with this monolith is enabled by the modularity of your code that was enforced as microservices. This benefit will wane and if you had started with a monolith would be much worse.
- aimatt 8y agoSomeone better tell Google to merge all their code back into one repo, haha
- _Codemonkeyism 8y ago"[...] the small team found themselves mired in exploding complexity." Key sentence. Using the wrong tool for a small team, blaming the tool.
- franzwong 8y agoGoing back from 100+ microservice to 1 monolith is another extreme to me...Perhaps you can have 10 macroservices. And shared library shouldn't contain business logic...
- romanovcode 8y ago> And shared library shouldn't contain business logic... What's the point then? They become libraries then, not services.
- franzwong 8y agoI would put the business logic inside the service rather than inside shared library. For shared library, I usually put utility functions.
- primeblue 8y agoWhy not use a package manager to publish a library for a destination? Then just release a package and reference it whenever there is a change to the target. Should work well in either a monolith or micro service.
- ChicagoDave 8y agoI’ve been tracking the comments and my sense is that almost no one here believes the business domain drives the technical solution. Microservices, when constructed from a well-designed model, provides a level of agility I’ve never seen in 33 years of software development. It also walls off change control between domains. My take from the Segment article is that they never modeled their business and just put services together using their best judgment on the fly. That’s the core reason for doing domain driven design. When you have a highly complex system, you should be focused on properly modeling your business. Then test this against UX, reporting, and throughput and build after you’ve identified the proper model. As for databases, there are complexities. Some microservices can be backed by a key-value store at a significantly lower cost, but some high-throughput services require a 12-cylinder relational database engine. The data store should match the needs of the service. One complexity of microservices I’ve seen is when real-time reporting is a requirement. This is the one thing that would make me balk at how I construct a service oriented architecture. See Eric Evans book and Vaughn Vernon’s follow up.
- eksemplar 8y agoWait till you get older. I’ve seen the hype swing from micro services to soa and back to micro services a few times already. I think the key is doing something that works, for you, and care a little less about what other people are doing. Most paradigms have their advantages and disadvantages and it’s really just about working around that to best utilize the stuff you have.
- smadge 8y ago>> 33 years of software development. > Wait till you get older. Presumably someone with 33 years of software development experience is around 50 years old. How old do you think someone needs to be before they are qualified to comment on trends in software development practices? 70 years old?
- some_account 8y agoHe is correct though. Not only software, but lots of things are cyclical and come back in different forms. And you do have to have a few decades of life experience to be part of that. I also would totally agree that companies should never just adapt "best practices" because those leads to super complex enterprise systems which are not necessary for most companies.
- dotdi 8y agoAnother article with the TL;DR similar to "we jumped blindly on the bandwagon, did obvious mistakes along the way, and microservices are the root of all evil". Thanks, but no thanks.
- pavbelshippable 8y agoAll software architects are just going gaga over this Microservices architecture. Microservices architecture is not for everyone. You need to consider a lot of factors before adopting this model in your system and organization. Everybody is only talking about the positive aspects of moving to microservices. Ask yourself, do you really need microservices architecture? To find out, you should ask yourself these questions: How independent do you want your services to be? How well your code is organized? Mono repo or multiple repositories? How much operational complexity can your team handle? Do you have a Continuous Delivery platform in place? How is your team organized? Do they have knowledge about different toolsets and microservices architecture? Then focus on the existing case studies like this and make an architectural strategy.
- spion 8y agoSeems like an instance of "The wrong abstraction" ? https://www.sandimetz.com/blog/2016/1/20/the-wrong-abstraction https://www.sandimetz.com/blog/2016/1/20/the-wrong-abstracti...
- andy_ppp 8y agoOh man, this is just the tip of the iceberg with Microservices. There is nothing Micro about them, they are so difficult to deal with that it becomes impossible to actually iterate or build user value and introduces loads of difficult to debug problems. We have architects here that dictate the design of the system but IMO they have not done the simplest implementation of anything. We have Kafka to provide ways of making each service eventually consistent so we delete something out of our domain service, but it requires absolutely huge amounts of code in various different other services to delete things in each place listening for events. Every feature is split across N different services which means N times more work + N times more difficult to debug + N times more difficult to deploy. The system has been designed with buzzwords in mind - Go and GRPC have been a disaster in terms of how quickly people have developed software (as has concourse - so many man hours wasted trying to run our own CI infrastructure it's unreal), loads of small services that are individually difficult to deploy and configure (and come with scary defaults like shared secret keys for auth - use a dev JWT on prod for example). The difficulty in dealing with debugging the system - there simply aren't the tools to understand what is going wrong or why - you have to build dashboards yourself and make your application resilient to services not existing. Never ever try to build Microservices before you know what your customers really want - we've spent the last 6 months building a really buggy CRUD app that doesn't even have C and D fully yet. Love your Monolith.
- organsnyder 8y agoUgh. Sounds like a severe case of resume-driven architecture. In my experience, you won't know whether you need microservices until you're on at least v2.0 of your application. By then, you have a better understanding of what your real problems are.
- andy_ppp 8y agoResume-driven architecture will benefit me assuming I every want to waste this much time writing a delete documents method (4 people on my team worked on it on and off for a month).
- Quarrelsome 8y agoMy question is why they thought it was a good idea to poke retries into the same queue when this was degrading their ability to process new items that were less likely to fail. Isn't that just an architectural fuck up? Surely the retries become a lower priority and get handled differently.
- akerro 8y agoMicroservices are damn hard to implement correctly. Think how much math and algorithms must there be to correctly, test, version your model and API, load balance, gather logs, collect logs, archive logs, grep in logs, setup VMs/Dockers, DNS+DHCP and firewalls, load balance databases (master/slaves?) and your services, horizontal scaling maybe?, manage privileges and access, manage disk space, store and collect user-uploaded/generated files, isolate environments... If you don't understand these concepts and how much work must be don't to correctly implement microservice architecture - you SHOULD STAY with monoliths, they are much easier. http://principlesofchaos.org/ http://principlesofchaos.org/
- sobani 8y agoAll of these things you mentioned are either also required for a monolith (like DNS+DHCP) or are required for neither (isolated environments). Does that mean that because of all the work required to properly set those up for a monolith, we better stay with microservices?
- LukeB42 8y agoSeems oversimplified to me. Not one mention of authentication and single-sign-on in the article.
- johnlbevan2 8y agoIt will be interesting to read your post in a few months time, when you hit the issues caused by using a monolith which could be resolved by splitting that service apart. Introducing components to match on various forms of value seems odd. Ideally standardise on those names; but if that's not possible (and you're using microservices), why not have a service to correct the name, rather than a library deployed to every endpoint? Then you call that service with data containing `first_name` or `givenname` and it returns that message with the standardised form. Or have a "global key" service; where you send the source system name and value, and have that translated for the destination via a lookup, allowing any field names or defined values to be translated by a generic reusable component: Entity | System Name | System Value | GlobalKey ------------------------------------------------------------------------------------------- Boolean | HR | TRUE | 52582622-4322-445b-bb7a-8ca118d0ca2b Boolean | HR | FALSE | 688c2298-6b99-4c31-a356-1ab9b1caacde Boolean | Finance | 1 | 52582622-4322-445b-bb7a-8ca118d0ca2b Boolean | Finance | 0 | 688c2298-6b99-4c31-a356-1ab9b1caacde Boolean | Sales | Yes | 52582622-4322-445b-bb7a-8ca118d0ca2b Boolean | Sales | No | 688c2298-6b99-4c31-a356-1ab9b1caacde FieldName | HR | GivenName | 3de31cff-e4bb-4d4a-a819-fd96c7c5032e FieldName | HR | Surname | e799b891-1eeb-4598-85a5-a9534d3a3a4c FieldName | Finance | First_Name | 3de31cff-e4bb-4d4a-a819-fd96c7c5032e FieldName | Finance | Last_Name | e799b891-1eeb-4598-85a5-a9534d3a3a4c FieldName | Sales | FirstName | 3de31cff-e4bb-4d4a-a819-fd96c7c5032e FieldName | Sales | Surname | e799b891-1eeb-4598-85a5-a9534d3a3a4c As for "handcrafted XML"... the alternative looks like handcrafted code... For translation a language like XSLT which was designed for translating data from one format to another seems like a good choice. You can also use this to get around your translation issue by defining a central "universal" format; so you can have XSLTs to translate messages from source systems to the universal format, then other XSLTs to translate from universal to the destination; so that adding or removing a system (/service) only impacts that system / you don't need to rewrite every point-to-point interaction of that service with another (OK, not point to point since we have microservices; but if they're not adding value by abstracting you away from point to point then essentially you've just got complex point-to-point interactions rather than simple point-to-point).
- HeavyStorm 8y agoI am a strong believer that microservices is over hyped. I usually resist when senior management asks for us to use it (that proves the point of hype). But by reading the first paragraphs of the article you see that the guys from Segment made a series of grave mistakes on their "microservices architecture", the most important one being the use of a shared library on many services. The goal of microservices is to achieve isolation, and sharing components with specific business rules between them not only defeats the purpose, but results in increased headaches. Without deep knowledge of the solution, it's hard to judge, but it seems this was never the case for microservices. They needed infrastructure isolation when the first delay issues surfaced, but there wasn't anything driving splitting the code up. Sam Newman discusses on his book how to find the proper seams to split services (DDD aggregates being the most common answer) and it seems to people are making rather arbitrary decisions.
- inetknght 8y ago> I am a strong believer that microservices is over hyped. In general, I fully concur: there are very few services that actually warrant a microservice architecture.
- andrew_ 8y ago"Everything old is new again" the mantra of software development, fashion, etc, etc.
- Hendrikto 8y agoSounds like a typical case of hype driven development. When faced with a problem they threw buzzwords at it, hoping it would go away. Had they taken the time to explore the root causes of these problems and how to best approach them, they probably wouldn‘t have taken the microservices approach to begin with. I see this far too often. Instead of looking for solutions to problems, people look for problems to try the new hot solution they read about. Happened with ML/AI, NoSQL, Microservices, Blockchain, …
- amorousf00p 8y agoI don't know. If this is really how software is written in large scale web service environments you will always have problems. It just seems like sh*t to me.
- tsenart 8y ago#MakeMonolithsGreatAgain https://twitter.com/tsenart/status/1017061272188280833 https://twitter.com/tsenart/status/1017061272188280833
- martin_drapeau 8y agoLots of people judging/bashing the author (and team) that they made bad choices jumpinng on the microservice bandwagon when it clearly wasn't the correct thing to do. Sure, but at the same time the team learned some valuable lessons, gained experience and hopefully matured. I applaud Alexandra for opening up and sharing that with us.
- bsclifton 8y agoMicroservices are a great tool for the toolbox. Not everybody needs them. It has definitely seemed like a fad over the past few years to push everybody to use them (even projects or orgs that don't need them). I suspect that is what happened here
- dunk010 8y agoPerhaps they should have read this: https://programmingisterrible.com/post/162346490883/how-do-you-cut-a-monolith-in-half https://programmingisterrible.com/post/162346490883/how-do-y... It would have saved them a lot of man-years of time.
- egidijus 8y agodoes anyone know if segment is profitable? revenue? number of clients? tech team size? tech team maturity? it could be a business decision... where their dev/infratructure salary costs do not translate to profits. the words "developer productivity" are mentioned a lot, depends how segment quantifies "productivity". businesses/companies at early stages of growth make decisions that allow them to be competitive and profitable, "the holy grail of application architecture" might not apply yet or for the first 5 years, or ever.
- kennethh 8y agoThe failure her does seems like someone who have made microservices with the wrong bounded context (Domain Driven design). When one splits a monolith up into microservices it is essential to not split a bounded context into several micoservices (or 100+ in this case).