10 ms·

Why Twilio Segment moved from microservices back to a monolith

281 points by birdculture 11mo ago

develatio 11mo ago
can you add [2018] to the title, please?
- pmbanugo 11mo ago
  have they reverted to microservices?
  - Towaway69 11mo ago
    Mono services in a micro repository. /s
- andrewmuia 11mo ago
  No kidding, not cool to be rehashing an article that is 7 years old. In tech terms, that is antiquity.
yieldcrv 11mo ago
I feel like microservices have gotten a lot easier over the last 7 years from when Twilio experienced this, not just from my experience but from refinements in architectures There are infinite permutations in architecture and we've collectively narrowed them down to things that are cheap to deploy, automatically scale for low costs, and easily replicable with a simple script We should be talking about how AI knows those scripts too and can synthetize adjustments, dedicated Site Reliability Engineers and DevOps is great for maintaining convoluted legacy setups, but irrelevant for doing the same thing from scratch nowadays
- moltar 11mo ago
  Do you have any recommended reading on the topic of refinements in architectures? Thank you.
- eYrKEC2 11mo ago
  You know what I think is better than a push of the CPU stack pointer and a jump to a library? A network call. Because nothing could be better for your code than putting the INTERNET into the middle of your application. -- The "micro" of microservices has always been ridiculous. If it can run on one machine then do it. Otherwise you have to deal with networking. Only do networking when you have to. Not as a hobby, unless your program really is a hobby.
  - ikiris 11mo ago
    Not everything you think you know is right. https://github.com/sirupsen/napkin-math https://github.com/sirupsen/napkin-math
    - josephg 11mo ago
      Well implemented network hardware can have high bandwidth and low latency. But that doesn't get around the complexity and headaches it brings. Even with the best fiber optics, wires can be cut or tripped over. Controllers can fail. Drivers can be buggy. Networks can be misconfigured. And so on. Any request - even sent over a local network - can and will fail on you eventually. And you can't really make a microservice system keep working properly when links start failing. Local function calls are infinitely more reliable. The main operational downside with a binary monolith is that a bug in one part of the program will crash the whole thing. Honestly, I still think Erlang got it right here with supervisor trees. Use "microservices". But let them all live on the same computer, in the same process. And add tooling to the runtime environment to allow individual "services" to fail or get replaced without taking down the rest of the system.
  - yieldcrv 11mo ago
    it's not really "micro" but more so "discreet" as in special purpose, one off. to ensure consistent performance, as opposed to shared performance. yes, networking is the bottleneck between the processes, while one machine is the bottleneck to end users
    - Nextgrid 11mo ago
AndrewKemendo 11mo ago
> Microservices is a service-oriented software architecture in which server-side applications are constructed by combining many single-purpose, low-footprint network services. Gonna stop you right there. Microservices have nothing to do with the hosting or operating architecture. Martin Fowler who formalized the term, Microservices are: “In short, the microservice architectural style is an approach to developing a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms, often an HTTP resource API. These services are built around business capabilities and independently deployable by fully automated deployment machinery” You can have an entirely local application built on the “microservice architectural style.” Saying they are “often HTTP and API” is besides the point. The problem Twilio actually describe is that they messed up service granularity and distributed systems engineering processes Twilio's experience was not a failure of the microservice architectural style. This was a failure to correctly define service boundaries based on business capabilities. Their struggles with serialization, network hops, and complex queueing were symptoms of building a distributed monolith, which they finally made explicit with this move. So they accidentally built a system with the overhead of distribution but the tight coupling of a single application. Now they are making their foundations of architecture fit what they built, likely cause they poorly planned it. The true lesson is that correctly applying microservices requires insanely hard domain modeling and iteration and meticulous attention to the "Distributed Systems Premium." https://martinfowler.com/microservices/ https://martinfowler.com/microservices/
- deleted 11mo ago
  [deleted]
- Scubabear68 11mo ago
  Please don’t fall into the Fowler-said-so trap. Just because he says something does not mean Fowler “formalized the term”. Martin wrote about every topic under the sun, and he loved renaming and or redefining things to fit his world view, and incidentally drive people not just to his blog but also to his consultancy, Thoughtworks. PS The “single application” line shows how dated Fowlers view were then and certainly are today.
MrDarcy 11mo ago
Reading it with hindsight, their problems have less to do with the technical trade off of micro or monolith services and much more to do with the quality and organizational structure of their engineering department. The decisions and reasons given shine a light on the quality. The repository and test layout shine a light on the structure. Given the quality and the structure neither approach really matters much. The root problems are elsewhere.
- CharlieDigital 11mo ago
  My observation is that many teams lack strong "technical discipline"; someone that says "no, don't do that", makes the case, and takes a stand. It's easy to let the complexity genie out of the bottle if the team doesn't have someone like this with enough clout/authority to actually make the team pause.
  - Otek 11mo ago
    I 100% agree with you but also sad fact is that it’s easy to understand why people don’t want to take this role. You can make enemies easily, you need to deliver “bad news” and convince people to put more effort or prove that effort they did was not enough. Why bother when you probably won’t be the one that have to clean it up
    - CharlieDigital 11mo ago
      > You can make enemies easily... Short term, definitely. In the long tail? If you are right more than you are wrong, then that manifests as respect.
      - AlwaysRock 11mo ago
        Ha! I wish I worked at the places you have worked!
  - Aeolun 11mo ago
    I think the problem is that this microservices vs monolith decision is a really hard one to convince people of. I made a passionate case for ECS instead of lambda for a long time, but only after the rest of the team and leadership see the problems the popular strategy generates do we get something approaching uptake (and the balance has already shifted to kubernetes instead, which is at least better)
mjr00 11mo ago
> Once the code for all destinations lived in a single repo, they could be merged into a single service. With every destination living in one service, our developer productivity substantially improved. We no longer had to deploy 140+ services for a change to one of the shared libraries. One engineer can deploy the service in a matter of minutes. If you must to deploy every service because of a library change, you don't have services, you have a distributed monolith. The entire idea of a "shared library" which must be kept updated across your entire service fleet is antithetical to how you need to treat services.
- wowohwow 11mo ago
  I think your point while valid, it is probably a lot more nuanced. From the post it's more akin to an Amazon shared build and deployment system than "every library update needs to redeploy every time scenario". It's likely there's a single source of truth where you pull libraries or shared resources from, when team A wants to update the pointer to library-latest to 2.0 but the current reference of library-latest is still 1.0, everyone needs to migrate off of it otherwise things will break due to backwards compatibility or whatever. Likewise, if there's a -need- to remove a version for a vulnerability or what have you, then everyone needs to redeploy, sure, but the centralized benefit of this likely outweighs the security cost and complexity of tracking the patching and deployment process for each and every service. I would say those systems -are- and likely would be classified as micro services but from a cost and ease perspective operate within a shared services environment. I don't think it's fair to consider this style of design decision as a distributed monolith. By that level of logic, having a singular business entity vs 140 individual business entities for each service would mean it's a distributed monolith.
  - mjr00 11mo ago
    > It's likely there's a single source of truth for where you pull libraries or shared resources from, when team A wants to update the pointer to library-latest to 2.0 but the current reference of library-latest is still 1.0, everyone needs to migrate off of it otherwise things will break due to backwards compatibility or whatever. No, this misses one of the biggest benefits of services; you explicitly don't need everyone to upgrade library-latest to 2.0 at the same time. If you do find yourself in a situation where you can't upgrade a core library like e.g. SQLAlchemy or Spring, or the underlying Python/Java/Go/etc runtime, without requiring updates to every service, you are back in the realm of a distributed monolith.
chmod775 11mo ago
In practice most monoliths turned into "microservices" are just monoliths in disguise. They still have most of the failure modes of the original monolith, but now with all the complexity and considerable challenges of distributed computing layered on top. Microservices as a goal is mostly touted by people who don't know what the heck they're doing - the kind of people who tend to mistakenly believe blind adherence to one philosophy or the other will help them turn their shoddy work into something passable. Engineer something that makes sense. If, once you're done, whatever you've built fits the description of "monolith" or "microservices", that's fine. However if you're just following some cult hoping it works out for your particular use-case, it's time to reevaluate whether you've chosen the right profession.
- deleted 11mo ago
  [deleted]
- Nextgrid 11mo ago
  Microservices were a fad during a period where complexity and solving self-inflicted problems were rewarded more than building an actual sustainable business. It was purely a career- & resume-polishing move for everyone involved. Putting this anywhere near "engineering" is an insult to even the shoddiest, OceanGate-levels of engineering.
  - abernard1 11mo ago
    I remember when microservices were introduced and they were solving real problems around 1) independent technological decisions with languages, data stores, and scaling, and 2) separating team development processes. They came out of Amazon, eBay, Google and a host of successful tech titans that were definitely doing "engineering." The Bezos mandate for APIs in 2002 was the beginning of that era. It was when the "microservices considered harmful" articles started popping up that microservices had become a fad. Most of the HN early-startup energy will continue to do monoliths because of team communication reasons. And I predict that if any of those startups are successful, they will have need for separate services for engineering reasons. If anything, the historical faddishness of HN shows that hackers pick the new and novel because that's who they are, for better or worse.
readthenotes1 11mo ago
Some this sounds like the journey to ejb's and back.
nyrikki 11mo ago
> Once the code for all destinations lived in a single repo, they could be merged into a single service. With every destination living in one service, our developer productivity substantially improved. We no longer had to deploy 140+ services for a change to one of the shared libraries. One engineer can deploy the service in a matter of minutes. This is the problem with the undefined nature of the term `microservices`, In my experience if you can't develop in a way that allows you to deploy all services independently and without coordination between services, it may not be a good fit for your orgs needs. In the parent SOA(v2), what they described is a well known anti-pattern: [0] Application Silos to SOA Silos * Doing SOA right is not just about technology. It also requires optimal cross-team communications. Web Service Sprawl * Create services only where and when they are needed. Target areas of greatest ROI, and avoid the service sprawl headache. If you cannot, due to technical or political reasons, retain the ability to independently deploy a service, no matter if you choose to actually independently deploy, you will not gain most of the advantages that were the original selling point of microservices, which had to do more with organizational scaling than technical conserns. There are other reasons to consider the pattern, especially due to the tooling available, but it is simply not a silver bullet. And yes, I get that not everyone is going to accept Chris Richardson's definitions[1], but even in more modern versions of this, people always seem to run into the most problems because they try to shove it in a place where the pattern isn't appropriate, or isn't possible. But kudos to Twilio for doing what every team should be, reassessing if their previous decisions were still valid and moving forward with new choices when they aren't. [0] https://www.oracle.com/technetwork/topics/entarch/oea-soa-antipatterns-133388.pdf https://www.oracle.com/technetwork/topics/entarch/oea-soa-an... [1] https://microservices.io/post/architecture/2022/05/04/microservice-architecture-essentials-deployability.html https://microservices.io/post/architecture/2022/05/04/micros...
- yearolinuxdsktp 11mo ago
  I would caution that microservices should be architected with technical concerns first—-being able to deploy independently is a valid technical concern too. Doing it for organizational scaling can lead to insular vision with turf defensive attitude, as teams are rewarded on the individual service’s performance and not the complete product’s performance. Also refactoring services now means organizational refactoring, so the friction to refactor is massively increased. I agree that patterns should be used where most appropriate, instead of blindly. What pains me is that a language like “Cloud-Native” has been usurped to mean microservices. Did Twilio just stop having a “Cloud-Native” product due to shipping a monolith? According to CNCF, yes. According to reason, no.
mikert89 11mo ago
"Microservices is the software industry’s most successful confidence scam. It convinces small teams that they are “thinking big” while systematically destroying their ability to move at all. It flatters ambition by weaponizing insecurity: if you’re not running a constellation of services, are you even a real company? Never mind that this architecture was invented to cope with organizational dysfunction at planetary scale. Now it’s being prescribed to teams that still share a Slack channel and a lunch table. Small teams run on shared context. That is their superpower. Everyone can reason end-to-end. Everyone can change anything. Microservices vaporize that advantage on contact. They replace shared understanding with distributed ignorance. No one owns the whole anymore. Everyone owns a shard. The system becomes something that merely happens to the team, rather than something the team actively understands. This isn’t sophistication. It’s abdication. Then comes the operational farce. Each service demands its own pipeline, secrets, alerts, metrics, dashboards, permissions, backups, and rituals of appeasement. You don’t “deploy” anymore—you synchronize a fleet. One bug now requires a multi-service autopsy. A feature release becomes a coordination exercise across artificial borders you invented for no reason. You didn’t simplify your system. You shattered it and called the debris “architecture.” Microservices also lock incompetence in amber. You are forced to define APIs before you understand your own business. Guesses become contracts. Bad ideas become permanent dependencies. Every early mistake metastasizes through the network. In a monolith, wrong thinking is corrected with a refactor. In microservices, wrong thinking becomes infrastructure. You don’t just regret it—you host it, version it, and monitor it. The claim that monoliths don’t scale is one of the dumbest lies in modern engineering folklore. What doesn’t scale is chaos. What doesn’t scale is process cosplay. What doesn’t scale is pretending you’re Netflix while shipping a glorified CRUD app. Monoliths scale just fine when teams have discipline, tests, and restraint. But restraint isn’t fashionable, and boring doesn’t make conference talks. Microservices for small teams is not a technical mistake—it is a philosophical failure. It announces, loudly, that the team does not trust itself to understand its own system. It replaces accountability with protocol and momentum with middleware. You don’t get “future proofing.” You get permanent drag. And by the time you finally earn the scale that might justify this circus, your speed, your clarity, and your product instincts will already be gone." -DHH
- untwerp 11mo ago
maxdo 11mo ago
Both approaches can fail. Especially in environments like Node.js or Python, there's a clear limit to how much code an event loop can handle before performance seriously degrades. I managed a product where a team of 6–8 people handles 200+ microservices. I've also managed other teams at the same time on another product where 80+ people managed a monolith. What i learned? Both approaches have pros and cons. With microservices, it's much easier to push isolated changes with just one or two people. At the same time, global changes become significantly harder. That's the trade-off, and your mental model needs to align with your business logic. If your software solves a tightly connected business problem, microservices probably aren't the right fit. On the other hand, if you have a multitude of integrations with different lifecycles but a stable internal protocol, microservices can be a lifesaver. If someone tries to tell you one approach is universally better, they're being dogmatic/religious rather than rational. Ultimately, it's not about architecture, it's about how you build abstractions and approach testing and decoupling.
- strken 11mo ago
  Can you explain a bit more about what you mean by a limit on how much code an event loop can handle? What's the limit, numerically, and which units does it use? Are you running out of CPU cache?
  - joker666 11mo ago
    I assume he means, how much work you let the event loop do without yielding. It doesn't matter if there's 200K lines of code but no real traffic to keep the event loop busy.
  - BoorishBears 11mo ago
    Most people don't realize their applications are running like dogwater on Node because serverless is letting them smooth it over by paying 4x what they would be paying if they moved 10 or so lines of code and a few regexes to a web worker. (and I say that as someone who caught themselves doing the same: severless is really good at hiding this.)
- rozap 11mo ago
btown 11mo ago
Some important context to this 2018 article is given here: https://www.twilio.com/en-us/blog/archive/2018/introducing-centrifuge https://www.twilio.com/en-us/blog/archive/2018/introducing-c... TL;DR they have a highly partitioned job database, where a job is a delivery of a specific event to a specific destination, and each partition is acted upon by at-most-one worker at a time, so lock contention is only at the infrastructure level. In that context, each worker can handle a similar balanced workload between destinations, with a fraction of production traffic, so a monorepo makes all the sense in the world. IMO it speaks to the way in which microservices can be a way to enforce good boundaries between teams... but the drawbacks are significant, and a cross-team review process for API changes and extensions can be equally effective and enable simplified architectures that sidestep many distributed-system problems at scale.
- abernard1 11mo ago
  They also failed as a company, which is why that's on Twilio's blog now. So there's that. Undoubtedly their microservices architecture was a bad fit because of how technically focused the product was. But their solution with a monolith didn't have the desired effect either.
  - btown 11mo ago
    Failed? It was a $3.2B acquisition with a total of 283M raised. I don’t see any way that’s a failure. That said I’m curious if you’re basing this on service degradation you’ve seen since the acquisition. We were thinking of starting to use them - is that a bad move?
    - abernard1 11mo ago
      By all means use Segment. Segment was a great technology with an incredible technical vision for what they wanted to do. I was in conversations in that office on Market far beyond what they ended up doing post-acquisition. But a company that can't stand on its own isn't a success in my opinion. Similar things can be said about companies that continue to need round after round of funding without an IPO. My comment is of the "(2018)" variety. Old news that didn't age well like the people jumping on the "Uber: why we switched to MySQL from Postgres" post. (How many people would choose that decision today?) People tend to divorce the actual results of a lot of these companies from the gripes of the developers of the tech blogs.
blatherard 11mo ago
(2018)
ShakataGaNai 11mo ago
Too much of anything sucks. Too big of a monolith? Sucks. Too many microservices? Sucks. Getting the right balance is HARD. Plus, it's ALWAYS easier/better to run v2 of something when you completely re-write v1 from scratch. The article could have just as easily been "Why Segment moved from 100 microservices to 5" or "Why Segment rewrote every microservice". The benefits of hindsight and real-world data shouldn't be undersold. At the end of the day, write something, get it out there. Make decisions, accept some of them will be wrong. Be willing to correct for those mistakes or at least accept they will be a pain for a while. In short: No matter what you do the first time around... it's wrong.
rtpg 11mo ago
I am _not_ a microservices guy (like... at all) but reading this the "monorepo"/"microservices" false dichotomy stands out to me. I think way too much tooling assumes 1:1 pairings between services and repos (_especially_ CI work). In huge orgs Git/whatever VCS you're using would have problems with everything in one repo, but I do think that there's loads of value in having everything in one spot even if it's all deployed more or less independently. But so many settings and workflows couple repos together so it's hard to even have a frontend and backend in the same place if both teams manage those differently. So you end up having to mess around with N repos and can't send the one cross-cutting pull request very easily. I would very much like to see improvements on this front, where one repo could still be split up on the forge side (or the CI side) in interesting ways, so review friction and local dev work friction can go down. (shorter: github and friends should let me point to a folder and say that this is a different thing, without me having to interact with git submodules. I think this is easier than it used to be _but_)
- GeneralMayhem 11mo ago
  I worked on building this at $PREV_EMPLOYER. We used a single repo for many services, so that you could run tests on all affected binaries/downstream libraries when a library changed. We used Bazel to maintain the dependency tree, and then triggered builds based on a custom Github Actions hook that would use `bazel query` to find the transitive closure of affected targets. Then, if anything in a directory was affected, we'd trigger the set of tests defined in a config file in that directory (defaulting to :...), each as its own workflow run that would block PR submission. That worked really well, with the only real limiting factor being the ultimate upper limit of a repo in Github, but of course took a fair amount (a few SWE-months) to build all the tooling.
  - physicles 11mo ago
    We’re in the middle of this right now. Go makes this easier: there’s a go CLI command that you can use to list a package’s dependencies, which can be cross-referenced with recent git changes. (duplicating the dependency graph in another build tool is a non-starter for me) But there are corner cases that we’re currently working through. This, and if you want build + deploy that’s faster than doing it manually from your dev machine, you pay $$$ for either something like Depot, or a beefy VM to host CI. A bit more work on those dependency corner cases, along with an auto-sleeping VM, should let us achieve nirvana. But it’s not like we have a lot of spare time on our small team.
otterley 11mo ago
Discussion in 2018, when this blog post was published: https://news.ycombinator.com/item?id=17499137 https://news.ycombinator.com/item?id=17499137
shoo 11mo ago
Great writeup. Much of this is more about testing, how package dependencies are expressed and many-repo/singlerepo tradeoffs than "microservices"! Maintaining and testing a codebase containing many external integrations ("Destinations") was one of the drivers behind the earlier decision to shatter into many repos, to isolate the impact of Destination-specific test suite failures caused because some tests were actually testing integration to external 3rd party services. One way to think about that situation is in terms of packages, their dependency structure, how those dependencies are expressed (e.g. decoupled via versioned artefact releases, directly coupled via monorepo style source checkout), their rates of change, and the quality of their automated tests suites (high quality meaning the test suite runs really fast, tests only the thing it is meant to test, has low rates of false negatives and false positives, low quality meaning the opposite). Their initial situation was one that rapidly becomes unworkable: a shared library package undergoing a high rate of change depended on by many Destination packages, each with low quality test suites, where the dependencies were expressed in a directly-coupled way by virtue of everything existing in a single repo. There's a general principle here: multiple packages in a single repo with directly-coupled dependencies, where those packages have test suites with wildly varying levels of quality, quickly becomes a nightmare to maintain. The packages with low quality test suites that depend upon high quality rapidly changing shared packages generate spurious test failures that need to be triaged and slow down development. Maintainers of packages that depend upon rapidly changing shared package but do not have high quality test suites able to detect regressions may find their package frequently gets broken without anyone realising in time. Their initial move solves this problem by shattering the single repo and trade directly-coupled dependencies with decoupled versioned dependencies, to decouple the rate of change of the shared package from the per Destination packages. That was an incremental improvement but added the complexity and overhead of maintaining multiple versions of the "shared" library and per-repo boilerplate, which grows over time as more Destinations are added or more changes are made to the shared library while deferring the work to upgrade and retest Destinations to use it. Their later move was to reverse this, go back to directly-coupled dependencies, but instead improve the quality of their per-Destination test suites, particularly by introducing record/replay style testing of Destinations. Great move. This means that the test suite of each Destination is measuring "is the Destination package adhering to its contract in how it should integrate with the 3rd party API & integrate with the shared package?" without being conflated with testing stuff that's outside of the control of code in the repo (is the 3rd party service even up, etc).
0xbadcafebee 11mo ago
These "we moved from X to Y" posts are like Dunning-Kruger humblebrags. Yes, we all lack information and make mistakes. But there's never an explanation in these posts of how they've determined their new decision is any less erroneous than their old decision. It's like they threw darts at a wall and said "cool, that's our new system design (and SDLC)". If you have not built it yourself before, and have not studied in depth an identical system, just assume you are doing the wrong thing. Otherwise you are running towards another Dunning-Kruger pit. If you have a company that writes software, please ask a professional software/systems architect to review your plans before you build. The initial decisions here would be a huge red flag to any experienced architect, and the subsequent decisions are full of hidden traps, and are setting them up for more failure. If you don't already have a very skilled architect on staff (99% chance you don't) you need to find one and consult with them. Otherwise your business will suffer from being trapped in unnecessary time-consuming expensive rework, or worse, the whole thing collapsing.
mcrk 11mo ago
They obviously read https://grugbrain.dev/#grug-on-microservices https://grugbrain.dev/#grug-on-microservices
pss314 11mo ago
A recent blog post from Docker mentions about Twilio and Amazon Prime Video seeing gains by moving away from microservices to monolith You Want Microservices, But Do You Really Need Them? https://www.docker.com/blog/do-you-really-need-microservices/ https://www.docker.com/blog/do-you-really-need-microservices...
brightstep 11mo ago
They have a monolith but struggle with individual subsystem failures bringing down the whole thing. Sounds like they would benefit from Elixir’s isolated, fail-fast architecture.
mlhpdx 11mo ago
Wow. Their experience could not be more different than mine. As I’m contemplating the first year of my startup I’ve tallied 6000 deployments and 99.997 percent uptime and a low single digit rollback percentage (MTTR in low single digit minutes and fractional, single cell impact for them so far). While I’m sure it’s possible for a solo entrepreneur to hit numbers like that with a monolith I have never done so, and haven’t see others do so. Edit: I’d love to eat the humble pie here. If you have examples of places where monoliths are updated 10-20 times a day by a small (or large) team post the link. I’ll read them all.
- AlotOfReading 11mo ago
  The idea of deploying to production 10-20 times per day sounds terrifying. What's the rationale for doing so? I'll assume you're not writing enough bugs that customers are reporting 10-20 new ones per day, but that leaves me confused why you would want to expose customers to that much churn. If we assume an observable issue results in a rollback and you're only rolling back 1-2% of the time (very impressive), once a month or so customers should experience observable issues across multiple subsequent days. That would turn me off making a service integral to my workflow.
  - et1337 11mo ago
    If something is difficult or scary, do it more often. Smaller changes are less risky. Code that is merged but not deployed is essentially “inventory” in the factory metaphor. You want to keep inventory low. If the distance between the main branch and production is kept low, then you can always feel pretty confident that the main branch is in a good state, or at least close to one. That’s invaluable when you inevitably need to ship an emergency fix. You can just commit the fix to main instead of trying to find a known good version and patching it. And when a deployment does break something, you’ll have a much smaller diff to search for the problem.
    - AlotOfReading 11mo ago
      There's a lot of middle ground between "deploy to production 20x a day" and "deploy so infrequently that you forget how to deploy". Like, once a day? I have nothing against emergency fixes, unless you're doing them 9-19x a day. Hotfixes should be uncommon (neither rare nor standard practice).
_pdp_ 11mo ago
In my previous company we did everything as a micro service. In the company before that it was serverless on AWS! In both cases we had to come up with clever solutions to simply get by because communication between services is of a problem. It is difficult (not impossible) to keep all the contracts in sync and deployment has to be coordinated in a very specific way sometimes. The initial speed you get is soon lost further down the path due to added complexities. There was fear-driven development at play. Service ownership is a problem. Far too much meetings are spent on coordination. In my latest company everything is part of the same monolith. Yes the code is huge but it is so much easier to work with. We use a lot more unit tests then integration tests. Types make sense. Refactoring is just so easy. All the troubleshooting tools including specialised AI agents built on top of our own platform are part of the code-base which is kind of interesting because I can see how this is turning into a self-improving system. It is fascinating! We are not planning to break up the monolith unless we grow so much that is impossible to manage from a single git repository. As far as I can tell this may never happen as it is obvious that much larger projects are perfectly well maintained in the exact same way. The only downside is that build takes longer but honestly we found ways around that as well in the past and now with further improvements in the toolchains delivered by the awesome open-source communities around the world, I expect to see at least 10x improvement in deployment time in 2026. Overall, in my own assessment, the decision to go for a monolith allowed us to build and scale much faster than if we had used micro services. I hope this helps.
- sethammons 11mo ago
  My experience is the opposite. I worked at SendGrid for a decade and we scaled the engineering org from a dozen to over 500 operating at scale sending billions of messages daily. Micro services. Well, services. The word micro messes people up. I have also worked at half a dozen other shops with various architectures. In every monolith, people violate separation of concerns and things are tightly coupled. I have only ever seen good engineering velocity happen when teams are decoupled from one another. I have only seen this happen in a (micro) service architecture. Overall, in my own assessment, the decision to stick with a monolith has slowed down velocity and placed limits on scale at every other company I have been at and require changes towards decoupled services to be able to ship with any kind of velocity. The place I just left took 2 years, over 50 teams, and over 150 individual contributors to launch a product that required us to move over an interface for sending messages from ORM querysets to DTOs. We needed to unlock our ability to start rearchitecting the modules because before it was impossible to know the actual edges of the system and how it used the data. This was incredibly expensive and hard and would never have happened but for the ability to reach into other's domains and making assumptions making things hard. Don't couple systems. Micro services are the only arch I have seen successfully do this.
doctor_phil 11mo ago
I don't think this blog post reflects so well on this engineering team. Kudos to them to be so transparent about it though. "We had so many flaky tests that depended on 3rd parties that broke pipelines that we decided on micro-services" is not something I would put on my CV at least.
- electromech 11mo ago
  That seems unfair. There's a lot we don't know about the politics behind the scenes. I'd bet that the individuals who created the microservice architecture aren't the same people who re-consolidated them into one service. If true, the authors of the article are being generous to the original creators of the microservices, which I think reflects well on them for not badmouthing their predecessors.
andrewprock 11mo ago
The whole point of micro-services is to manage dependencies independently across service boundaries, using the API as the contract, not the internal libraries. Then you can implement a service in Java, Python, Rust, C++, etc, and it doesn't matter. Coupling your postgres db to your elasticsearch cluster via a hard library dependency impossibly heavy. The same insight applies to your bespoke services.
wg0 11mo ago
Thanks. It was a stupid most idea for MOST shops. I think maybe it works for AWS, Google and Netflix but everywhere in my career, I saw 90% of the problem was due to microservices. Diving system into composable parts is a very very difficult problem already and it is only foolish to introduce further network boundaries between them. Next comeback I see is away from React and SPAs as view transitions become more common.
bob1029 11mo ago
Monolith is definitely what you want to start with. Being able to ~instantly obtain a perfect list of all references to all symbols is an extraordinarily powerful capability. The stronger the type system, the more leverage you get. If you have only ever had experience with weak type systems or poor tooling, I could understand how the notion of putting everything into one compilation context seems pointless.
- sethammons 11mo ago
  /me raises hand. Any system that passes querysets around for one. Can't know who is using what.
sethammons 11mo ago
I left Twilio in 2018. I spent a decade at SendGrid. I spent a small time in Segment. The shitty arch is not a point against (micro)services. SendGrid, another Twilio property, uses (micro)services to great effect. Services there were fully independently deployable.
sriku 11mo ago
In a discussion I was in recently, a participant mentioned "culture eats strategy for breakfast" .. which perhaps makes sense in this context. Be bold enough to do what makes the team and the product thrive.
akoumjian 11mo ago
I humbly post this little widget to help your team decide if some functionality warrants being a separate service or not: https://mulch.dev/service-scorecard/ https://mulch.dev/service-scorecard/
waterproof 11mo ago
The only rationale given for the initial switch to microservices is this: > Initially, when the destinations were divided into separate services, all of the code lived in one repo. A huge point of frustration was that a single broken test caused tests to fail across all destinations. You kept breaking tests in main so you thought the solution was to revamp your entire codebase structure? Seems a bit backward.
- kgeist 11mo ago
  We had a similar problem in our monolith. Team #1 works on a feature, their code breaks tests. Team #2 works on another feature, their code is OK, but they can't move forward because of the failing tests from team #1. Plus often it takes additional time to figure out if it the tests fail because of feature #1 or feature #2, and who must fix them. We solved it by simply giving every team their own dev environment (before merging to main). So if tests break in feature #1, it doesn't break anything for feature #2 or team #2. It's all confined to their environment. It's just an additional VM + a config in CI/CD. The only downside of this is that if there are conflicts between features they won't be caught immediately (only after one of the teams finally merges to main). But in our case it wasn't a problem because different teams rarely worked on the same parts of the monorepo at the same time due to explicit code ownership.
wiradikusuma 11mo ago
> With everything running in a monolith, if a bug is introduced in one destination that causes the service to crash, the service will crash for all destinations We can have a service with 100 features, but only enable the features relevant to a given "purpose". That way, we can still have "micro services" but they're running the same code: "bla.exe -foo" and "bla.exe -bar".
tonymet 11mo ago
Your “microservice” is just a clumsy & slow symbol lookup over the network, at 1000x the cpu and 10000x the latency.
TZubiri 11mo ago
This is not the first time that an engineer working at a big company thinks they are using a monolith when in reality they are a small team in charge of a single microservice, which in turn is part of a company that definitely does not run a monolith. Last time it was an aws engineer that worked on route 53, and they dismissed microservices in a startup claiming that in AWS they ran a monolith (as in the r53 dns). Everything is a monolith if you zoom in enough and ignore everything else. Which I guess you can do when you work on a big company and are in charge of a very specific role.
necovek 11mo ago
A few thoughts: this is not really a move to a monolith. Their system is still a SOA (service-oriented architecture), just like microservices (make services as small as they can be), but with larger scope. Having 140 services managed by what sounds like one team reinforces another point that I believe should be well known by now: you use SOAs (incuding microservices) to scale teams, and not services. Eg. if a single team builds a shared library for all the 140 microservices and needs to maintain them, it's going to become very expensive quickly: you'll be using v2.3.1 in one service and v1.0.8 in another, and you won't even know yourself what API is available. Operationally, yes, you'll have to watch over 140 individual "systems" too. There are ways to mitigate this, but they have their own trade-offs (I've posted them in another comment). As per Conway's law, software architecture always follows the organizational structure, and this seems to have happened here: a single team is moving away from unneeded complexity to more effectively manage their work and produce better outcomes for the business. It is not a monolith, but properly-scoped service level (scoped to the team). This is, in my experience, the sweet spot. A single team can run and operate multiple independent services, but with growth in those services, they will look to unify, so you need to restructure the team if you don't want that to happen. This is why I don't accept "system architect" roles as those don't give you the tools to really drive the architecture how it can be driven, and I really got into "management" :)
honkycat 11mo ago
I don't care how it is done just dont rely on your database schema for data modeling and business logic
majgr 11mo ago
I have feeling that microservices improve overall design when they can live on their own, as microapps perhaps, also with their own UI. What is the point of service if it is not usable beyond its original design and just bound to other similar services?
andrewstuart 11mo ago
You can have infrastructure complexity (microservices) or trade it for development complexity (monolith). Choose one.
renewiltord 11mo ago
Is it 2018? Are you guys going to repost the MySQL DB as a queue story again? Perhaps an announcement that you’re migrating to Java 9 and what you learned about generics?
ram_shares 11mo ago
The “distributed monolith” line is the key takeaway here. Microservices only buy you something if teams can deploy, version, and reason about them independently. Once shared libraries or coordinated deploys creep in, you’ve taken on all the operational cost with none of the autonomy benefits. I’ve seen monoliths with clear module boundaries outperform microservice setups by an order of magnitude in developer throughput.
- YetAnotherNick 11mo ago
  If microservice or monolith is giving order of magnitude improvement in productivity, you clearly are doing something wrong, or having terrible practices.
dev_l1x_be 11mo ago
I can't believe how many times I have seen companies trying to implement microservices with multirepo and get lost in the access management, versioning and ending up producing a fragile, overly complex hot mess.
axelthegerman 11mo ago
> we had to spend time fixing the broken test even if the changes had nothing to do with the initial change. In response to this problem, it was decided to break out the code for each destination into their own repos. One could also change the way tests are run or selected. Or allow manual overrides to still deploy. Separating repos doesn't sound like the only logical solution
gostsamo 11mo ago
This is a horror story of being totally unable to understand your product and its behavior and throwing people and resources in large rewrites to only learn that you still don't understand your product and its behavior. Badly done tests used as a justification to write multiple suites of badly done tests and it is all blamed on the architecture.
joeyguerra 11mo ago
Cool.

281 points by birdculture 11mo ago

develatio 11mo ago

can you add [2018] to the title, please?

pmbanugo 11mo ago
have they reverted to microservices?
- Towaway69 11mo ago
  Mono services in a micro repository. /s
andrewmuia 11mo ago
No kidding, not cool to be rehashing an article that is 7 years old. In tech terms, that is antiquity.

yieldcrv 11mo ago

I feel like microservices have gotten a lot easier over the last 7 years from when Twilio experienced this, not just from my experience but from refinements in architectures There are infinite permutations in architecture and we've collectively narrowed them down to things that are cheap to deploy, automatically scale for low costs, and easily replicable with a simple script We should be talking about how AI knows those scripts too and can synthetize adjustments, dedicated Site Reliability Engineers and DevOps is great for maintaining convoluted legacy setups, but irrelevant for doing the same thing from scratch nowadays

moltar 11mo ago
Do you have any recommended reading on the topic of refinements in architectures? Thank you.
eYrKEC2 11mo ago
You know what I think is better than a push of the CPU stack pointer and a jump to a library? A network call. Because nothing could be better for your code than putting the INTERNET into the middle of your application. -- The "micro" of microservices has always been ridiculous. If it can run on one machine then do it. Otherwise you have to deal with networking. Only do networking when you have to. Not as a hobby, unless your program really is a hobby.
- ikiris 11mo ago
  Not everything you think you know is right. https://github.com/sirupsen/napkin-math https://github.com/sirupsen/napkin-math
  - josephg 11mo ago
    Well implemented network hardware can have high bandwidth and low latency. But that doesn't get around the complexity and headaches it brings. Even with the best fiber optics, wires can be cut or tripped over. Controllers can fail. Drivers can be buggy. Networks can be misconfigured. And so on. Any request - even sent over a local network - can and will fail on you eventually. And you can't really make a microservice system keep working properly when links start failing. Local function calls are infinitely more reliable. The main operational downside with a binary monolith is that a bug in one part of the program will crash the whole thing. Honestly, I still think Erlang got it right here with supervisor trees. Use "microservices". But let them all live on the same computer, in the same process. And add tooling to the runtime environment to allow individual "services" to fail or get replaced without taking down the rest of the system.
- yieldcrv 11mo ago
  it's not really "micro" but more so "discreet" as in special purpose, one off. to ensure consistent performance, as opposed to shared performance. yes, networking is the bottleneck between the processes, while one machine is the bottleneck to end users
  - Nextgrid 11mo ago

AndrewKemendo 11mo ago

> Microservices is a service-oriented software architecture in which server-side applications are constructed by combining many single-purpose, low-footprint network services. Gonna stop you right there. Microservices have nothing to do with the hosting or operating architecture. Martin Fowler who formalized the term, Microservices are: “In short, the microservice architectural style is an approach to developing a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms, often an HTTP resource API. These services are built around business capabilities and independently deployable by fully automated deployment machinery” You can have an entirely local application built on the “microservice architectural style.” Saying they are “often HTTP and API” is besides the point. The problem Twilio actually describe is that they messed up service granularity and distributed systems engineering processes Twilio's experience was not a failure of the microservice architectural style. This was a failure to correctly define service boundaries based on business capabilities. Their struggles with serialization, network hops, and complex queueing were symptoms of building a distributed monolith, which they finally made explicit with this move. So they accidentally built a system with the overhead of distribution but the tight coupling of a single application. Now they are making their foundations of architecture fit what they built, likely cause they poorly planned it. The true lesson is that correctly applying microservices requires insanely hard domain modeling and iteration and meticulous attention to the "Distributed Systems Premium." https://martinfowler.com/microservices/ https://martinfowler.com/microservices/

deleted 11mo ago
[deleted]
Scubabear68 11mo ago
Please don’t fall into the Fowler-said-so trap. Just because he says something does not mean Fowler “formalized the term”. Martin wrote about every topic under the sun, and he loved renaming and or redefining things to fit his world view, and incidentally drive people not just to his blog but also to his consultancy, Thoughtworks. PS The “single application” line shows how dated Fowlers view were then and certainly are today.

MrDarcy 11mo ago

Reading it with hindsight, their problems have less to do with the technical trade off of micro or monolith services and much more to do with the quality and organizational structure of their engineering department. The decisions and reasons given shine a light on the quality. The repository and test layout shine a light on the structure. Given the quality and the structure neither approach really matters much. The root problems are elsewhere.

CharlieDigital 11mo ago
My observation is that many teams lack strong "technical discipline"; someone that says "no, don't do that", makes the case, and takes a stand. It's easy to let the complexity genie out of the bottle if the team doesn't have someone like this with enough clout/authority to actually make the team pause.
- Otek 11mo ago
  I 100% agree with you but also sad fact is that it’s easy to understand why people don’t want to take this role. You can make enemies easily, you need to deliver “bad news” and convince people to put more effort or prove that effort they did was not enough. Why bother when you probably won’t be the one that have to clean it up
  - CharlieDigital 11mo ago
    > You can make enemies easily... Short term, definitely. In the long tail? If you are right more than you are wrong, then that manifests as respect.
    - AlwaysRock 11mo ago
      Ha! I wish I worked at the places you have worked!
- Aeolun 11mo ago
  I think the problem is that this microservices vs monolith decision is a really hard one to convince people of. I made a passionate case for ECS instead of lambda for a long time, but only after the rest of the team and leadership see the problems the popular strategy generates do we get something approaching uptake (and the balance has already shifted to kubernetes instead, which is at least better)

mjr00 11mo ago

wowohwow 11mo ago
I think your point while valid, it is probably a lot more nuanced. From the post it's more akin to an Amazon shared build and deployment system than "every library update needs to redeploy every time scenario". It's likely there's a single source of truth where you pull libraries or shared resources from, when team A wants to update the pointer to library-latest to 2.0 but the current reference of library-latest is still 1.0, everyone needs to migrate off of it otherwise things will break due to backwards compatibility or whatever. Likewise, if there's a -need- to remove a version for a vulnerability or what have you, then everyone needs to redeploy, sure, but the centralized benefit of this likely outweighs the security cost and complexity of tracking the patching and deployment process for each and every service. I would say those systems -are- and likely would be classified as micro services but from a cost and ease perspective operate within a shared services environment. I don't think it's fair to consider this style of design decision as a distributed monolith. By that level of logic, having a singular business entity vs 140 individual business entities for each service would mean it's a distributed monolith.
- mjr00 11mo ago
  > It's likely there's a single source of truth for where you pull libraries or shared resources from, when team A wants to update the pointer to library-latest to 2.0 but the current reference of library-latest is still 1.0, everyone needs to migrate off of it otherwise things will break due to backwards compatibility or whatever. No, this misses one of the biggest benefits of services; you explicitly don't need everyone to upgrade library-latest to 2.0 at the same time. If you do find yourself in a situation where you can't upgrade a core library like e.g. SQLAlchemy or Spring, or the underlying Python/Java/Go/etc runtime, without requiring updates to every service, you are back in the realm of a distributed monolith.

chmod775 11mo ago

In practice most monoliths turned into "microservices" are just monoliths in disguise. They still have most of the failure modes of the original monolith, but now with all the complexity and considerable challenges of distributed computing layered on top. Microservices as a goal is mostly touted by people who don't know what the heck they're doing - the kind of people who tend to mistakenly believe blind adherence to one philosophy or the other will help them turn their shoddy work into something passable. Engineer something that makes sense. If, once you're done, whatever you've built fits the description of "monolith" or "microservices", that's fine. However if you're just following some cult hoping it works out for your particular use-case, it's time to reevaluate whether you've chosen the right profession.

deleted 11mo ago
[deleted]
Nextgrid 11mo ago
Microservices were a fad during a period where complexity and solving self-inflicted problems were rewarded more than building an actual sustainable business. It was purely a career- & resume-polishing move for everyone involved. Putting this anywhere near "engineering" is an insult to even the shoddiest, OceanGate-levels of engineering.
- abernard1 11mo ago
  I remember when microservices were introduced and they were solving real problems around 1) independent technological decisions with languages, data stores, and scaling, and 2) separating team development processes. They came out of Amazon, eBay, Google and a host of successful tech titans that were definitely doing "engineering." The Bezos mandate for APIs in 2002 was the beginning of that era. It was when the "microservices considered harmful" articles started popping up that microservices had become a fad. Most of the HN early-startup energy will continue to do monoliths because of team communication reasons. And I predict that if any of those startups are successful, they will have need for separate services for engineering reasons. If anything, the historical faddishness of HN shows that hackers pick the new and novel because that's who they are, for better or worse.

readthenotes1 11mo ago

Some this sounds like the journey to ejb's and back.

nyrikki 11mo ago

> Once the code for all destinations lived in a single repo, they could be merged into a single service. With every destination living in one service, our developer productivity substantially improved. We no longer had to deploy 140+ services for a change to one of the shared libraries. One engineer can deploy the service in a matter of minutes. This is the problem with the undefined nature of the term `microservices`, In my experience if you can't develop in a way that allows you to deploy all services independently and without coordination between services, it may not be a good fit for your orgs needs. In the parent SOA(v2), what they described is a well known anti-pattern: [0] Application Silos to SOA Silos * Doing SOA right is not just about technology. It also requires optimal cross-team communications. Web Service Sprawl * Create services only where and when they are needed. Target areas of greatest ROI, and avoid the service sprawl headache. If you cannot, due to technical or political reasons, retain the ability to independently deploy a service, no matter if you choose to actually independently deploy, you will not gain most of the advantages that were the original selling point of microservices, which had to do more with organizational scaling than technical conserns. There are other reasons to consider the pattern, especially due to the tooling available, but it is simply not a silver bullet. And yes, I get that not everyone is going to accept Chris Richardson's definitions[1], but even in more modern versions of this, people always seem to run into the most problems because they try to shove it in a place where the pattern isn't appropriate, or isn't possible. But kudos to Twilio for doing what every team should be, reassessing if their previous decisions were still valid and moving forward with new choices when they aren't. [0] https://www.oracle.com/technetwork/topics/entarch/oea-soa-antipatterns-133388.pdf https://www.oracle.com/technetwork/topics/entarch/oea-soa-an... [1] https://microservices.io/post/architecture/2022/05/04/microservice-architecture-essentials-deployability.html https://microservices.io/post/architecture/2022/05/04/micros...

yearolinuxdsktp 11mo ago
I would caution that microservices should be architected with technical concerns first—-being able to deploy independently is a valid technical concern too. Doing it for organizational scaling can lead to insular vision with turf defensive attitude, as teams are rewarded on the individual service’s performance and not the complete product’s performance. Also refactoring services now means organizational refactoring, so the friction to refactor is massively increased. I agree that patterns should be used where most appropriate, instead of blindly. What pains me is that a language like “Cloud-Native” has been usurped to mean microservices. Did Twilio just stop having a “Cloud-Native” product due to shipping a monolith? According to CNCF, yes. According to reason, no.

mikert89 11mo ago

"Microservices is the software industry’s most successful confidence scam. It convinces small teams that they are “thinking big” while systematically destroying their ability to move at all. It flatters ambition by weaponizing insecurity: if you’re not running a constellation of services, are you even a real company? Never mind that this architecture was invented to cope with organizational dysfunction at planetary scale. Now it’s being prescribed to teams that still share a Slack channel and a lunch table. Small teams run on shared context. That is their superpower. Everyone can reason end-to-end. Everyone can change anything. Microservices vaporize that advantage on contact. They replace shared understanding with distributed ignorance. No one owns the whole anymore. Everyone owns a shard. The system becomes something that merely happens to the team, rather than something the team actively understands. This isn’t sophistication. It’s abdication. Then comes the operational farce. Each service demands its own pipeline, secrets, alerts, metrics, dashboards, permissions, backups, and rituals of appeasement. You don’t “deploy” anymore—you synchronize a fleet. One bug now requires a multi-service autopsy. A feature release becomes a coordination exercise across artificial borders you invented for no reason. You didn’t simplify your system. You shattered it and called the debris “architecture.” Microservices also lock incompetence in amber. You are forced to define APIs before you understand your own business. Guesses become contracts. Bad ideas become permanent dependencies. Every early mistake metastasizes through the network. In a monolith, wrong thinking is corrected with a refactor. In microservices, wrong thinking becomes infrastructure. You don’t just regret it—you host it, version it, and monitor it. The claim that monoliths don’t scale is one of the dumbest lies in modern engineering folklore. What doesn’t scale is chaos. What doesn’t scale is process cosplay. What doesn’t scale is pretending you’re Netflix while shipping a glorified CRUD app. Monoliths scale just fine when teams have discipline, tests, and restraint. But restraint isn’t fashionable, and boring doesn’t make conference talks. Microservices for small teams is not a technical mistake—it is a philosophical failure. It announces, loudly, that the team does not trust itself to understand its own system. It replaces accountability with protocol and momentum with middleware. You don’t get “future proofing.” You get permanent drag. And by the time you finally earn the scale that might justify this circus, your speed, your clarity, and your product instincts will already be gone." -DHH

untwerp 11mo ago

maxdo 11mo ago

Both approaches can fail. Especially in environments like Node.js or Python, there's a clear limit to how much code an event loop can handle before performance seriously degrades. I managed a product where a team of 6–8 people handles 200+ microservices. I've also managed other teams at the same time on another product where 80+ people managed a monolith. What i learned? Both approaches have pros and cons. With microservices, it's much easier to push isolated changes with just one or two people. At the same time, global changes become significantly harder. That's the trade-off, and your mental model needs to align with your business logic. If your software solves a tightly connected business problem, microservices probably aren't the right fit. On the other hand, if you have a multitude of integrations with different lifecycles but a stable internal protocol, microservices can be a lifesaver. If someone tries to tell you one approach is universally better, they're being dogmatic/religious rather than rational. Ultimately, it's not about architecture, it's about how you build abstractions and approach testing and decoupling.

strken 11mo ago
Can you explain a bit more about what you mean by a limit on how much code an event loop can handle? What's the limit, numerically, and which units does it use? Are you running out of CPU cache?
- joker666 11mo ago
  I assume he means, how much work you let the event loop do without yielding. It doesn't matter if there's 200K lines of code but no real traffic to keep the event loop busy.
- BoorishBears 11mo ago
  Most people don't realize their applications are running like dogwater on Node because serverless is letting them smooth it over by paying 4x what they would be paying if they moved 10 or so lines of code and a few regexes to a web worker. (and I say that as someone who caught themselves doing the same: severless is really good at hiding this.)
rozap 11mo ago

btown 11mo ago

Some important context to this 2018 article is given here: https://www.twilio.com/en-us/blog/archive/2018/introducing-centrifuge https://www.twilio.com/en-us/blog/archive/2018/introducing-c... TL;DR they have a highly partitioned job database, where a job is a delivery of a specific event to a specific destination, and each partition is acted upon by at-most-one worker at a time, so lock contention is only at the infrastructure level. In that context, each worker can handle a similar balanced workload between destinations, with a fraction of production traffic, so a monorepo makes all the sense in the world. IMO it speaks to the way in which microservices can be a way to enforce good boundaries between teams... but the drawbacks are significant, and a cross-team review process for API changes and extensions can be equally effective and enable simplified architectures that sidestep many distributed-system problems at scale.

abernard1 11mo ago
They also failed as a company, which is why that's on Twilio's blog now. So there's that. Undoubtedly their microservices architecture was a bad fit because of how technically focused the product was. But their solution with a monolith didn't have the desired effect either.
- btown 11mo ago
  Failed? It was a $3.2B acquisition with a total of 283M raised. I don’t see any way that’s a failure. That said I’m curious if you’re basing this on service degradation you’ve seen since the acquisition. We were thinking of starting to use them - is that a bad move?
  - abernard1 11mo ago
    By all means use Segment. Segment was a great technology with an incredible technical vision for what they wanted to do. I was in conversations in that office on Market far beyond what they ended up doing post-acquisition. But a company that can't stand on its own isn't a success in my opinion. Similar things can be said about companies that continue to need round after round of funding without an IPO. My comment is of the "(2018)" variety. Old news that didn't age well like the people jumping on the "Uber: why we switched to MySQL from Postgres" post. (How many people would choose that decision today?) People tend to divorce the actual results of a lot of these companies from the gripes of the developers of the tech blogs.

blatherard 11mo ago

(2018)

ShakataGaNai 11mo ago

Too much of anything sucks. Too big of a monolith? Sucks. Too many microservices? Sucks. Getting the right balance is HARD. Plus, it's ALWAYS easier/better to run v2 of something when you completely re-write v1 from scratch. The article could have just as easily been "Why Segment moved from 100 microservices to 5" or "Why Segment rewrote every microservice". The benefits of hindsight and real-world data shouldn't be undersold. At the end of the day, write something, get it out there. Make decisions, accept some of them will be wrong. Be willing to correct for those mistakes or at least accept they will be a pain for a while. In short: No matter what you do the first time around... it's wrong.

rtpg 11mo ago

I am _not_ a microservices guy (like... at all) but reading this the "monorepo"/"microservices" false dichotomy stands out to me. I think way too much tooling assumes 1:1 pairings between services and repos (_especially_ CI work). In huge orgs Git/whatever VCS you're using would have problems with everything in one repo, but I do think that there's loads of value in having everything in one spot even if it's all deployed more or less independently. But so many settings and workflows couple repos together so it's hard to even have a frontend and backend in the same place if both teams manage those differently. So you end up having to mess around with N repos and can't send the one cross-cutting pull request very easily. I would very much like to see improvements on this front, where one repo could still be split up on the forge side (or the CI side) in interesting ways, so review friction and local dev work friction can go down. (shorter: github and friends should let me point to a folder and say that this is a different thing, without me having to interact with git submodules. I think this is easier than it used to be _but_)

GeneralMayhem 11mo ago
I worked on building this at $PREV_EMPLOYER. We used a single repo for many services, so that you could run tests on all affected binaries/downstream libraries when a library changed. We used Bazel to maintain the dependency tree, and then triggered builds based on a custom Github Actions hook that would use `bazel query` to find the transitive closure of affected targets. Then, if anything in a directory was affected, we'd trigger the set of tests defined in a config file in that directory (defaulting to :...), each as its own workflow run that would block PR submission. That worked really well, with the only real limiting factor being the ultimate upper limit of a repo in Github, but of course took a fair amount (a few SWE-months) to build all the tooling.
- physicles 11mo ago
  We’re in the middle of this right now. Go makes this easier: there’s a go CLI command that you can use to list a package’s dependencies, which can be cross-referenced with recent git changes. (duplicating the dependency graph in another build tool is a non-starter for me) But there are corner cases that we’re currently working through. This, and if you want build + deploy that’s faster than doing it manually from your dev machine, you pay $$$ for either something like Depot, or a beefy VM to host CI. A bit more work on those dependency corner cases, along with an auto-sleeping VM, should let us achieve nirvana. But it’s not like we have a lot of spare time on our small team.

otterley 11mo ago

Discussion in 2018, when this blog post was published: https://news.ycombinator.com/item?id=17499137 https://news.ycombinator.com/item?id=17499137

shoo 11mo ago

Great writeup. Much of this is more about testing, how package dependencies are expressed and many-repo/singlerepo tradeoffs than "microservices"! Maintaining and testing a codebase containing many external integrations ("Destinations") was one of the drivers behind the earlier decision to shatter into many repos, to isolate the impact of Destination-specific test suite failures caused because some tests were actually testing integration to external 3rd party services. One way to think about that situation is in terms of packages, their dependency structure, how those dependencies are expressed (e.g. decoupled via versioned artefact releases, directly coupled via monorepo style source checkout), their rates of change, and the quality of their automated tests suites (high quality meaning the test suite runs really fast, tests only the thing it is meant to test, has low rates of false negatives and false positives, low quality meaning the opposite). Their initial situation was one that rapidly becomes unworkable: a shared library package undergoing a high rate of change depended on by many Destination packages, each with low quality test suites, where the dependencies were expressed in a directly-coupled way by virtue of everything existing in a single repo. There's a general principle here: multiple packages in a single repo with directly-coupled dependencies, where those packages have test suites with wildly varying levels of quality, quickly becomes a nightmare to maintain. The packages with low quality test suites that depend upon high quality rapidly changing shared packages generate spurious test failures that need to be triaged and slow down development. Maintainers of packages that depend upon rapidly changing shared package but do not have high quality test suites able to detect regressions may find their package frequently gets broken without anyone realising in time. Their initial move solves this problem by shattering the single repo and trade directly-coupled dependencies with decoupled versioned dependencies, to decouple the rate of change of the shared package from the per Destination packages. That was an incremental improvement but added the complexity and overhead of maintaining multiple versions of the "shared" library and per-repo boilerplate, which grows over time as more Destinations are added or more changes are made to the shared library while deferring the work to upgrade and retest Destinations to use it. Their later move was to reverse this, go back to directly-coupled dependencies, but instead improve the quality of their per-Destination test suites, particularly by introducing record/replay style testing of Destinations. Great move. This means that the test suite of each Destination is measuring "is the Destination package adhering to its contract in how it should integrate with the 3rd party API & integrate with the shared package?" without being conflated with testing stuff that's outside of the control of code in the repo (is the 3rd party service even up, etc).

0xbadcafebee 11mo ago

These "we moved from X to Y" posts are like Dunning-Kruger humblebrags. Yes, we all lack information and make mistakes. But there's never an explanation in these posts of how they've determined their new decision is any less erroneous than their old decision. It's like they threw darts at a wall and said "cool, that's our new system design (and SDLC)". If you have not built it yourself before, and have not studied in depth an identical system, just assume you are doing the wrong thing. Otherwise you are running towards another Dunning-Kruger pit. If you have a company that writes software, please ask a professional software/systems architect to review your plans before you build. The initial decisions here would be a huge red flag to any experienced architect, and the subsequent decisions are full of hidden traps, and are setting them up for more failure. If you don't already have a very skilled architect on staff (99% chance you don't) you need to find one and consult with them. Otherwise your business will suffer from being trapped in unnecessary time-consuming expensive rework, or worse, the whole thing collapsing.

mcrk 11mo ago

They obviously read https://grugbrain.dev/#grug-on-microservices https://grugbrain.dev/#grug-on-microservices

pss314 11mo ago

A recent blog post from Docker mentions about Twilio and Amazon Prime Video seeing gains by moving away from microservices to monolith You Want Microservices, But Do You Really Need Them? https://www.docker.com/blog/do-you-really-need-microservices/ https://www.docker.com/blog/do-you-really-need-microservices...

brightstep 11mo ago

They have a monolith but struggle with individual subsystem failures bringing down the whole thing. Sounds like they would benefit from Elixir’s isolated, fail-fast architecture.

mlhpdx 11mo ago

Wow. Their experience could not be more different than mine. As I’m contemplating the first year of my startup I’ve tallied 6000 deployments and 99.997 percent uptime and a low single digit rollback percentage (MTTR in low single digit minutes and fractional, single cell impact for them so far). While I’m sure it’s possible for a solo entrepreneur to hit numbers like that with a monolith I have never done so, and haven’t see others do so. Edit: I’d love to eat the humble pie here. If you have examples of places where monoliths are updated 10-20 times a day by a small (or large) team post the link. I’ll read them all.

AlotOfReading 11mo ago
The idea of deploying to production 10-20 times per day sounds terrifying. What's the rationale for doing so? I'll assume you're not writing enough bugs that customers are reporting 10-20 new ones per day, but that leaves me confused why you would want to expose customers to that much churn. If we assume an observable issue results in a rollback and you're only rolling back 1-2% of the time (very impressive), once a month or so customers should experience observable issues across multiple subsequent days. That would turn me off making a service integral to my workflow.
- et1337 11mo ago
  If something is difficult or scary, do it more often. Smaller changes are less risky. Code that is merged but not deployed is essentially “inventory” in the factory metaphor. You want to keep inventory low. If the distance between the main branch and production is kept low, then you can always feel pretty confident that the main branch is in a good state, or at least close to one. That’s invaluable when you inevitably need to ship an emergency fix. You can just commit the fix to main instead of trying to find a known good version and patching it. And when a deployment does break something, you’ll have a much smaller diff to search for the problem.
  - AlotOfReading 11mo ago
    There's a lot of middle ground between "deploy to production 20x a day" and "deploy so infrequently that you forget how to deploy". Like, once a day? I have nothing against emergency fixes, unless you're doing them 9-19x a day. Hotfixes should be uncommon (neither rare nor standard practice).

_pdp_ 11mo ago

In my previous company we did everything as a micro service. In the company before that it was serverless on AWS! In both cases we had to come up with clever solutions to simply get by because communication between services is of a problem. It is difficult (not impossible) to keep all the contracts in sync and deployment has to be coordinated in a very specific way sometimes. The initial speed you get is soon lost further down the path due to added complexities. There was fear-driven development at play. Service ownership is a problem. Far too much meetings are spent on coordination. In my latest company everything is part of the same monolith. Yes the code is huge but it is so much easier to work with. We use a lot more unit tests then integration tests. Types make sense. Refactoring is just so easy. All the troubleshooting tools including specialised AI agents built on top of our own platform are part of the code-base which is kind of interesting because I can see how this is turning into a self-improving system. It is fascinating! We are not planning to break up the monolith unless we grow so much that is impossible to manage from a single git repository. As far as I can tell this may never happen as it is obvious that much larger projects are perfectly well maintained in the exact same way. The only downside is that build takes longer but honestly we found ways around that as well in the past and now with further improvements in the toolchains delivered by the awesome open-source communities around the world, I expect to see at least 10x improvement in deployment time in 2026. Overall, in my own assessment, the decision to go for a monolith allowed us to build and scale much faster than if we had used micro services. I hope this helps.

sethammons 11mo ago
My experience is the opposite. I worked at SendGrid for a decade and we scaled the engineering org from a dozen to over 500 operating at scale sending billions of messages daily. Micro services. Well, services. The word micro messes people up. I have also worked at half a dozen other shops with various architectures. In every monolith, people violate separation of concerns and things are tightly coupled. I have only ever seen good engineering velocity happen when teams are decoupled from one another. I have only seen this happen in a (micro) service architecture. Overall, in my own assessment, the decision to stick with a monolith has slowed down velocity and placed limits on scale at every other company I have been at and require changes towards decoupled services to be able to ship with any kind of velocity. The place I just left took 2 years, over 50 teams, and over 150 individual contributors to launch a product that required us to move over an interface for sending messages from ORM querysets to DTOs. We needed to unlock our ability to start rearchitecting the modules because before it was impossible to know the actual edges of the system and how it used the data. This was incredibly expensive and hard and would never have happened but for the ability to reach into other's domains and making assumptions making things hard. Don't couple systems. Micro services are the only arch I have seen successfully do this.

doctor_phil 11mo ago

I don't think this blog post reflects so well on this engineering team. Kudos to them to be so transparent about it though. "We had so many flaky tests that depended on 3rd parties that broke pipelines that we decided on micro-services" is not something I would put on my CV at least.

electromech 11mo ago
That seems unfair. There's a lot we don't know about the politics behind the scenes. I'd bet that the individuals who created the microservice architecture aren't the same people who re-consolidated them into one service. If true, the authors of the article are being generous to the original creators of the microservices, which I think reflects well on them for not badmouthing their predecessors.

andrewprock 11mo ago

The whole point of micro-services is to manage dependencies independently across service boundaries, using the API as the contract, not the internal libraries. Then you can implement a service in Java, Python, Rust, C++, etc, and it doesn't matter. Coupling your postgres db to your elasticsearch cluster via a hard library dependency impossibly heavy. The same insight applies to your bespoke services.

wg0 11mo ago

Thanks. It was a stupid most idea for MOST shops. I think maybe it works for AWS, Google and Netflix but everywhere in my career, I saw 90% of the problem was due to microservices. Diving system into composable parts is a very very difficult problem already and it is only foolish to introduce further network boundaries between them. Next comeback I see is away from React and SPAs as view transitions become more common.

bob1029 11mo ago

Monolith is definitely what you want to start with. Being able to ~instantly obtain a perfect list of all references to all symbols is an extraordinarily powerful capability. The stronger the type system, the more leverage you get. If you have only ever had experience with weak type systems or poor tooling, I could understand how the notion of putting everything into one compilation context seems pointless.

sethammons 11mo ago
/me raises hand. Any system that passes querysets around for one. Can't know who is using what.

sethammons 11mo ago

I left Twilio in 2018. I spent a decade at SendGrid. I spent a small time in Segment. The shitty arch is not a point against (micro)services. SendGrid, another Twilio property, uses (micro)services to great effect. Services there were fully independently deployable.

sriku 11mo ago

In a discussion I was in recently, a participant mentioned "culture eats strategy for breakfast" .. which perhaps makes sense in this context. Be bold enough to do what makes the team and the product thrive.

akoumjian 11mo ago

I humbly post this little widget to help your team decide if some functionality warrants being a separate service or not: https://mulch.dev/service-scorecard/ https://mulch.dev/service-scorecard/

waterproof 11mo ago

The only rationale given for the initial switch to microservices is this: > Initially, when the destinations were divided into separate services, all of the code lived in one repo. A huge point of frustration was that a single broken test caused tests to fail across all destinations. You kept breaking tests in main so you thought the solution was to revamp your entire codebase structure? Seems a bit backward.

kgeist 11mo ago
We had a similar problem in our monolith. Team #1 works on a feature, their code breaks tests. Team #2 works on another feature, their code is OK, but they can't move forward because of the failing tests from team #1. Plus often it takes additional time to figure out if it the tests fail because of feature #1 or feature #2, and who must fix them. We solved it by simply giving every team their own dev environment (before merging to main). So if tests break in feature #1, it doesn't break anything for feature #2 or team #2. It's all confined to their environment. It's just an additional VM + a config in CI/CD. The only downside of this is that if there are conflicts between features they won't be caught immediately (only after one of the teams finally merges to main). But in our case it wasn't a problem because different teams rarely worked on the same parts of the monorepo at the same time due to explicit code ownership.

wiradikusuma 11mo ago

> With everything running in a monolith, if a bug is introduced in one destination that causes the service to crash, the service will crash for all destinations We can have a service with 100 features, but only enable the features relevant to a given "purpose". That way, we can still have "micro services" but they're running the same code: "bla.exe -foo" and "bla.exe -bar".

tonymet 11mo ago

Your “microservice” is just a clumsy & slow symbol lookup over the network, at 1000x the cpu and 10000x the latency.

TZubiri 11mo ago

This is not the first time that an engineer working at a big company thinks they are using a monolith when in reality they are a small team in charge of a single microservice, which in turn is part of a company that definitely does not run a monolith. Last time it was an aws engineer that worked on route 53, and they dismissed microservices in a startup claiming that in AWS they ran a monolith (as in the r53 dns). Everything is a monolith if you zoom in enough and ignore everything else. Which I guess you can do when you work on a big company and are in charge of a very specific role.

necovek 11mo ago

A few thoughts: this is not really a move to a monolith. Their system is still a SOA (service-oriented architecture), just like microservices (make services as small as they can be), but with larger scope. Having 140 services managed by what sounds like one team reinforces another point that I believe should be well known by now: you use SOAs (incuding microservices) to scale teams, and not services. Eg. if a single team builds a shared library for all the 140 microservices and needs to maintain them, it's going to become very expensive quickly: you'll be using v2.3.1 in one service and v1.0.8 in another, and you won't even know yourself what API is available. Operationally, yes, you'll have to watch over 140 individual "systems" too. There are ways to mitigate this, but they have their own trade-offs (I've posted them in another comment). As per Conway's law, software architecture always follows the organizational structure, and this seems to have happened here: a single team is moving away from unneeded complexity to more effectively manage their work and produce better outcomes for the business. It is not a monolith, but properly-scoped service level (scoped to the team). This is, in my experience, the sweet spot. A single team can run and operate multiple independent services, but with growth in those services, they will look to unify, so you need to restructure the team if you don't want that to happen. This is why I don't accept "system architect" roles as those don't give you the tools to really drive the architecture how it can be driven, and I really got into "management" :)

honkycat 11mo ago

I don't care how it is done just dont rely on your database schema for data modeling and business logic

majgr 11mo ago

I have feeling that microservices improve overall design when they can live on their own, as microapps perhaps, also with their own UI. What is the point of service if it is not usable beyond its original design and just bound to other similar services?

andrewstuart 11mo ago

You can have infrastructure complexity (microservices) or trade it for development complexity (monolith). Choose one.

renewiltord 11mo ago

Is it 2018? Are you guys going to repost the MySQL DB as a queue story again? Perhaps an announcement that you’re migrating to Java 9 and what you learned about generics?

ram_shares 11mo ago

The “distributed monolith” line is the key takeaway here. Microservices only buy you something if teams can deploy, version, and reason about them independently. Once shared libraries or coordinated deploys creep in, you’ve taken on all the operational cost with none of the autonomy benefits. I’ve seen monoliths with clear module boundaries outperform microservice setups by an order of magnitude in developer throughput.

YetAnotherNick 11mo ago
If microservice or monolith is giving order of magnitude improvement in productivity, you clearly are doing something wrong, or having terrible practices.

dev_l1x_be 11mo ago

I can't believe how many times I have seen companies trying to implement microservices with multirepo and get lost in the access management, versioning and ending up producing a fragile, overly complex hot mess.

axelthegerman 11mo ago

> we had to spend time fixing the broken test even if the changes had nothing to do with the initial change. In response to this problem, it was decided to break out the code for each destination into their own repos. One could also change the way tests are run or selected. Or allow manual overrides to still deploy. Separating repos doesn't sound like the only logical solution

gostsamo 11mo ago

This is a horror story of being totally unable to understand your product and its behavior and throwing people and resources in large rewrites to only learn that you still don't understand your product and its behavior. Badly done tests used as a justification to write multiple suites of badly done tests and it is all blamed on the architecture.

joeyguerra 11mo ago

Cool.