10 ms·

Keep the monolith, but split the workloads

230 points by kiyanwang 4y ago

lawrjone 4y ago
Author here, thanks for sharing! This is a strategy I’ve used across many projects in several languages, from big Rails apps to the Go monolith we deploy at incident.io today. It’s just one way you can make a monolith much more robust without moving into separate services, which helps you keep the benefits of a monolithic application for longer. Hope people find it interesting.
- herodoturtle 4y ago
  Thank you for sharing this article! > But having spent half a decade stewarding a Ruby monolith from 20 to 200 engineers and watched its modest 10GB Postgres database grow beyond 5TB, there’s definitely a point where the pain outweighs the benefits. Woah! I for one would also love to hear your insights on scaling personnel from 20 to 200. We’re in a similar boat / anticipated growth phase (at 20 odd engineers), and whilst there’s a lot of content on this topic, I’d appreciate your practical “from the coal face” take (much like what you’ve done with this monolith article). Perhaps a follow up article? ^_^
  - lawrjone 4y ago
    Glad you enjoyed the article! The experience I reference was from my time at GoCardless, where I was a Principal SRE leading the team who were on-call for the big Rails monolith. I’ll put the topic of “what does 10x eng team scaling feel like” on my todo list for posts, but if you’re interested there’s a bunch of articles on my personal blog from this time. One that might be interesting is “Growing into Platform Engineering” about how the SRE function evolved: https://blog.lawrencejones.dev/growing-into-platform-engineering/ https://blog.lawrencejones.dev/growing-into-platform-enginee...
- rfoo 4y ago
  Question: Have you considered one separate binary (but still sharing the entire codebase) for each "workload" instead of using a flag to switch? If so, can you highlight the advantage of the flag way?
  - lawrjone 4y ago
    Yep: this is definitely a valid alternative. The reasons we went with one binary are: - We can ensure common runtime setup by only having one codepath that runs the initialisation. It's nice that every deployment definitely instantiates the Prometheus server and initialises secrets from Google Secret Manager at the same point and in the same way. - Compiling three targets adds time to our build process. Even when Go will cache much of the compilation, linking separate binaries is still another ~30s and ~300MB of binary to our resulting docker image. - It's very convenient in development to run all deployments inside of a single binary for hot-reloading purposes. This is how we develop the app locally. I don't think it makes much difference which you choose in terms of the operational elements. But for the reasons above, there are clear advantages to the single binary for our use cases.
    - rfoo 4y ago
      Thanks!
- Jupe 4y ago
  While I completely agree that the efforts to build and maintain a set of micro-services are often better leveraged by a single monolith (even one that has several "run modes" as you've done here), a few questions inevitably come up: How do you coordinate the efforts of 200 engineers on a single repo/code base? Do engineers frequently get into long/drawn-out merge sessions as common code may be modified by a number of engineers who are all trying to merge around the same time? This is actually one of the reasons I really like GOLANG: "A little copying is better than a little dependency."
  - krab 4y ago
    I have worked in a company with a monolith and about ten teams working on it. This is what helped: - Merging was automated (a robot tried to run tests against fresh master and merge only if green). - Deploy was fully automated and limited to working hours. - We added tests for problematic parts. For example static analysis for database migrations to prevent only safe actions in an automated fashion. However, if something goes wrong in some component, you have to revert and stop the deploys for everyone which sucks. I'd say around 8 - 10 deploys per day, it makes sense to start splitting the components or at least not adding new teams to the same monolith.
    - withinboredom 4y ago
      I worked on a monolith with 100+ devs working on it daily. Merging was automated as well for us, and a few unit tests were run on a production env — basically tests that asserted an engineer didn’t do anything stupid like add an infinite loop, or any of the other dozens things engineers had done that caused downtime. Deployment was done by running a script that took a lock, monitored the deployment progress, then released the lock. Sometimes you’d run the script and see people who hadn’t deployed yet, so you’d send them a message and ask if it was good to deploy. Sometimes they would say no if they found a last minute bug, so they would deploy your code for you. Surprisingly, coordination isn’t that difficult. Humans are pretty good at talking to each other.
rezonant 4y ago
This is standard practice on monolithic apps. It is effectively an N-tier architecture. On Rails it is very expected that you would deploy the same codebase to two tiers (web and worker) and simply run rails s in one and sidekiq in the other.
- heipei 4y ago
  We're doing this with our monolithic Node.js / Express application as well, but I wasn't aware that it's a common pattern. For us having a monolithic app greatly simplifies development, deployment, updating dependencies, shared code, etc. Just having a single set of third-party dependencies to update makes things so much easier.
  - lawrjone 4y ago
    It’s ‘standard practice’ for many frameworks like Rails which encourage the pattern via a separate deployment of the async worker tier. But outside of these frameworks, you won’t often see it. And especially in languages like Go which tend to avoid opinionated frameworks it’s common not to see a split like this, even when it might be really beneficial. As with most programming concepts, standard practice varies a lot depending on the tools you use. It’s why I thought this post was worth writing, in that there are many people who haven’t run Rails/etc frameworks before and may not have considered this approach!
  - Rapzid 4y ago
    It's so common Heroku has worker dynos and cron support via the scheduler add-on.
- KyeRussell 4y ago
  Ditto in Django + Celery
  - slig 4y ago
    Anyone using a more modern/lightweight alternative to Celery?
avodonosov 4y ago
I came to the same approach in a past job. One minor problem was that some functionality of that monolith was based on keeping big chunks of DB in memory. So on separated workloads, the instances dedicated to serve other functions, were keeping unnecessary data in memory, putting unneeded load on DB during startup. This could be solved by introducing some startup parameters. But for beginning this problem can be ignored.
invalidname 4y ago
Or maybe modular monolith? Modulith? https://debugagent.com/is-it-time-to-go-back-to-the-monolith https://debugagent.com/is-it-time-to-go-back-to-the-monolith
- lawrjone 4y ago
  Yep, that article is about very similar concepts but grounded in Spring as the framework. I like what they do around package imports and it looks a lot like what we do at incident.io, with some rules about which packages can import what. For people in the Ruby world who want a similar solution, Shopify provide an open-source framework called packwerk that is designed just for this: https://github.com/Shopify/packwerk https://github.com/Shopify/packwerk
- LilBytes 4y ago
  Also affectionately called macroservices. No idea if I stole it some where or came up with the idea when I was falling into slumber after losing a few days to a P0 incident, but I think it describes the N-tier architecture and what I adopt at work quite well.
dumpster_fire 4y ago
One critique I have is that this presents a binary option of either full monolith and microservices. The truth is somewhere in between where you have dependent services large enough to warrant being their own monolith being split off. Breaking up of a monolith is almost never (anecdotal observation after 15 years of seeing this argument surface in every company I've been in) a technical need, but a combination of organizational and business requirements. This is not to say it's never technical, just that it's rare for it to be a reason. Organizational: The teams are so huge that they prefer their own development cycle, deployment, and rollout safeguards. Business: Partitioning of products for different SLAs, which is difficult to guarantee when you have a different team introducing some bug every other week that takes down the servers.
- lawrjone 4y ago
  You’re right, that’s a very fair point. I should’ve made it clear that the middle ground of moderately-sized but logically separate services is very much a good world to be in. I’d actively encourage splitting a monolith into services when the candidate split would be: 1. A service that has a clear purpose that can be explained either separately from the monolithic whole or as a complementary piece 2. It will receive enough active development that the cost of upgrading/on-going investment is an acceptable percent of total work time 3. It can improve the organisations state of ‘ownership’ and better align the codebase with the team structure that own it There is always a middle ground, and I should’ve done better to explain that.
- npn 4y ago
  > This is not to say it's never technical, just that it's rare for it to be a reason. It will be a lot more technical reason from now on, when we begin to utilize more "AI" stuff, like language models. all of those stuffs are slow to start, because they need time to load the huge model, so it is better to make a service that dedicated to the AI stuff, so the development of the business side do not get hold back by the slow restart cycle. and AI is just an example, there is a lot of case when it more reasonable to split it to another service too, for example some CPU intensive tasks. Even Erlang/BEAM can't do anything about it if the code is writing in C/Zig/Rust and get called using NIF.
- 4y ago
tlrobinson 4y ago
Yes! Having worked on both monoliths and microservice spaghetti this is how I'd do it. I wonder if at scale you could see additional benefits by further splitting the web tier by route?
ahofmann 4y ago
One often finds derogatory remarks about PHP, or that the popularity of the language is declining. Interestingly, the concept of PHP prevents exactly such problems, as a monolith written in PHP only ever executes the code paths that are necessary for the respective workload. The failure that the Rails app experienced in the post simply wouldn't have happened with PHP. Especially for web applications, PHP's concept of "One Request = One freshly started process" is absolutely brilliant. So, OP is right, write more monoliths, and I would add: write them in languages that fit the stateless nature of http requests.
- lawrjone 4y ago
  Small note that the app in the post was not Rails, it’s a Golang app. You’re right in that PHP couldn’t fail like this though, as the isolation level is at a unix process, where the OS should contain all your errors.
  - stefano_c 4y ago
    The irony is that in a normal Rails app workloads are already split by default (requests and background jobs are handled by different processes that share the same codebase).
    - lawrjone 4y ago
      Yep, it's a very normal thing in the Rails world to separate things like this. For Django too. I'd encourage people to look a past just the splitting of workloads though. The other point about resource limits for shared resources across workload tiers is really key, such as having very granular database connection pools even inside of a single workload deployment. That's less common – though not unheard of – in Rails-land.
- anditherobot 4y ago
  PHP is amazing, yes the syntax is a little awkward. I love working with it. Especially with libraries/ framework that are 10+ years old Their documentation is extremely detailed. So you have more confidence that you'll deliver a working product. Instead of reading github issues to fix an obscure error message. Old does not always mean outdated.
afhammad 4y ago
> Bad code in a specific part of the codebase bringing down the whole app, as in our November incident. This is a non-issue if you're using a Elixir/Erlang monolith given its fault tolerant nature. The noisy neighbour issue (resource hogging) is still something you need to manage though. If you use something like Oban[1] (for background job queues and cron jobs), you can set both local and global limits. Local being a single node, and global across the cluster. Operating in a shared cluster (vs split workload deployments) give you the benefit of being much more efficient with your hardware. I've heard many stories of massive infra savings due to moving to an Elixir/Erlang system. 1. https://github.com/sorentwo/oban https://github.com/sorentwo/oban
sharepower123 4y ago
This post suggest to main rules: Rule 1: Never mix workloads Rule 2: Apply guardrails Applying those two rules help to decompose a monolith into smaller components to get the benefits of monolith and avoid the problem with microservices.
sethammons 4y ago
Splitting the workloads is a good, natural progression. The next pain point will be data boundaries. What tables are shared and joined against in different areas of the code? The ability to scale data access can be hard. As teams grow, shared understanding goes down and eventually data will get tangled together. Scaling systems is scaling data access. Scans, joins and aggregations will put pressure on the db and moving segments of data to their own data stores will be called for. Will you be able to untangle the monolith?
- jbverschoor 4y ago
  Doesn’t mean microservices is a good fit. You’ll need a team per serviceC but you don’t have the people who know the boundaries. Better keep it as a monolith in that case. It’s far far far worse to split something up with bad understanding about the data and access patterns than to just struggle along.
drewcoo 4y ago
> Writing code is hard enough without each function call requiring a network request And eating food is difficult enough without all that time between plate and mouth, which is why only nutrient slurry forced through a tube makes sense! Why to scale from 20 to 200 engineers we only need to add more tubes and none of that pesky dinnerware! This is just silly. It's ok to like monoliths (or tubed slurry), but there's no reason to make up nonsense to justify it.
dallyo 4y ago
We used this approach in a previous project, and can work quite well in my experience. It's wonderful to be able to avoid the insanity of microservices.
jeremycarter 4y ago
Keep the monolith but break your system up into virtual actors. https://learn.microsoft.com/en-us/dotnet/orleans/overview https://learn.microsoft.com/en-us/dotnet/orleans/overview
stavros 4y ago
This is very good advice. I've been writing an article that touches on splitting workloads as part of a strategy for having an easily maintainable monolith, it goes a bit further than splitting workloads by type (worker/server/publisher) to splitting them by domain (literally redirect some endpoints to one server and others to another). It also details some rules for restricting access between domains via clear boundaries. I should finish it and post it, I think it's a great balance between team separation and maintainability.
dontlaugh 4y ago
I've found this works well in practice. A couple other somewhat common workloads are long polling/WebSockets and web-but-slow. The latter is relevant especially when making further requests to external slow services to deal with load balancing problems.
KronisLV 4y ago
I quite like the article and the advice it presents, building what I'd call modular monoliths (that can have modules be enabled or disabled based on feature flags) is indeed a good approach for both increasing resiliency and decreasing the blast radius of various issues. However, this bit stuck out to me: > When a bad Pub/Sub message was pulled into the binary, an unhandled panic would crash the entire app, meaning web, workers and crons all died. I've never seen a production ready web framework or library that would let your entire process crash because of a bad request. There's always some error handling logic wrapping around the request handling, so that even if you let something slip by, the fallout will still be limited to the threads handling the requests and its context. Now, of course there can still be greater failures, such as issues with DB access stalling for whatever reason, leading to the HTTP requests not being processed, leading to the corresponding thread pool for those filling up, as well as any request queues maxing out and thus all new requests getting rejected. But for the most part, requests with bad data and such, short of CVEs, always have limited impact. I can imagine very few situations where things working differently wouldn't be jarring.
- lawrjone 4y ago
  Have you worked with Go codebases before? Standard practice is to wrap all your entrypoints - the start of a web request, the moment you begin to run a job - in a defer recover() which will catch panics. Sadly, recover won’t apply to any subsequently goroutine’d work. That means even if your entrypoints recover, if anything go func()s down the call stack, then if that function then panics it will bring down the entire process. We were aware of this but ended up including one by accident anyway. It’s very sad to me that you can’t apply a global process handler to ensure this type of thing doesn’t happen, but to my knowledge that isn’t possible. Worth mentioning Go doesn’t really encourage ‘frameworks’, and most Go apps compose together various libraries to make their app, rather than using something that packages everything together. Failures like this are an obvious downside to not having the well reviewed nuts-and-bolts-included framework to handle this for you.
  - Gys 4y ago
    I probably misunderstand what you wrote. Because I think a (wrapped) panic will only result in crashing the one request that caused it? For example Gin provides a middleware wrapper for handling panics: CustomRecoveryWithWriter returns a middleware for a given writer that recovers from any panics and calls the provided handle func to handle it. (https://github.com/gin-gonic/gin/blob/master/recovery.go https://github.com/gin-gonic/gin/blob/master/recovery.go)
gregwebs 4y ago
It’s odd that there’s still one binary. You can have a mostly monolithic library but produce a web binary and a worker binary, etc. Go can produce them efficiently with a single build command.
- Kinrany 4y ago
  It's more convenient to have a single binary that does all related operations depending on how it is run.
- dontlaugh 4y ago
  Agreed, you can even put all the binaries in the same image. Ultimately not a big deal.
cheeseface 4y ago
We have a similar setup with TS/Node.js app and it is very easy to work with.
jabradoodle 4y ago
I appreciate the keep your monolith argument. It seems like a lot of people have the attitude of let's start with a monolith, and switch to microservices when the monolith becomes painful. Having seen how much effort goes into getting microservices running well for any non trivial setup, I wonder what could have been achieved if 25% of that resource had gone into improving the monolith.
simo7 4y ago
Isn’t a monolith with split workloads just a monorepo?
jerrygenser 4y ago
If I have a python monolith and I have a specific set of heavy dependency like a chromium browser, I don't want my webserver to build with it only async task. Is there a straightforward way to build a separate environment in python that just had the extra dependencies while still having benefits described in the article?
speedbird 4y ago
I just don’t get the obsession with micro services (by with I mean a complete tech stack for each service). I can only see the relevance in a sub optimal org structure - huge org, merger etc. SOA, sure (hello CICS :-)). DB sharding if needed for load / resources. Otherwise logical division into sets of related services that can be their own little monolith or go full monorepo. Perhaps I'm missing something …
- andrewflnr 4y ago
  The difference with SOA being that all the services use roughly the same stack?
  - jameshart 4y ago
    I always think of the difference between SOA and microservices as being that in SOA the services often share access to the same database.
ianpurton 4y ago
Monolith is not a best practice. With most web applications you'll end up with a web part and at some point some crob jobs. In this case they also have a service that processes a queue. They should in be in the same repo, that's a Monorepo. A web app with some cronjobs is best deployed as separate executables (ideally docker containers). In this case they had everything in one exe, including threads to run the queue etc. So it looks like there solution was more complicated by aiming to be a Monolith.
mjarrett 4y ago
> No complex dev setup required, or RPC framework needed, it’s the same old monolith, just operated differently. It seems to me like "operated differently" is doing a lot of heavy lifting that often involves those same frameworks or dev/testing environments. If a monolith used to communicate between workloads in-process, now there needs to be some way for those workloads to communicate between processes, and it has to continue to work in dev. The example in the article mentions roundtripping through something like Redis or Postgres, but now doesn't your dev environment need to spin up a database? What if the communication pattern isn't a shared cache, but instead a direct function call that, say, spins up a new goroutine to process some work asynchronously? Now you need to operate and maintain either an RPC protocol or an event queue, and make sure those things can be started up and discoverable during testing and local development.
- mattmanser 4y ago
  It's pretty normal to have a local dev database on your machine. It's how we did it for decades. Are there really developers in the wild now that have never been exposed to working with a locally installed SQL instance? These days you can even set it up in docker, automatically migrate the database to the latest version and install a bunch of test data. So you can wipe the whole thing and go back to a good state if you muck it up with a bad or ill-thought-out local migration, etc. Same with redis, etc. And it's still much, much simpler than a microservice architecture.
- nickdandakis 4y ago
  You should already be spinning up caches and databases in dev anyway? I agree though that the article is missing some explicit insight into how this change is handled on the local dev environment. I'm assuming the local dev environment run commands were also updated to be these three commands, one per workload. Basically, this distinction should be represented throughout all environments, dev/test/prod
exabrial 4y ago
Current job we have a rule: we should not write no more than four layers deep of abstraction in our codebase. The thought being any more that is too much for more than a single person to keep in their head. I feel like this slays the monolith and forces developers to really think out stuff before they write it. There is of course exceptions to everything including this, but it’s served us pretty well.
- svieira 4y ago
  I'm curious as to how this plays out in practice (though I love the theory). Some questions: 1. Is this measured in _depth_, in breadth, or in both directions (and if both is the total calculated with `MAX` or `SUM`)? 2. Do you run multiple `main`s, each with this restriction? 3. Do you count third-party frameworks and libraries as layers of abstraction?
travisgriggs 4y ago
Rule 3: Consider using Elixir
mherdeg 4y ago
Best of luck to the author on their tiered monolith journey! Some things to think about: (1) How many distinct total binary versions of your monolith will you permit to run in production? Some options might include At most two globally ("current" and "new" canary/blue-green deployment, no special code on particular tiers) At most two globally, but sometimes you're willing to deploy a special build to a single tier to mitigate an emergency, with eventual convergence At most two per tier, but with no attempt to keep each tier running the same binary code (maybe you don't want to redeploy your async consumers as frequently as you redeploy your http handlers) An unlimited number (maybe you deploy customer-specific binary code to specific instances within a tier) Would you like an alert when there are too many distinct versions running in production? Who should get that alert and what should they do when it fires? (2) Does this thing deploy simultaneously everywhere? If so, is any specific person or team responsible for making sure the deployment worked ok on every tier, declaring an incident if not, and rolling back and finding an owner to resolve the issue? Will every team who owns a part of the monolith contribute someone to a shared rotation for release monitoring? (3) Suppose there is a blocking problem in one part of the monolith, for example async message processing stops working reliably. Should this block deployment or development for other teams whose changes are outside this blast radius? (4) Suppose some low-level intermittent compilation error prevents the binary from starting up 10% of the time after a certain build revision for every tier. What team will work to resolve this kind of problem? Is there a team writing telemetry and common logging for your monolith everywhere? Is there a team who will implement common operational concerns like feature flags to gate binary changes? (5) Does your monolith run in any non-production environments? Is every tier running in each environment? Does somebody publish an SLO for those environment? Is one team allowed to break everyone else in pre-prod by deploying experimental code to the monolith in some environment? Who deploys to the pre-production environment and how? (6) Suppose you discover you need to split up your workload (one kind of http request is much slower than all others and you want to separate failure domains). How much work does it take to create an additional tier -- updating deployment jobs, quality gates, and CI/CD pipelines throughout various environments, provisioning resources, setting up graphs and alerts, creating new tests? Who does this work? (7) How will you manage configuration for your monolith? Will configuration directives be delivered to every tier simultaneously? Can someone accidentally break the behavior of another team's tier with a typo or logic error in a configuration change? (8) When it comes time to split this thing into microservices or macroservices for a few years before a successor team looks at the mess and decides to reimplement a monolith, how do you set up your architecture to successfully allow a split? (9) Are you absolutely sure your tiers do what you think they do? Can API customers bypass rate limiting by pointing to the hostname of your async-worker tier? If a security vulnerability in a particular http route affects your monolith, will you remember to block the route on every tier (even the ones you think don't normally serve web traffic)?
jameshart 4y ago
* edit: I am an idiot who can’t recognize golang code (facepalm) So it’s a Ruby* app, which means code you put in the container but never use doesn’t do you any harm - it doesn’t bloat a binary or take up space in memory. but… … whenever you make a change in one of your pub-sub handlers, you still have to redeploy your web workers. … code and dependencies for your pub-sub handlers is sitting idle in your web workers, increasing your security risk and attack surface Why not just turn this into a monorepo that publishes three artifacts? Having just one container image seems like a weird thing to optimize for
- krab 4y ago
  > … whenever you make a change in one of your pub-sub handlers, you still have to redeploy your web workers. We have a similar model with Django. We deploy at least four times a day, so it's much easier to just redeploy everything than reason about what needs and what doesn't need to be redeployed.
- twosdai 4y ago
  I think that publishing 3 artifacts causes a number of other changes which expand the difficulty of management of the backend of the app. While they do allow, hopefully, independent scaling and seperate recovery processes. Separate artifacts also make it a bit more difficult to do local development, and deployments become more complex so it's not a completely clean tradeoff. I know systems kubernetes / dockerswarm which handle the management of multiple artifacts but that is now another tool needed in order to do development. So I guess what I'm saying is that "just" publishing multiple artifacts has more knock-on effects which need to be considered. Eventhough I do agree with you that it's probably for the best in the long term if these processes are in seperate containers.
  - jameshart 4y ago
    In this model there are already three different command lines to run the thing in three different modes. Deployment complexity of moving that decision one notch left to buildtime instead of startup time seems like it doesn’t increase the accidental complexity at all. In fact now you can eliminate variation in how your three different processes are started up and health checked and monitored, so instead of code that knows how to operate three different things, you have code that knows how to start and operate an arbitrary thing, and you parameterize it with your three different containers. Oh hey - that’s what kubernetes is for. Maybe all this ‘complexity’ of tooling isn’t dumb after all.
0xbadcafebee 4y ago
I never see a blog post looking at the specific requirements of an application and fitting the architecture, data model, and implementation to those requirements. "Monolith vs Microservices" is like saying "18-wheeler vs 20 Toyota Corollas". There are other forms of transportation, and using just one mode may not serve your business well. By the way, it's 2023... if you have a greenfield project, you should be using event-driven data processing. We got rid of the horse and buggy, let's please get rid of cron jobs.
- ac2u 4y ago
  >if you have a greenfield project, you should be using event-driven data processing. We got rid of the horse and buggy, let's please get rid of cron jobs. I don't see why event driven processing and cron-jobs (or any other time-based task scheduling mechanism), are strictly mutually exclusive.
  - 0xbadcafebee 4y ago
    They're not mutually exclusive. But the vast majority of tasks run by cron jobs are not time-sensitive.
    - ac2u 4y ago
      Example of these jobs people are using for cron that are not time sensitive? Every shop I’ve worked in, events driven stuff might happen on producer consumer queue mechanism, whereas stuff that needs to happen at a certain time (ie, time sensitive) uses a scheduler/cron, or a idempotent scheduling system that’s consistently polled for things that should be happening “now”.
Scubabear68 4y ago
I think this is a very pragmatic and common sense approach to getting the most out of a monolith. Whether there is 1 binary or N binaries I think really depends on context of the code, and is more of an implementation detail than something vital.
ryanthedev 4y ago
I often feel the word monolith is as poorly defined as the term microservices. What’s in between a monolith and microservices? Is it SOA?
lincpa 4y ago
[dead]
rektide 4y ago
Front end gateways make this so easy. Stand up the service a couple of times, then send different routes to different pools. This is the easiest implementation of the Bulkhead Pattern one can see, it's great. https://learn.microsoft.com/en-us/azure/architecture/patterns/bulkhead https://learn.microsoft.com/en-us/azure/architecture/pattern... I've had so many jobs where we either have some awful slow routes, or we have some latency critical routes. Thusfar I have yet to convince a single company what an incredible & vital win this would be, to create different service pools for certain classes of routes, but wow have I tried to make it happen & man what a win it would be for users.
Lightbody 4y ago
We do something similar with Reclaim.ai: It's one monolith Java repo, split into two major workloads that have the same runtime code but are configured differently. In our case, the two workflows are "api" and "jobs". This allows for operational issues on, say, jobs while still allowing user-facing APIs to be less likely to be impacted. The source code itself has many modules, but we deploy it as a single unified runtime. We're very happy with this approach and so far have only been tempted a few times to break out micro/mini-services. I suspect we will eventually break out a few services, but by the time we do, it'll be for very good reasons.
- klysm 4y ago
  When you deploy a new version does it go out to all nodes? Or are the distinct workloads versioned separately?
  - Lightbody 4y ago
    For now, it goes out to all nodes, though the pace/rollout is different. For our jobs cluster, it swaps them out pretty quickly. For our API cluster, it takes longer so that in-flight requests gracefully get handled by our load balancer before pulled out of the cluster rotation. So far that has worked well enough for us.
vivegi 4y ago
Leave the Gun, Take the Cannoli > When a bad Pub/Sub message was pulled into the binary, an unhandled panic would crash the entire app, meaning web, workers and crons all died. Several years ago while working on a Message Queue based application, we had a poison message, one where the consumer process would read the message and then die because the message was malformed and then the message would get back to the queue and due to the FIFO nature of the queue, the retries will also fail. Ideally, there should be enough checks to ensure that a poison message can never be created. But if edge cases cause this issue, the handler should have some way of knowing that it is retrying the message for the n-th time and then if it fails, move it to a dead-letter-queue and report failure on handling that message. Unfortunately, this pattern of issues are quite common. So, we fully expect this and design around it. Then, any message in a dead-letter-queue gives you an example of a crash-inducing message which can then be debugged for root-cause.
- Arbortheus 4y ago
  I suppose PubSub is at least slightly better than this because it's not a perfect queue. I.E. messages that are not acknowledged will be retried with exponential backoff. That at least gives some time to process some stuff before you encounter the poison message again. Whereas with a FIFO queue you're completely screwed.

230 points by kiyanwang 4y ago

Author here, thanks for sharing! This is a strategy I’ve used across many projects in several languages, from big Rails apps to the Go monolith we deploy at incident.io today. It’s just one way you can make a monolith much more robust without moving into separate services, which helps you keep the benefits of a monolithic application for longer. Hope people find it interesting.

herodoturtle 4y ago
Thank you for sharing this article! > But having spent half a decade stewarding a Ruby monolith from 20 to 200 engineers and watched its modest 10GB Postgres database grow beyond 5TB, there’s definitely a point where the pain outweighs the benefits. Woah! I for one would also love to hear your insights on scaling personnel from 20 to 200. We’re in a similar boat / anticipated growth phase (at 20 odd engineers), and whilst there’s a lot of content on this topic, I’d appreciate your practical “from the coal face” take (much like what you’ve done with this monolith article). Perhaps a follow up article? ^_^
- lawrjone 4y ago
  Glad you enjoyed the article! The experience I reference was from my time at GoCardless, where I was a Principal SRE leading the team who were on-call for the big Rails monolith. I’ll put the topic of “what does 10x eng team scaling feel like” on my todo list for posts, but if you’re interested there’s a bunch of articles on my personal blog from this time. One that might be interesting is “Growing into Platform Engineering” about how the SRE function evolved: https://blog.lawrencejones.dev/growing-into-platform-engineering/ https://blog.lawrencejones.dev/growing-into-platform-enginee...
rfoo 4y ago
Question: Have you considered one separate binary (but still sharing the entire codebase) for each "workload" instead of using a flag to switch? If so, can you highlight the advantage of the flag way?
- lawrjone 4y ago
  Yep: this is definitely a valid alternative. The reasons we went with one binary are: - We can ensure common runtime setup by only having one codepath that runs the initialisation. It's nice that every deployment definitely instantiates the Prometheus server and initialises secrets from Google Secret Manager at the same point and in the same way. - Compiling three targets adds time to our build process. Even when Go will cache much of the compilation, linking separate binaries is still another ~30s and ~300MB of binary to our resulting docker image. - It's very convenient in development to run all deployments inside of a single binary for hot-reloading purposes. This is how we develop the app locally. I don't think it makes much difference which you choose in terms of the operational elements. But for the reasons above, there are clear advantages to the single binary for our use cases.
  - rfoo 4y ago
    Thanks!
Jupe 4y ago
While I completely agree that the efforts to build and maintain a set of micro-services are often better leveraged by a single monolith (even one that has several "run modes" as you've done here), a few questions inevitably come up: How do you coordinate the efforts of 200 engineers on a single repo/code base? Do engineers frequently get into long/drawn-out merge sessions as common code may be modified by a number of engineers who are all trying to merge around the same time? This is actually one of the reasons I really like GOLANG: "A little copying is better than a little dependency."
- krab 4y ago
  I have worked in a company with a monolith and about ten teams working on it. This is what helped: - Merging was automated (a robot tried to run tests against fresh master and merge only if green). - Deploy was fully automated and limited to working hours. - We added tests for problematic parts. For example static analysis for database migrations to prevent only safe actions in an automated fashion. However, if something goes wrong in some component, you have to revert and stop the deploys for everyone which sucks. I'd say around 8 - 10 deploys per day, it makes sense to start splitting the components or at least not adding new teams to the same monolith.
  - withinboredom 4y ago
    I worked on a monolith with 100+ devs working on it daily. Merging was automated as well for us, and a few unit tests were run on a production env — basically tests that asserted an engineer didn’t do anything stupid like add an infinite loop, or any of the other dozens things engineers had done that caused downtime. Deployment was done by running a script that took a lock, monitored the deployment progress, then released the lock. Sometimes you’d run the script and see people who hadn’t deployed yet, so you’d send them a message and ask if it was good to deploy. Sometimes they would say no if they found a last minute bug, so they would deploy your code for you. Surprisingly, coordination isn’t that difficult. Humans are pretty good at talking to each other.

rezonant 4y ago

This is standard practice on monolithic apps. It is effectively an N-tier architecture. On Rails it is very expected that you would deploy the same codebase to two tiers (web and worker) and simply run rails s in one and sidekiq in the other.

heipei 4y ago
We're doing this with our monolithic Node.js / Express application as well, but I wasn't aware that it's a common pattern. For us having a monolithic app greatly simplifies development, deployment, updating dependencies, shared code, etc. Just having a single set of third-party dependencies to update makes things so much easier.
- lawrjone 4y ago
  It’s ‘standard practice’ for many frameworks like Rails which encourage the pattern via a separate deployment of the async worker tier. But outside of these frameworks, you won’t often see it. And especially in languages like Go which tend to avoid opinionated frameworks it’s common not to see a split like this, even when it might be really beneficial. As with most programming concepts, standard practice varies a lot depending on the tools you use. It’s why I thought this post was worth writing, in that there are many people who haven’t run Rails/etc frameworks before and may not have considered this approach!
- Rapzid 4y ago
  It's so common Heroku has worker dynos and cron support via the scheduler add-on.
KyeRussell 4y ago
Ditto in Django + Celery
- slig 4y ago
  Anyone using a more modern/lightweight alternative to Celery?

avodonosov 4y ago

I came to the same approach in a past job. One minor problem was that some functionality of that monolith was based on keeping big chunks of DB in memory. So on separated workloads, the instances dedicated to serve other functions, were keeping unnecessary data in memory, putting unneeded load on DB during startup. This could be solved by introducing some startup parameters. But for beginning this problem can be ignored.

invalidname 4y ago

Or maybe modular monolith? Modulith? https://debugagent.com/is-it-time-to-go-back-to-the-monolith https://debugagent.com/is-it-time-to-go-back-to-the-monolith

lawrjone 4y ago
Yep, that article is about very similar concepts but grounded in Spring as the framework. I like what they do around package imports and it looks a lot like what we do at incident.io, with some rules about which packages can import what. For people in the Ruby world who want a similar solution, Shopify provide an open-source framework called packwerk that is designed just for this: https://github.com/Shopify/packwerk https://github.com/Shopify/packwerk
LilBytes 4y ago
Also affectionately called macroservices. No idea if I stole it some where or came up with the idea when I was falling into slumber after losing a few days to a P0 incident, but I think it describes the N-tier architecture and what I adopt at work quite well.

dumpster_fire 4y ago

One critique I have is that this presents a binary option of either full monolith and microservices. The truth is somewhere in between where you have dependent services large enough to warrant being their own monolith being split off. Breaking up of a monolith is almost never (anecdotal observation after 15 years of seeing this argument surface in every company I've been in) a technical need, but a combination of organizational and business requirements. This is not to say it's never technical, just that it's rare for it to be a reason. Organizational: The teams are so huge that they prefer their own development cycle, deployment, and rollout safeguards. Business: Partitioning of products for different SLAs, which is difficult to guarantee when you have a different team introducing some bug every other week that takes down the servers.

lawrjone 4y ago
You’re right, that’s a very fair point. I should’ve made it clear that the middle ground of moderately-sized but logically separate services is very much a good world to be in. I’d actively encourage splitting a monolith into services when the candidate split would be: 1. A service that has a clear purpose that can be explained either separately from the monolithic whole or as a complementary piece 2. It will receive enough active development that the cost of upgrading/on-going investment is an acceptable percent of total work time 3. It can improve the organisations state of ‘ownership’ and better align the codebase with the team structure that own it There is always a middle ground, and I should’ve done better to explain that.
npn 4y ago
> This is not to say it's never technical, just that it's rare for it to be a reason. It will be a lot more technical reason from now on, when we begin to utilize more "AI" stuff, like language models. all of those stuffs are slow to start, because they need time to load the huge model, so it is better to make a service that dedicated to the AI stuff, so the development of the business side do not get hold back by the slow restart cycle. and AI is just an example, there is a lot of case when it more reasonable to split it to another service too, for example some CPU intensive tasks. Even Erlang/BEAM can't do anything about it if the code is writing in C/Zig/Rust and get called using NIF.
4y ago

tlrobinson 4y ago

Yes! Having worked on both monoliths and microservice spaghetti this is how I'd do it. I wonder if at scale you could see additional benefits by further splitting the web tier by route?

ahofmann 4y ago

One often finds derogatory remarks about PHP, or that the popularity of the language is declining. Interestingly, the concept of PHP prevents exactly such problems, as a monolith written in PHP only ever executes the code paths that are necessary for the respective workload. The failure that the Rails app experienced in the post simply wouldn't have happened with PHP. Especially for web applications, PHP's concept of "One Request = One freshly started process" is absolutely brilliant. So, OP is right, write more monoliths, and I would add: write them in languages that fit the stateless nature of http requests.

lawrjone 4y ago
Small note that the app in the post was not Rails, it’s a Golang app. You’re right in that PHP couldn’t fail like this though, as the isolation level is at a unix process, where the OS should contain all your errors.
- stefano_c 4y ago
  The irony is that in a normal Rails app workloads are already split by default (requests and background jobs are handled by different processes that share the same codebase).
  - lawrjone 4y ago
    Yep, it's a very normal thing in the Rails world to separate things like this. For Django too. I'd encourage people to look a past just the splitting of workloads though. The other point about resource limits for shared resources across workload tiers is really key, such as having very granular database connection pools even inside of a single workload deployment. That's less common – though not unheard of – in Rails-land.
anditherobot 4y ago
PHP is amazing, yes the syntax is a little awkward. I love working with it. Especially with libraries/ framework that are 10+ years old Their documentation is extremely detailed. So you have more confidence that you'll deliver a working product. Instead of reading github issues to fix an obscure error message. Old does not always mean outdated.

afhammad 4y ago

> Bad code in a specific part of the codebase bringing down the whole app, as in our November incident. This is a non-issue if you're using a Elixir/Erlang monolith given its fault tolerant nature. The noisy neighbour issue (resource hogging) is still something you need to manage though. If you use something like Oban[1] (for background job queues and cron jobs), you can set both local and global limits. Local being a single node, and global across the cluster. Operating in a shared cluster (vs split workload deployments) give you the benefit of being much more efficient with your hardware. I've heard many stories of massive infra savings due to moving to an Elixir/Erlang system. 1. https://github.com/sorentwo/oban https://github.com/sorentwo/oban

sharepower123 4y ago

This post suggest to main rules: Rule 1: Never mix workloads Rule 2: Apply guardrails Applying those two rules help to decompose a monolith into smaller components to get the benefits of monolith and avoid the problem with microservices.

sethammons 4y ago

Splitting the workloads is a good, natural progression. The next pain point will be data boundaries. What tables are shared and joined against in different areas of the code? The ability to scale data access can be hard. As teams grow, shared understanding goes down and eventually data will get tangled together. Scaling systems is scaling data access. Scans, joins and aggregations will put pressure on the db and moving segments of data to their own data stores will be called for. Will you be able to untangle the monolith?

jbverschoor 4y ago
Doesn’t mean microservices is a good fit. You’ll need a team per serviceC but you don’t have the people who know the boundaries. Better keep it as a monolith in that case. It’s far far far worse to split something up with bad understanding about the data and access patterns than to just struggle along.

drewcoo 4y ago

> Writing code is hard enough without each function call requiring a network request And eating food is difficult enough without all that time between plate and mouth, which is why only nutrient slurry forced through a tube makes sense! Why to scale from 20 to 200 engineers we only need to add more tubes and none of that pesky dinnerware! This is just silly. It's ok to like monoliths (or tubed slurry), but there's no reason to make up nonsense to justify it.

dallyo 4y ago

We used this approach in a previous project, and can work quite well in my experience. It's wonderful to be able to avoid the insanity of microservices.

jeremycarter 4y ago

Keep the monolith but break your system up into virtual actors. https://learn.microsoft.com/en-us/dotnet/orleans/overview https://learn.microsoft.com/en-us/dotnet/orleans/overview

stavros 4y ago

This is very good advice. I've been writing an article that touches on splitting workloads as part of a strategy for having an easily maintainable monolith, it goes a bit further than splitting workloads by type (worker/server/publisher) to splitting them by domain (literally redirect some endpoints to one server and others to another). It also details some rules for restricting access between domains via clear boundaries. I should finish it and post it, I think it's a great balance between team separation and maintainability.

dontlaugh 4y ago

I've found this works well in practice. A couple other somewhat common workloads are long polling/WebSockets and web-but-slow. The latter is relevant especially when making further requests to external slow services to deal with load balancing problems.

KronisLV 4y ago

I quite like the article and the advice it presents, building what I'd call modular monoliths (that can have modules be enabled or disabled based on feature flags) is indeed a good approach for both increasing resiliency and decreasing the blast radius of various issues. However, this bit stuck out to me: > When a bad Pub/Sub message was pulled into the binary, an unhandled panic would crash the entire app, meaning web, workers and crons all died. I've never seen a production ready web framework or library that would let your entire process crash because of a bad request. There's always some error handling logic wrapping around the request handling, so that even if you let something slip by, the fallout will still be limited to the threads handling the requests and its context. Now, of course there can still be greater failures, such as issues with DB access stalling for whatever reason, leading to the HTTP requests not being processed, leading to the corresponding thread pool for those filling up, as well as any request queues maxing out and thus all new requests getting rejected. But for the most part, requests with bad data and such, short of CVEs, always have limited impact. I can imagine very few situations where things working differently wouldn't be jarring.

lawrjone 4y ago
Have you worked with Go codebases before? Standard practice is to wrap all your entrypoints - the start of a web request, the moment you begin to run a job - in a defer recover() which will catch panics. Sadly, recover won’t apply to any subsequently goroutine’d work. That means even if your entrypoints recover, if anything go func()s down the call stack, then if that function then panics it will bring down the entire process. We were aware of this but ended up including one by accident anyway. It’s very sad to me that you can’t apply a global process handler to ensure this type of thing doesn’t happen, but to my knowledge that isn’t possible. Worth mentioning Go doesn’t really encourage ‘frameworks’, and most Go apps compose together various libraries to make their app, rather than using something that packages everything together. Failures like this are an obvious downside to not having the well reviewed nuts-and-bolts-included framework to handle this for you.
- Gys 4y ago
  I probably misunderstand what you wrote. Because I think a (wrapped) panic will only result in crashing the one request that caused it? For example Gin provides a middleware wrapper for handling panics: CustomRecoveryWithWriter returns a middleware for a given writer that recovers from any panics and calls the provided handle func to handle it. (https://github.com/gin-gonic/gin/blob/master/recovery.go https://github.com/gin-gonic/gin/blob/master/recovery.go)

gregwebs 4y ago

It’s odd that there’s still one binary. You can have a mostly monolithic library but produce a web binary and a worker binary, etc. Go can produce them efficiently with a single build command.

Kinrany 4y ago
It's more convenient to have a single binary that does all related operations depending on how it is run.
dontlaugh 4y ago
Agreed, you can even put all the binaries in the same image. Ultimately not a big deal.

cheeseface 4y ago

We have a similar setup with TS/Node.js app and it is very easy to work with.

jabradoodle 4y ago

I appreciate the keep your monolith argument. It seems like a lot of people have the attitude of let's start with a monolith, and switch to microservices when the monolith becomes painful. Having seen how much effort goes into getting microservices running well for any non trivial setup, I wonder what could have been achieved if 25% of that resource had gone into improving the monolith.

simo7 4y ago

Isn’t a monolith with split workloads just a monorepo?

jerrygenser 4y ago

If I have a python monolith and I have a specific set of heavy dependency like a chromium browser, I don't want my webserver to build with it only async task. Is there a straightforward way to build a separate environment in python that just had the extra dependencies while still having benefits described in the article?

speedbird 4y ago

I just don’t get the obsession with micro services (by with I mean a complete tech stack for each service). I can only see the relevance in a sub optimal org structure - huge org, merger etc. SOA, sure (hello CICS :-)). DB sharding if needed for load / resources. Otherwise logical division into sets of related services that can be their own little monolith or go full monorepo. Perhaps I'm missing something …

andrewflnr 4y ago
The difference with SOA being that all the services use roughly the same stack?
- jameshart 4y ago
  I always think of the difference between SOA and microservices as being that in SOA the services often share access to the same database.

ianpurton 4y ago

Monolith is not a best practice. With most web applications you'll end up with a web part and at some point some crob jobs. In this case they also have a service that processes a queue. They should in be in the same repo, that's a Monorepo. A web app with some cronjobs is best deployed as separate executables (ideally docker containers). In this case they had everything in one exe, including threads to run the queue etc. So it looks like there solution was more complicated by aiming to be a Monolith.

mjarrett 4y ago

> No complex dev setup required, or RPC framework needed, it’s the same old monolith, just operated differently. It seems to me like "operated differently" is doing a lot of heavy lifting that often involves those same frameworks or dev/testing environments. If a monolith used to communicate between workloads in-process, now there needs to be some way for those workloads to communicate between processes, and it has to continue to work in dev. The example in the article mentions roundtripping through something like Redis or Postgres, but now doesn't your dev environment need to spin up a database? What if the communication pattern isn't a shared cache, but instead a direct function call that, say, spins up a new goroutine to process some work asynchronously? Now you need to operate and maintain either an RPC protocol or an event queue, and make sure those things can be started up and discoverable during testing and local development.

mattmanser 4y ago
It's pretty normal to have a local dev database on your machine. It's how we did it for decades. Are there really developers in the wild now that have never been exposed to working with a locally installed SQL instance? These days you can even set it up in docker, automatically migrate the database to the latest version and install a bunch of test data. So you can wipe the whole thing and go back to a good state if you muck it up with a bad or ill-thought-out local migration, etc. Same with redis, etc. And it's still much, much simpler than a microservice architecture.
nickdandakis 4y ago
You should already be spinning up caches and databases in dev anyway? I agree though that the article is missing some explicit insight into how this change is handled on the local dev environment. I'm assuming the local dev environment run commands were also updated to be these three commands, one per workload. Basically, this distinction should be represented throughout all environments, dev/test/prod

exabrial 4y ago

Current job we have a rule: we should not write no more than four layers deep of abstraction in our codebase. The thought being any more that is too much for more than a single person to keep in their head. I feel like this slays the monolith and forces developers to really think out stuff before they write it. There is of course exceptions to everything including this, but it’s served us pretty well.

svieira 4y ago
I'm curious as to how this plays out in practice (though I love the theory). Some questions: 1. Is this measured in _depth_, in breadth, or in both directions (and if both is the total calculated with `MAX` or `SUM`)? 2. Do you run multiple `main`s, each with this restriction? 3. Do you count third-party frameworks and libraries as layers of abstraction?

travisgriggs 4y ago

Rule 3: Consider using Elixir

mherdeg 4y ago

Best of luck to the author on their tiered monolith journey! Some things to think about: (1) How many distinct total binary versions of your monolith will you permit to run in production? Some options might include At most two globally ("current" and "new" canary/blue-green deployment, no special code on particular tiers) At most two globally, but sometimes you're willing to deploy a special build to a single tier to mitigate an emergency, with eventual convergence At most two per tier, but with no attempt to keep each tier running the same binary code (maybe you don't want to redeploy your async consumers as frequently as you redeploy your http handlers) An unlimited number (maybe you deploy customer-specific binary code to specific instances within a tier) Would you like an alert when there are too many distinct versions running in production? Who should get that alert and what should they do when it fires? (2) Does this thing deploy simultaneously everywhere? If so, is any specific person or team responsible for making sure the deployment worked ok on every tier, declaring an incident if not, and rolling back and finding an owner to resolve the issue? Will every team who owns a part of the monolith contribute someone to a shared rotation for release monitoring? (3) Suppose there is a blocking problem in one part of the monolith, for example async message processing stops working reliably. Should this block deployment or development for other teams whose changes are outside this blast radius? (4) Suppose some low-level intermittent compilation error prevents the binary from starting up 10% of the time after a certain build revision for every tier. What team will work to resolve this kind of problem? Is there a team writing telemetry and common logging for your monolith everywhere? Is there a team who will implement common operational concerns like feature flags to gate binary changes? (5) Does your monolith run in any non-production environments? Is every tier running in each environment? Does somebody publish an SLO for those environment? Is one team allowed to break everyone else in pre-prod by deploying experimental code to the monolith in some environment? Who deploys to the pre-production environment and how? (6) Suppose you discover you need to split up your workload (one kind of http request is much slower than all others and you want to separate failure domains). How much work does it take to create an additional tier -- updating deployment jobs, quality gates, and CI/CD pipelines throughout various environments, provisioning resources, setting up graphs and alerts, creating new tests? Who does this work? (7) How will you manage configuration for your monolith? Will configuration directives be delivered to every tier simultaneously? Can someone accidentally break the behavior of another team's tier with a typo or logic error in a configuration change? (8) When it comes time to split this thing into microservices or macroservices for a few years before a successor team looks at the mess and decides to reimplement a monolith, how do you set up your architecture to successfully allow a split? (9) Are you absolutely sure your tiers do what you think they do? Can API customers bypass rate limiting by pointing to the hostname of your async-worker tier? If a security vulnerability in a particular http route affects your monolith, will you remember to block the route on every tier (even the ones you think don't normally serve web traffic)?

jameshart 4y ago

* edit: I am an idiot who can’t recognize golang code (facepalm) So it’s a Ruby* app, which means code you put in the container but never use doesn’t do you any harm - it doesn’t bloat a binary or take up space in memory. but… … whenever you make a change in one of your pub-sub handlers, you still have to redeploy your web workers. … code and dependencies for your pub-sub handlers is sitting idle in your web workers, increasing your security risk and attack surface Why not just turn this into a monorepo that publishes three artifacts? Having just one container image seems like a weird thing to optimize for

krab 4y ago
> … whenever you make a change in one of your pub-sub handlers, you still have to redeploy your web workers. We have a similar model with Django. We deploy at least four times a day, so it's much easier to just redeploy everything than reason about what needs and what doesn't need to be redeployed.
twosdai 4y ago
I think that publishing 3 artifacts causes a number of other changes which expand the difficulty of management of the backend of the app. While they do allow, hopefully, independent scaling and seperate recovery processes. Separate artifacts also make it a bit more difficult to do local development, and deployments become more complex so it's not a completely clean tradeoff. I know systems kubernetes / dockerswarm which handle the management of multiple artifacts but that is now another tool needed in order to do development. So I guess what I'm saying is that "just" publishing multiple artifacts has more knock-on effects which need to be considered. Eventhough I do agree with you that it's probably for the best in the long term if these processes are in seperate containers.
- jameshart 4y ago
  In this model there are already three different command lines to run the thing in three different modes. Deployment complexity of moving that decision one notch left to buildtime instead of startup time seems like it doesn’t increase the accidental complexity at all. In fact now you can eliminate variation in how your three different processes are started up and health checked and monitored, so instead of code that knows how to operate three different things, you have code that knows how to start and operate an arbitrary thing, and you parameterize it with your three different containers. Oh hey - that’s what kubernetes is for. Maybe all this ‘complexity’ of tooling isn’t dumb after all.

0xbadcafebee 4y ago

I never see a blog post looking at the specific requirements of an application and fitting the architecture, data model, and implementation to those requirements. "Monolith vs Microservices" is like saying "18-wheeler vs 20 Toyota Corollas". There are other forms of transportation, and using just one mode may not serve your business well. By the way, it's 2023... if you have a greenfield project, you should be using event-driven data processing. We got rid of the horse and buggy, let's please get rid of cron jobs.

ac2u 4y ago
>if you have a greenfield project, you should be using event-driven data processing. We got rid of the horse and buggy, let's please get rid of cron jobs. I don't see why event driven processing and cron-jobs (or any other time-based task scheduling mechanism), are strictly mutually exclusive.
- 0xbadcafebee 4y ago
  They're not mutually exclusive. But the vast majority of tasks run by cron jobs are not time-sensitive.
  - ac2u 4y ago
    Example of these jobs people are using for cron that are not time sensitive? Every shop I’ve worked in, events driven stuff might happen on producer consumer queue mechanism, whereas stuff that needs to happen at a certain time (ie, time sensitive) uses a scheduler/cron, or a idempotent scheduling system that’s consistently polled for things that should be happening “now”.

Scubabear68 4y ago

I think this is a very pragmatic and common sense approach to getting the most out of a monolith. Whether there is 1 binary or N binaries I think really depends on context of the code, and is more of an implementation detail than something vital.

ryanthedev 4y ago

I often feel the word monolith is as poorly defined as the term microservices. What’s in between a monolith and microservices? Is it SOA?

lincpa 4y ago

[dead]

rektide 4y ago

Front end gateways make this so easy. Stand up the service a couple of times, then send different routes to different pools. This is the easiest implementation of the Bulkhead Pattern one can see, it's great. https://learn.microsoft.com/en-us/azure/architecture/patterns/bulkhead https://learn.microsoft.com/en-us/azure/architecture/pattern... I've had so many jobs where we either have some awful slow routes, or we have some latency critical routes. Thusfar I have yet to convince a single company what an incredible & vital win this would be, to create different service pools for certain classes of routes, but wow have I tried to make it happen & man what a win it would be for users.

Lightbody 4y ago

We do something similar with Reclaim.ai: It's one monolith Java repo, split into two major workloads that have the same runtime code but are configured differently. In our case, the two workflows are "api" and "jobs". This allows for operational issues on, say, jobs while still allowing user-facing APIs to be less likely to be impacted. The source code itself has many modules, but we deploy it as a single unified runtime. We're very happy with this approach and so far have only been tempted a few times to break out micro/mini-services. I suspect we will eventually break out a few services, but by the time we do, it'll be for very good reasons.

klysm 4y ago
When you deploy a new version does it go out to all nodes? Or are the distinct workloads versioned separately?
- Lightbody 4y ago
  For now, it goes out to all nodes, though the pace/rollout is different. For our jobs cluster, it swaps them out pretty quickly. For our API cluster, it takes longer so that in-flight requests gracefully get handled by our load balancer before pulled out of the cluster rotation. So far that has worked well enough for us.

vivegi 4y ago

Leave the Gun, Take the Cannoli > When a bad Pub/Sub message was pulled into the binary, an unhandled panic would crash the entire app, meaning web, workers and crons all died. Several years ago while working on a Message Queue based application, we had a poison message, one where the consumer process would read the message and then die because the message was malformed and then the message would get back to the queue and due to the FIFO nature of the queue, the retries will also fail. Ideally, there should be enough checks to ensure that a poison message can never be created. But if edge cases cause this issue, the handler should have some way of knowing that it is retrying the message for the n-th time and then if it fails, move it to a dead-letter-queue and report failure on handling that message. Unfortunately, this pattern of issues are quite common. So, we fully expect this and design around it. Then, any message in a dead-letter-queue gives you an example of a crash-inducing message which can then be debugged for root-cause.

Arbortheus 4y ago
I suppose PubSub is at least slightly better than this because it's not a perfect queue. I.E. messages that are not acknowledged will be retried with exponential backoff. That at least gives some time to process some stuff before you encounter the poison message again. Whereas with a FIFO queue you're completely screwed.