13 ms·

I can offer one data point. This is from purely startup-based experience (seed to Series A). A while ago I moved from microservices to monolith because they we

I can offer one data point. This is from purely startup-based experience (seed to Series A). A while ago I moved from microservices to monolith because they were too complicated and had a lot of duplicated code. Without microservices there's less need for a message queue. For async stuff, I used RabbitMQ for one project, but it just felt...old and over-architected? And a lot of the tooling around it (celery) just wasn't as good as the modern stuff built around redis (bullmq). For multi-step, DAG-style processes, I prefer to KISS and just do that all in a single, large job if I can, or break it into a small number of jobs. If I REALLY needed a DAG thing, there are tools out there that are specifically built for that (Airflow). But I hear they're difficult to debug issues in, so would avoid at most costs. I have run into scaling issues with redis, because their multi-node architectures are just ridiculously over-complicated, and so I stick with single-node. But sharding by hand is fine for me, and works well.

kypro 2y ago
In my experience monoliths don't reduce complexity, they just shift it. The main issue with monoliths is that they don't have clear and explicit separation of concern between domain concerns, therefore it's very easy for your monolith codebase to devolve into a mess of highly interconnected spaghetti code with time. This is especially true if you're building something large with a lot of developers who don't necessarily understand all of the domain complexity of the code they're touching. Monoliths imo are better for smaller projects with a few devs, but otherwise within a few years most of the time you'll regret building a monolith. I also disagree with the duplicated code point. I don't understand why that would be a significant problem assuming you're using the same language and sharing packages between projects. This isn't a problem I've ever had while working on microservices anyway. I'd also debate whether they're anymore more complex than monoliths on average. My favourite thing about microservice architecture is how simple individual microservices are to understand and contribute to. The architecture and provisioning of microservices can be more complicated, but from the perspective of a developer working on a microservice it should be much simpler to work on compared to a monolith.
- Noneofya 2y ago
  >My favourite thing about microservice architecture is how simple individual microservices are to understand and contribute to. I generally agree with your stance and I would add that I find the whole simplistic „microservices suck“ talking point as nonsensical as viewing them as a panacea. They do solve a few specific (for most companies, mostly organizational/human factors because scale and redundancy don’t matter that much) problems that are harder to solve with monoliths. I still think this point is a bit misleading, because yes, the components become simpler, but their interaction becomes more complex and that complexity is now less apparent. See: >The architecture and provisioning of microservices can be more complicated, but from the perspective of a developer working on a microservice it should be much simpler to work on compared to a monolith. I think that perspective often doesn’t match reality. Maybe most microservice domains I‘ve seen had poorly separated domains, but changes small enough to ignore the interactions with other services have been rare in my experience.
- tstrimple 2y ago
  Microservices necessarily add more complexity and overhead when compared to a monolith. Just the fact that you have to orchestrate N services instead of just pressing run on a single project demonstrates some of the additional complexity.
  - erulabs 2y ago
    Counterpoint: a monolith usually contains a complex init system which allows multiple ways of running the codebase. Microservices can avoid at least that one complexity.
    - devoutsalsa 2y ago
      Another advantage of microservices is that you can avoid the overhead of multiple services by having one really big microservice.
    - The_Colonel 2y ago
      You mean like profiles? Monolith can run like a front service or background worker depending on the config? In a technical sense it is a complexity, but IME pales in comparison with the alternative of having to manage multiple services. I actually really like this model, it's pretty flexible. You can still have specialized server instances dedicated for certain tasks, but you have one codebase. What's very sweet is that for local development, you just run a single application which (with all enabled profiles) can fulfill all the roles at the same time.
appplication 2y ago
To your comment on Airflow, I’ve been around that block a few times. I’ve found Airflow (and really any orchestration) be the most manageable when it’s nearly devoid of all logic to the point of DAGs being little more than a series of function or API calls, with each of those responsible for managing state transfer to the next call (as opposed to relying on orchestration to do so). For example, you need some ETL to happen every day. Instead of having your pipeline logic inside an airflow task, you put your logic in a library, where you can test and establish boundaries for this behavior in isolation, and compose this logic portably into any system that can accept your library code. When you need to orchestrate, you just call this function inside an airflow task. This has a few benefits. You now decouple, to a significant extent, your logic and state transfer from your orchestration. That means if you want to debug your DAG, you don’t need to do it in Airflow. You can take the same series of function calls and run them, for example, sequentially in a notebook and you would achieve the same effect. This also can reveal just how little logic you really need in orchestration. There are some other tricks to making this work really well, such as reducing dependency injection to primatives only where possible, and focusing on decoupling logic from configuration. Some of this is pretty standard, but I’ve seen teams not have a strong philosophy on this and then struggle with maintaining clean orchestration interfaces.
- Noneofya 2y ago
  Thanks, this was really helpful.
- RangerScience 2y ago
  Helpful comment! If I could pick your brain... I'm looking at a green field implementation of a task system, for human tasks - people need to do a thing, and then mark that they've done it, and that "unlocks" subsequent human tasks, and near as I can tell the overall task flow is a DAG. I'm currently considering how (if?) to allow for complex logic about things like which tasks are present in the overall DAG - things like skipping a node based on some criteria (which, it occurs to me in typing this up, can benefit from your above advice, as that can just be a configured function call that returns skip/no-skip) - and, well... thoughts? (:

kypro 2y ago

In my experience monoliths don't reduce complexity, they just shift it. The main issue with monoliths is that they don't have clear and explicit separation of concern between domain concerns, therefore it's very easy for your monolith codebase to devolve into a mess of highly interconnected spaghetti code with time. This is especially true if you're building something large with a lot of developers who don't necessarily understand all of the domain complexity of the code they're touching. Monoliths imo are better for smaller projects with a few devs, but otherwise within a few years most of the time you'll regret building a monolith. I also disagree with the duplicated code point. I don't understand why that would be a significant problem assuming you're using the same language and sharing packages between projects. This isn't a problem I've ever had while working on microservices anyway. I'd also debate whether they're anymore more complex than monoliths on average. My favourite thing about microservice architecture is how simple individual microservices are to understand and contribute to. The architecture and provisioning of microservices can be more complicated, but from the perspective of a developer working on a microservice it should be much simpler to work on compared to a monolith.

Noneofya 2y ago
>My favourite thing about microservice architecture is how simple individual microservices are to understand and contribute to. I generally agree with your stance and I would add that I find the whole simplistic „microservices suck“ talking point as nonsensical as viewing them as a panacea. They do solve a few specific (for most companies, mostly organizational/human factors because scale and redundancy don’t matter that much) problems that are harder to solve with monoliths. I still think this point is a bit misleading, because yes, the components become simpler, but their interaction becomes more complex and that complexity is now less apparent. See: >The architecture and provisioning of microservices can be more complicated, but from the perspective of a developer working on a microservice it should be much simpler to work on compared to a monolith. I think that perspective often doesn’t match reality. Maybe most microservice domains I‘ve seen had poorly separated domains, but changes small enough to ignore the interactions with other services have been rare in my experience.
tstrimple 2y ago
Microservices necessarily add more complexity and overhead when compared to a monolith. Just the fact that you have to orchestrate N services instead of just pressing run on a single project demonstrates some of the additional complexity.
- erulabs 2y ago
  Counterpoint: a monolith usually contains a complex init system which allows multiple ways of running the codebase. Microservices can avoid at least that one complexity.
  - devoutsalsa 2y ago
    Another advantage of microservices is that you can avoid the overhead of multiple services by having one really big microservice.
  - The_Colonel 2y ago
    You mean like profiles? Monolith can run like a front service or background worker depending on the config? In a technical sense it is a complexity, but IME pales in comparison with the alternative of having to manage multiple services. I actually really like this model, it's pretty flexible. You can still have specialized server instances dedicated for certain tasks, but you have one codebase. What's very sweet is that for local development, you just run a single application which (with all enabled profiles) can fulfill all the roles at the same time.

appplication 2y ago

To your comment on Airflow, I’ve been around that block a few times. I’ve found Airflow (and really any orchestration) be the most manageable when it’s nearly devoid of all logic to the point of DAGs being little more than a series of function or API calls, with each of those responsible for managing state transfer to the next call (as opposed to relying on orchestration to do so). For example, you need some ETL to happen every day. Instead of having your pipeline logic inside an airflow task, you put your logic in a library, where you can test and establish boundaries for this behavior in isolation, and compose this logic portably into any system that can accept your library code. When you need to orchestrate, you just call this function inside an airflow task. This has a few benefits. You now decouple, to a significant extent, your logic and state transfer from your orchestration. That means if you want to debug your DAG, you don’t need to do it in Airflow. You can take the same series of function calls and run them, for example, sequentially in a notebook and you would achieve the same effect. This also can reveal just how little logic you really need in orchestration. There are some other tricks to making this work really well, such as reducing dependency injection to primatives only where possible, and focusing on decoupling logic from configuration. Some of this is pretty standard, but I’ve seen teams not have a strong philosophy on this and then struggle with maintaining clean orchestration interfaces.

Noneofya 2y ago
Thanks, this was really helpful.
RangerScience 2y ago
Helpful comment! If I could pick your brain... I'm looking at a green field implementation of a task system, for human tasks - people need to do a thing, and then mark that they've done it, and that "unlocks" subsequent human tasks, and near as I can tell the overall task flow is a DAG. I'm currently considering how (if?) to allow for complex logic about things like which tasks are present in the overall DAG - things like skipping a node based on some criteria (which, it occurs to me in typing this up, can benefit from your above advice, as that can just be a configured function call that returns skip/no-skip) - and, well... thoughts? (: