3 ms·

Because in larger monorepos, by definition most of the stuff in there is irrelevant to most people. So you're waiting for checkouts to finish because of a bunch

Because in larger monorepos, by definition most of the stuff in there is irrelevant to most people. So you're waiting for checkouts to finish because of a bunch of irrelevant stuff. You're waiting for tests to finish because of a bunch of irrelevant stuff. You're waiting for requirements to update because of a bunch of irrelevant stuff. You're waiting for compilation, CI/CD, deploys, because of a bunch of irrelevant stuff. Sure, tooling and configuration can mitigate a lot of that stuff, but most tools don't countenance codebases that are so large that most of the stuff in them is irrelevant to everyone. The natural thing to do is to split it up. I'm personally against microservices; I think they go way too far the other way and tend to encourage some of the worst software development practices (NIH, code duplication, super weird architecture and viral explosion of dependency injection everywhere), but "we have one repo for everything" is also pretty weird. I mean, the most famous monorepos (Linux, OpenBSD, Google) literally invented tools to deal with them. That should say something.

bckr 4y ago
What's NIH?
- simlevesque 4y ago
  Not invented here.
unity1001 4y ago
> code duplication That makes it very easy to roll out changes to any given service without breaking others and helps a lot with the backwards compatibility of services. Makes everything more resilient. Shared code is always an internal dependency. You think 'We are sharing code and this is efficient'. But you end up having to take into account many different parts of the application when rolling out one change for you can easily break something totally out of sight while trying to change another. I were mostly against duplication of code. But the more I develop and maintain larger systems, the better I see the value of separating codebases, even if this includes code duplication.
- camgunz 4y ago
  Yes! Great OK a microservices debate. Let's do this. My overarching argument is this: I would characterize microservices as a response to organizational and cultural challenges. However, separating a software project into multiple repositories with their own dependencies, deployments, tests, and philosophies adds needless overhead and complexity to systems. Further, because code can no longer be shared across these component parts of the system, requirements, code, deployments, documentation, etc. must be duplicated. But this duplication is done--if it is done at all--imperfectly, because it is laborious and tedious. Finally, microservices tend to drift. Some are written in Node, others are written Python or Go, etc. Someone wants to try functional programming, someone wants to try Hexagonal. This leads to brittle, badly documented systems that by definition no one completely understands and that require a mountain of ancillary software to manage and operate. > That makes it very easy to roll out changes to any given service without breaking others Functionally, there's no difference between this and copying a function to modify for the new functionality. You're saying "I need to modify [functionality X] in order to deliver [feature Y], but other systems rely on [functionality X], so I have to carefully modify [functionality X] to avoid breaking [feature A-X]". A solution to this is to just make [functionality X_new] and use some if statements. It's not elegant, but neither is forking a new repo to avoid refactoring. I'd characterize that as extreme technical debt, and would recommend refactoring instead. I understand that in lots of shops, forking a new repo is actually easier--you can break free of sclerotic design processes or overbearing colleagues/managers/architects. But the systems that allowed those problems into that project will soon force them into your new project. Microservices are a short-term solution to a long-term organizational or cultural problem. > helps a lot with the backwards compatibility of services There's two ways to do this. You can set up integration tests that prevent you from deploying if you've broken compatibility. Or you can fork a new repo whenever you need to make a (potentially breaking--which is all of them) change. I would recommend integration tests. You're gonna need them eventually. > Shared code is always an internal dependency. You think 'We are sharing code and this is efficient'. But you end up having to take into account many different parts of the application when rolling out one change for you can easily break something totally out of sight while trying to change another. This more or less applies to any part of any software system unless you're very careful about shared state and side effects. The alternative here is unit testing, which, similar to integration testing, I recommend because you'll also need that eventually too. > But the more I develop and maintain larger systems, the better I see the value of separating codebases, even if this includes code duplication. I'm not at all a hardcore "don't repeat yourself" person (more "rule of three") but to the extent I am, I tend to think code duplication indicates you need to reconceive the mental model of your application, not merely create a helper method or whatever. My point here is that for me, code duplication isn't really the worst part of microservices. The thing I find most objectionable is that they lead to weird, rickety systems that are hard to develop and maintain. You haven't lived until your tickets for a sprint involve fixing multiple bugs in code copy-pasted across half a dozen microservices, or you have to deploy your changes to this microservice you've never worked on before and it involves the most bonkers incantations you've ever read about (update this Jenkins script blah blah). I think people have this idea that microservices reduce scope, interdependence, and complexity. Maybe sometimes that's true. But for the engineers who have to work across multiple microservices, you really get all of the bad and none of the good. You're still dealing with a big software project, but now it's sprawled across multiple repositories all with their own idioms, idiosyncrasies, languages, copy-pasted code, deployment setups, etc. etc. ad nauseum. You might argue I've only seen bad implementations of microservices. Sure, that's possible. I'm not saying they can't work. I'm saying the forces that lead teams to adopt microservices inevitably corrupt all projects no matter their architecture, and that microservices incentivize a particular type of shorttermism and myopia that makes some of the scenarios I've described the path of least resistance. I think there are far fewer pitfalls with "1 repo per project" (not "1 repo per company") and that we have great tools and techniques to help teams using this structure.
frenchman99 4y ago
> You're waiting for compilation, CI/CD, deploys, because of a bunch of irrelevant stuff. You could setup your CI to only recompile what's changed, so you wouldn't be waiting for anything else than what you've changed. This usually requires a bit of work upfront, but once you've done it for 1 part of the codebase it is easy and low-maintenance to replicate to the entire codebase. With Gitlab CI (and others), you can import bits of yaml configuration here and there to avoid code duplication for such use cases. > I'm personally against microservices In some cases, you need microservices or at least being able to run only a single part of the monolith through configuration. For instance if you want to host parts of the codebase in a separate virtual machine for security or scalability. I don't think the choice is a matter of opinion but a matter of technical/business requirements.
- camgunz 4y ago
  Well, that's my point. The default for pretty much all your tools is "one repo one project". If you don't want tools to work on the entire repo, you've got to configure them specially, if that's even possible, and you have to hope it works well. You've gotta weigh the trade off of whatever benefit you're getting from monorepos against the irritation of tooling that doesn't really countenance them.
- FridgeSeal 4y ago
  > You could setup your CI to only recompile what's changed, so you wouldn't be waiting for anything else than what you've changed. What’s it like having CI pipelines that aren’t trash? I’m stuck with the awful choice of CI tool my work has made, and there’s not a lot I can do about that.
azemetre 4y ago
There’s also project repos where all the code pertaining to a project is under one repo. I think this setup is the best of both worlds. If you’re a team that has a client, several micro services, DB, etc it’s way better to have that under a single repo than spread to multiple. Monorepos don’t have to be gigantic monstrosities, they can encapsulated products.
- 8note 4y ago
  I'd certainly prefer to keep each microservice in its own repo. That way in a reorg, the services can be redistributed in a way that makes sense without having to make code changes. Having two monorepos where you only own half of each seems like the worst of both worlds
  - azemetre 4y ago
    What if your team is the only ones that consume the micro service? Doesn’t make sense to me to split it. It’s also way easier to take stuff out of monorepos than put them in IMO.
  - drewcoo 4y ago
    > I'd certainly prefer to keep each microservice in its own repo. Having a single monorepo solves that, too.
- lucasyvas 4y ago
  I like to call this a "macroservice" or "business function" repository. If you are dealing with the Billing function, everything to do with that is in the billing repository (UI, API, scripts, workers/jobs, database migrations). This is my favorite for sure in terms of balance - you don't need the overhead of one repo per deployable artifact, and the size of the repository has a reasonable theoretical maximum and can still come close to fitting in your head. You can also test it all together extremely easily and it's the perfect slice for a team to work on, or be shoveled off to a different owner of that function in the future.
deleted 4y ago
[deleted]

bckr 4y ago

What's NIH?

simlevesque 4y ago
Not invented here.

unity1001 4y ago

> code duplication That makes it very easy to roll out changes to any given service without breaking others and helps a lot with the backwards compatibility of services. Makes everything more resilient. Shared code is always an internal dependency. You think 'We are sharing code and this is efficient'. But you end up having to take into account many different parts of the application when rolling out one change for you can easily break something totally out of sight while trying to change another. I were mostly against duplication of code. But the more I develop and maintain larger systems, the better I see the value of separating codebases, even if this includes code duplication.

camgunz 4y ago
Yes! Great OK a microservices debate. Let's do this. My overarching argument is this: I would characterize microservices as a response to organizational and cultural challenges. However, separating a software project into multiple repositories with their own dependencies, deployments, tests, and philosophies adds needless overhead and complexity to systems. Further, because code can no longer be shared across these component parts of the system, requirements, code, deployments, documentation, etc. must be duplicated. But this duplication is done--if it is done at all--imperfectly, because it is laborious and tedious. Finally, microservices tend to drift. Some are written in Node, others are written Python or Go, etc. Someone wants to try functional programming, someone wants to try Hexagonal. This leads to brittle, badly documented systems that by definition no one completely understands and that require a mountain of ancillary software to manage and operate. > That makes it very easy to roll out changes to any given service without breaking others Functionally, there's no difference between this and copying a function to modify for the new functionality. You're saying "I need to modify [functionality X] in order to deliver [feature Y], but other systems rely on [functionality X], so I have to carefully modify [functionality X] to avoid breaking [feature A-X]". A solution to this is to just make [functionality X_new] and use some if statements. It's not elegant, but neither is forking a new repo to avoid refactoring. I'd characterize that as extreme technical debt, and would recommend refactoring instead. I understand that in lots of shops, forking a new repo is actually easier--you can break free of sclerotic design processes or overbearing colleagues/managers/architects. But the systems that allowed those problems into that project will soon force them into your new project. Microservices are a short-term solution to a long-term organizational or cultural problem. > helps a lot with the backwards compatibility of services There's two ways to do this. You can set up integration tests that prevent you from deploying if you've broken compatibility. Or you can fork a new repo whenever you need to make a (potentially breaking--which is all of them) change. I would recommend integration tests. You're gonna need them eventually. > Shared code is always an internal dependency. You think 'We are sharing code and this is efficient'. But you end up having to take into account many different parts of the application when rolling out one change for you can easily break something totally out of sight while trying to change another. This more or less applies to any part of any software system unless you're very careful about shared state and side effects. The alternative here is unit testing, which, similar to integration testing, I recommend because you'll also need that eventually too. > But the more I develop and maintain larger systems, the better I see the value of separating codebases, even if this includes code duplication. I'm not at all a hardcore "don't repeat yourself" person (more "rule of three") but to the extent I am, I tend to think code duplication indicates you need to reconceive the mental model of your application, not merely create a helper method or whatever. My point here is that for me, code duplication isn't really the worst part of microservices. The thing I find most objectionable is that they lead to weird, rickety systems that are hard to develop and maintain. You haven't lived until your tickets for a sprint involve fixing multiple bugs in code copy-pasted across half a dozen microservices, or you have to deploy your changes to this microservice you've never worked on before and it involves the most bonkers incantations you've ever read about (update this Jenkins script blah blah). I think people have this idea that microservices reduce scope, interdependence, and complexity. Maybe sometimes that's true. But for the engineers who have to work across multiple microservices, you really get all of the bad and none of the good. You're still dealing with a big software project, but now it's sprawled across multiple repositories all with their own idioms, idiosyncrasies, languages, copy-pasted code, deployment setups, etc. etc. ad nauseum. You might argue I've only seen bad implementations of microservices. Sure, that's possible. I'm not saying they can't work. I'm saying the forces that lead teams to adopt microservices inevitably corrupt all projects no matter their architecture, and that microservices incentivize a particular type of shorttermism and myopia that makes some of the scenarios I've described the path of least resistance. I think there are far fewer pitfalls with "1 repo per project" (not "1 repo per company") and that we have great tools and techniques to help teams using this structure.

frenchman99 4y ago

> You're waiting for compilation, CI/CD, deploys, because of a bunch of irrelevant stuff. You could setup your CI to only recompile what's changed, so you wouldn't be waiting for anything else than what you've changed. This usually requires a bit of work upfront, but once you've done it for 1 part of the codebase it is easy and low-maintenance to replicate to the entire codebase. With Gitlab CI (and others), you can import bits of yaml configuration here and there to avoid code duplication for such use cases. > I'm personally against microservices In some cases, you need microservices or at least being able to run only a single part of the monolith through configuration. For instance if you want to host parts of the codebase in a separate virtual machine for security or scalability. I don't think the choice is a matter of opinion but a matter of technical/business requirements.

camgunz 4y ago
Well, that's my point. The default for pretty much all your tools is "one repo one project". If you don't want tools to work on the entire repo, you've got to configure them specially, if that's even possible, and you have to hope it works well. You've gotta weigh the trade off of whatever benefit you're getting from monorepos against the irritation of tooling that doesn't really countenance them.
FridgeSeal 4y ago
> You could setup your CI to only recompile what's changed, so you wouldn't be waiting for anything else than what you've changed. What’s it like having CI pipelines that aren’t trash? I’m stuck with the awful choice of CI tool my work has made, and there’s not a lot I can do about that.

azemetre 4y ago

There’s also project repos where all the code pertaining to a project is under one repo. I think this setup is the best of both worlds. If you’re a team that has a client, several micro services, DB, etc it’s way better to have that under a single repo than spread to multiple. Monorepos don’t have to be gigantic monstrosities, they can encapsulated products.

8note 4y ago
I'd certainly prefer to keep each microservice in its own repo. That way in a reorg, the services can be redistributed in a way that makes sense without having to make code changes. Having two monorepos where you only own half of each seems like the worst of both worlds
- azemetre 4y ago
  What if your team is the only ones that consume the micro service? Doesn’t make sense to me to split it. It’s also way easier to take stuff out of monorepos than put them in IMO.
- drewcoo 4y ago
  > I'd certainly prefer to keep each microservice in its own repo. Having a single monorepo solves that, too.
lucasyvas 4y ago
I like to call this a "macroservice" or "business function" repository. If you are dealing with the Billing function, everything to do with that is in the billing repository (UI, API, scripts, workers/jobs, database migrations). This is my favorite for sure in terms of balance - you don't need the overhead of one repo per deployable artifact, and the size of the repository has a reasonable theoretical maximum and can still come close to fitting in your head. You can also test it all together extremely easily and it's the perfect slice for a team to work on, or be shoveled off to a different owner of that function in the future.

deleted 4y ago

[deleted]