4 ms·

Can you explain how a monorepo for microservices is a tooling nightmare and bandwidth sink? In my experience, a monorepo significantly makes things easier and

Can you explain how a monorepo for microservices is a tooling nightmare and bandwidth sink? In my experience, a monorepo significantly makes things easier and saves time everyday for all developers involved when compared with multi repo.

rtpg 4y ago
The biggest issue is that every CI provider speaks with "repo=project" language, so your way forward with monorepos is using stuff like Bazel. Bazel is extremely cool! But you end up with, like, a handful of people on the team who can write Bazel and eveyone else cargo culting their way through it. There are a lot of JS "monorepo management" tools but they all seem concerned about the release phase for a lot of libraries that really should just be one library instead of 300 npm packages or whatever.
- CrimsonRain 4y ago
  Thank you. > repo=project This has gotten better lately. At least, good enough for small-medium scale projects.
- dilyevsky 4y ago
  > Bazel is extremely cool! But you end up with, like, a handful of people on the team who can write Bazel and eveyone else cargo culting their way through it. It's not that difficult and docs are outstanding. It can and it will be worse with your own Bash-isms that likely won't have any docs at all.
  - rtpg 4y ago
    I think Bazel is doing a lot of good stuff... but I think it suffers a bit from the same thing as Angular, where it makes a lot of its own terminology. There is another aspect where if you are using a language like Python or JS then you have to kind of swim upstream to get an existing project onto Bazel. Far from impossible but if you just look at the default Bazel stuff without pulling in third-party libs it's pretty tedious to get a project with a good amount of dependencies working.
    - dilyevsky 4y ago
      Why would you not pull third-party libs? I presume you use loads of them in the actual codebase how is this different?
ithkuil 4y ago
>Can you explain how a monorepo for microservices is a tooling nightmare and bandwidth sink? bandwidth: I meant "bandwidth" as literal bandwidth. When your codebase becomes huge, your VCS repo size becomes enormous and it becomes harder and harder to keep a full checkout on all the development machines, especially if they are over the WAN (e.g. at home on your laptop). This has fuelled solutions like sparse checkouts (like MS vfs for git, now scalar) remote development (like TFA; but also Google's cider and srcfs etc). tooling: naïve monorepo tooling (which I've seen in various companies I worked for) simply perform a full build of the whole monorepo for each CI execution. At first this is just fine since you can parallelize builds and call it a day; but after a while, the builds just don't scale anymore ; flaky tests become an increasing frustration etc. The tooling that can help scale large monorepos does exist, but requires buy in and comes with its own learning curve and tradeoffs. One well known such tool is bazel (https://bazel.build https://bazel.build) and bazel remote builds and remote caches. These tools are hard to set up, although folks at https://www.buildbuddy.io/ https://www.buildbuddy.io/ can help smaller startups by offering a managed service. (Again, I'm talking about really large monorepos. A monorepo which includes a dozen or so modules, for which you can easily perform a full re-build on a single CI worker and on your laptop is not the kind of repo that creates tooling nightmares.)
- CrimsonRain 4y ago
  Thank you. Monorepos of _that_ size is definitely far more complex and comes with certain issues that you mention. Although, I feel like you'll only hit that wall once you are _very_ large as a company and at that point, you'll also have resources to climb that wall. For small-medium scale, monorepo has been a blessing after dealing with multi-repo systems for years.
  - ithkuil 4y ago
    I'm a fan of monorepos personally. That said, I cannot pretend I don't see the problems with suboptimal tooling working in a medium size codebase. "big" and "medium" and "small" are quite subjective things. At $work we have a monorepo whose git repo grew to 1GB in size and where the CI turn-around is so high, and the glitches so often, that it often takes hours or even days to land some code to prod. Developers instinctively react by making bigger and bigger changes because the very thought of going through PR/review/CI/merge cycle once again terrifies them. It's all compounded by a security policy that forces a code review approval every time the source code changed, including when you have to apply fixes to build failures induced by a component you don't own. All of these things can and should be be fixed. But this is work, and is not urgent work so it's not often done at the same pace other stuff is done. This induces fatigue in the team and, as usually things go, people tend to blame the most easiest thing to blame: the monorepo. That's why I try to phrase the problem to be a tooling problem and not a monorepo problem. Clearly if you don't have a problem with your monorepo, you either already have good tooling, or you don't need good tooling.

rtpg 4y ago

The biggest issue is that every CI provider speaks with "repo=project" language, so your way forward with monorepos is using stuff like Bazel. Bazel is extremely cool! But you end up with, like, a handful of people on the team who can write Bazel and eveyone else cargo culting their way through it. There are a lot of JS "monorepo management" tools but they all seem concerned about the release phase for a lot of libraries that really should just be one library instead of 300 npm packages or whatever.

CrimsonRain 4y ago
Thank you. > repo=project This has gotten better lately. At least, good enough for small-medium scale projects.
dilyevsky 4y ago
> Bazel is extremely cool! But you end up with, like, a handful of people on the team who can write Bazel and eveyone else cargo culting their way through it. It's not that difficult and docs are outstanding. It can and it will be worse with your own Bash-isms that likely won't have any docs at all.
- rtpg 4y ago
  I think Bazel is doing a lot of good stuff... but I think it suffers a bit from the same thing as Angular, where it makes a lot of its own terminology. There is another aspect where if you are using a language like Python or JS then you have to kind of swim upstream to get an existing project onto Bazel. Far from impossible but if you just look at the default Bazel stuff without pulling in third-party libs it's pretty tedious to get a project with a good amount of dependencies working.
  - dilyevsky 4y ago
    Why would you not pull third-party libs? I presume you use loads of them in the actual codebase how is this different?

ithkuil 4y ago

>Can you explain how a monorepo for microservices is a tooling nightmare and bandwidth sink? bandwidth: I meant "bandwidth" as literal bandwidth. When your codebase becomes huge, your VCS repo size becomes enormous and it becomes harder and harder to keep a full checkout on all the development machines, especially if they are over the WAN (e.g. at home on your laptop). This has fuelled solutions like sparse checkouts (like MS vfs for git, now scalar) remote development (like TFA; but also Google's cider and srcfs etc). tooling: naïve monorepo tooling (which I've seen in various companies I worked for) simply perform a full build of the whole monorepo for each CI execution. At first this is just fine since you can parallelize builds and call it a day; but after a while, the builds just don't scale anymore ; flaky tests become an increasing frustration etc. The tooling that can help scale large monorepos does exist, but requires buy in and comes with its own learning curve and tradeoffs. One well known such tool is bazel (https://bazel.build https://bazel.build) and bazel remote builds and remote caches. These tools are hard to set up, although folks at https://www.buildbuddy.io/ https://www.buildbuddy.io/ can help smaller startups by offering a managed service. (Again, I'm talking about really large monorepos. A monorepo which includes a dozen or so modules, for which you can easily perform a full re-build on a single CI worker and on your laptop is not the kind of repo that creates tooling nightmares.)

CrimsonRain 4y ago
Thank you. Monorepos of _that_ size is definitely far more complex and comes with certain issues that you mention. Although, I feel like you'll only hit that wall once you are _very_ large as a company and at that point, you'll also have resources to climb that wall. For small-medium scale, monorepo has been a blessing after dealing with multi-repo systems for years.
- ithkuil 4y ago
  I'm a fan of monorepos personally. That said, I cannot pretend I don't see the problems with suboptimal tooling working in a medium size codebase. "big" and "medium" and "small" are quite subjective things. At $work we have a monorepo whose git repo grew to 1GB in size and where the CI turn-around is so high, and the glitches so often, that it often takes hours or even days to land some code to prod. Developers instinctively react by making bigger and bigger changes because the very thought of going through PR/review/CI/merge cycle once again terrifies them. It's all compounded by a security policy that forces a code review approval every time the source code changed, including when you have to apply fixes to build failures induced by a component you don't own. All of these things can and should be be fixed. But this is work, and is not urgent work so it's not often done at the same pace other stuff is done. This induces fatigue in the team and, as usually things go, people tend to blame the most easiest thing to blame: the monorepo. That's why I try to phrase the problem to be a tooling problem and not a monorepo problem. Clearly if you don't have a problem with your monorepo, you either already have good tooling, or you don't need good tooling.