3 ms·

I work on CI/CD systems, and that’s one thing that definitely gets harder in a monorepo. So you made a commit. What artifacts change as a result? What do you

I work on CI/CD systems, and that’s one thing that definitely gets harder in a monorepo. So you made a commit. What artifacts change as a result? What do you need to rebuild, retest, and redeploy? It doesn’t take a large amount of scale to make rebuilding and retesting everything impossible. In a poly repo world, the repository is generally the unit of building and deployment. In monorepo it gets more messy. For instance, one perceived benefit of a monorepo is it removes the need for explicit versioning between libraries and the code that uses them, since they’re all versioned together. But now, if someone changes the library, you need to have a way to find all of its usages, and retest those to make sure the changed didn’t break their use. So there’s a dependency tree of components somewhere that needs to be established, but now it’s not explicit, and no one is given the option to pin to a particular version if they can’t/won’t update. This is the world of google & influcenced the (lack of) dependency management in go. You could very well publish everything independently, using semver, and put build descriptors inside each project subdirectory, but then, congratulations, you just invented the polyrepo, or an approximation thereof.

manigandham 8y ago
Rebuild and deploy everything, what's the actual problem? Like the OP said, that's a scale issue and most projects don't have it. Also building/testing is far more effective at finding dependencies than just going by repo structure. There are numerous package managers available to solve versioning if you need separate components.
- jskaggz 8y ago
  Yes! This brings to mind Donald Knuth: "Premature optimization is the root of all evil."
- yowlingcat 8y ago
  100% agree with your entire comment. This is what we do with our monorepo now -- it turns out the rebuilding and deploying everything is actually just fine. If your application services are stateless and decoupled from your state stores, it's completely harmless. If you need to do something fancy, congrats! You're at scale -- enjoy it but remember that it's something rare.
curtis 8y ago
> So you made a commit. What artifacts change as a result? What do you need to rebuild, retest, and redeploy? If you're using Git, then typically for each push to the remote repository you get a notification with this data in it: BRANCH # the remote branch getting updated OLD_COMMIT # the commit the branch ref was pointing to before the push NEW_COMMIT # the commit the branch ref was pointing to after the push # To get the list of files that changed in the push: git diff --name-only "$OLD_COMMIT" "$NEW_COMMIT" Once you know which files changed in a push you can figure out which artifacts you need to build. Right now you'll have to write that tooling yourself since I don't know of any off-the-shelf tools that do it. In my company's case, we have "project.yml" files scattered through the repo telling us which directories have buildable artifacts and what branches each one needs to be built for. The tooling to support this is a few hundred lines of Bash and Python. In our case we're still small enough that we can brute force some stuff, but we can easily improve the tooling as we go along.
- brokentone 8y ago
  This is something I've been working on a bit myself. Figuring out which files change is relatively easy (as you've demonstrated). Figuring out what the impact of that is quite hard in non-compiled languages (tools like Maven, Buck, Bazel, etc do this well for compiled languages). I.e. In a repo which is primarily JavaScript, I can get the list of changed files, and hopefully have unit test files which are obviously linear to those. However, knowing if these are depended on by other files/modules (at some depth) is much harder. Same for integration tests -- which of these are related?
  - Rapzid 8y ago
    I believe the typical approach is to have project.yml list its dependency projects. Build a DAG(error on cycles) and then build all changed and downstream projects.
jskaggz 8y ago
One thing I heavily enjoy about monorepo's (I'm talking java/c#/c++ projects) is the ability to navigate the entire codebase from within an ide. That alone has caused me to migrate projects (medium projects ~20 developers) from poly to mono repos. Dropping tons of duplication in the build system in the process. I can think of good reasons to split projects along boundaries when it makes sense, but not blindly by default, and not without carefully considering the tradeoffs.
joshuamorton 8y ago
bazel/buck/pants all solve this, but independently of that, they're probably the best build systems.
wildengineer 8y ago
In the java world this gets solved with gradle's incremental build system, which uses a build cache, a user configured dependency tree, and some hashing to determine what needs to build.

manigandham 8y ago

Rebuild and deploy everything, what's the actual problem? Like the OP said, that's a scale issue and most projects don't have it. Also building/testing is far more effective at finding dependencies than just going by repo structure. There are numerous package managers available to solve versioning if you need separate components.

jskaggz 8y ago
Yes! This brings to mind Donald Knuth: "Premature optimization is the root of all evil."
yowlingcat 8y ago
100% agree with your entire comment. This is what we do with our monorepo now -- it turns out the rebuilding and deploying everything is actually just fine. If your application services are stateless and decoupled from your state stores, it's completely harmless. If you need to do something fancy, congrats! You're at scale -- enjoy it but remember that it's something rare.

curtis 8y ago

> So you made a commit. What artifacts change as a result? What do you need to rebuild, retest, and redeploy? If you're using Git, then typically for each push to the remote repository you get a notification with this data in it: BRANCH # the remote branch getting updated OLD_COMMIT # the commit the branch ref was pointing to before the push NEW_COMMIT # the commit the branch ref was pointing to after the push # To get the list of files that changed in the push: git diff --name-only "$OLD_COMMIT" "$NEW_COMMIT" Once you know which files changed in a push you can figure out which artifacts you need to build. Right now you'll have to write that tooling yourself since I don't know of any off-the-shelf tools that do it. In my company's case, we have "project.yml" files scattered through the repo telling us which directories have buildable artifacts and what branches each one needs to be built for. The tooling to support this is a few hundred lines of Bash and Python. In our case we're still small enough that we can brute force some stuff, but we can easily improve the tooling as we go along.

brokentone 8y ago
This is something I've been working on a bit myself. Figuring out which files change is relatively easy (as you've demonstrated). Figuring out what the impact of that is quite hard in non-compiled languages (tools like Maven, Buck, Bazel, etc do this well for compiled languages). I.e. In a repo which is primarily JavaScript, I can get the list of changed files, and hopefully have unit test files which are obviously linear to those. However, knowing if these are depended on by other files/modules (at some depth) is much harder. Same for integration tests -- which of these are related?
- Rapzid 8y ago
  I believe the typical approach is to have project.yml list its dependency projects. Build a DAG(error on cycles) and then build all changed and downstream projects.

jskaggz 8y ago

One thing I heavily enjoy about monorepo's (I'm talking java/c#/c++ projects) is the ability to navigate the entire codebase from within an ide. That alone has caused me to migrate projects (medium projects ~20 developers) from poly to mono repos. Dropping tons of duplication in the build system in the process. I can think of good reasons to split projects along boundaries when it makes sense, but not blindly by default, and not without carefully considering the tradeoffs.

joshuamorton 8y ago

bazel/buck/pants all solve this, but independently of that, they're probably the best build systems.

wildengineer 8y ago

In the java world this gets solved with gradle's incremental build system, which uses a build cache, a user configured dependency tree, and some hashing to determine what needs to build.