6 ms·

In my case, I'm struggling to find good tooling to support monorepos. We're running a microservice architecture, but our CI is triggered by GitHub web hooks. Cu

In my case, I'm struggling to find good tooling to support monorepos. We're running a microservice architecture, but our CI is triggered by GitHub web hooks. Currently we're either doing a full build of everything on each webhook event or we're doing some error-prone git-diffing to try to make sure we're only rebuilding when necessary. I've looked at tools like Buck and Bazel, but they seem really heavy for our ~30 person engineering team and they also seem to have odd ways of doing things (no support for pulling from a package repository, vendoring is assumed to be vendored--which incidentally the author of this blog post characterizes as a Bad Thing). Folks who are using monorepos successfully--what tooling do you use to solve these problems?

porlune 8y ago
I'm working to solve the same issue with our CI/CD pipeline. Have you tried checking for version differences between your existing images and the services in your monorepo as an indicator of which services to rebuild(instead of using a git-difference)?
- weberc2 8y ago
  I'm not sure what you're proposing? Hashing the inputs to the build function and associating that with an image and using that checksum to determine if there's a change? If so, that doesn't seem to solve the problem of reliably identifying inputs to include in the checksum in the first place, which is a big part of the problem. I imagine we could do the build in a container and use the Docker build cache to identify what things need to be rebuilt, but that doesn't seem like it would scale well since afaik there's no good way to share the cache across multiple nodes.
justinwp 8y ago
We have been using Bazel in 20 dev python org. I think most of us have been happy with the transition while collaboration has improved and onboarding has become much easier. The biggest pain point is the python part and not having the resources to vendor everything but the Bazel team is working to fix this albeit very slowly. We get around some of these nonhermetic issues with python and dependencies by using a consistent docker based dev environment for everyone. Only takes a few minutes bazel build/test/run anything in our repo. EDIT: We run Bazel in drone as part of our CI/CD system and this is probably the most unoptimized part given how they do not work that well together. Bazel query allows us to due targeted builds/tests based upon the git changes and dependents of those changes.
- weberc2 8y ago
  Any chance you would be willing to describe your setup in greater detail? I'd really like to know more.
  - weberc2 8y ago
    Especially about how you're building Python. Seems like Bazel still doesn't support Python 3 at all.
mr_tristan 8y ago
I come back to this tooling issue again and again. It really, really depends on the platform, too. Some platforms, like C++, require a lot of tooling work. So it's kind of a natural thing to just add on to it. And in fact can speed up the build a bunch when you start tracking change of intermediate objects. For almost every web service kind of app I've been involved with, build tooling is typically very standard, and supports semantic version locking out of the box (e.g., bundler, Maven/Gradle, etc). Dealing with multiple repos is just easy to manage in a very controlled way. The places I've been at that use monorepos were C++ shops from long ago. Their tooling grew out of that approach.
Tempest1981 8y ago
Agreed. Would like to see an article on tooling, to address common pain points. This article simply says "try a monorepo", and when you hit problems, figure them out. It mentions that Twitter had issues, but no specifics.
gowld 8y ago
make solved dependency resolution for partial rebuilds over 30 years ago. There has been lots of improvement since. I don't doubt you have a real challenge, but the problem comes from some immature tool in your toolchain (that I'd be willing to speculate is because someone decided only Javascript tools are interesting after 2010, because reimplementing features classic tools is more fun than learning old reliable tools), not an aspect of the monorepo.
- weberc2 8y ago
  How does make solve the problem of only building what has changed in the current PR? It seems like this would only work if you have a single CI server and all of your artifacts live in the filesystem on that server, no? Solving that seems sufficiently difficult that saying "make solves the problem" seems disingenuous, but maybe it's easier than it appears to me?

porlune 8y ago

I'm working to solve the same issue with our CI/CD pipeline. Have you tried checking for version differences between your existing images and the services in your monorepo as an indicator of which services to rebuild(instead of using a git-difference)?

weberc2 8y ago
I'm not sure what you're proposing? Hashing the inputs to the build function and associating that with an image and using that checksum to determine if there's a change? If so, that doesn't seem to solve the problem of reliably identifying inputs to include in the checksum in the first place, which is a big part of the problem. I imagine we could do the build in a container and use the Docker build cache to identify what things need to be rebuilt, but that doesn't seem like it would scale well since afaik there's no good way to share the cache across multiple nodes.

justinwp 8y ago

We have been using Bazel in 20 dev python org. I think most of us have been happy with the transition while collaboration has improved and onboarding has become much easier. The biggest pain point is the python part and not having the resources to vendor everything but the Bazel team is working to fix this albeit very slowly. We get around some of these nonhermetic issues with python and dependencies by using a consistent docker based dev environment for everyone. Only takes a few minutes bazel build/test/run anything in our repo. EDIT: We run Bazel in drone as part of our CI/CD system and this is probably the most unoptimized part given how they do not work that well together. Bazel query allows us to due targeted builds/tests based upon the git changes and dependents of those changes.

weberc2 8y ago
Any chance you would be willing to describe your setup in greater detail? I'd really like to know more.
- weberc2 8y ago
  Especially about how you're building Python. Seems like Bazel still doesn't support Python 3 at all.

mr_tristan 8y ago

I come back to this tooling issue again and again. It really, really depends on the platform, too. Some platforms, like C++, require a lot of tooling work. So it's kind of a natural thing to just add on to it. And in fact can speed up the build a bunch when you start tracking change of intermediate objects. For almost every web service kind of app I've been involved with, build tooling is typically very standard, and supports semantic version locking out of the box (e.g., bundler, Maven/Gradle, etc). Dealing with multiple repos is just easy to manage in a very controlled way. The places I've been at that use monorepos were C++ shops from long ago. Their tooling grew out of that approach.

Tempest1981 8y ago

Agreed. Would like to see an article on tooling, to address common pain points. This article simply says "try a monorepo", and when you hit problems, figure them out. It mentions that Twitter had issues, but no specifics.

gowld 8y ago

make solved dependency resolution for partial rebuilds over 30 years ago. There has been lots of improvement since. I don't doubt you have a real challenge, but the problem comes from some immature tool in your toolchain (that I'd be willing to speculate is because someone decided only Javascript tools are interesting after 2010, because reimplementing features classic tools is more fun than learning old reliable tools), not an aspect of the monorepo.

weberc2 8y ago
How does make solve the problem of only building what has changed in the current PR? It seems like this would only work if you have a single CI server and all of your artifacts live in the filesystem on that server, no? Solving that seems sufficiently difficult that saying "make solves the problem" seems disingenuous, but maybe it's easier than it appears to me?