4 ms·

How does CI work with a monorepo? Do you always have to run all the tests and build all the artifacts? Or are there nice ways to say "just build this part of th

How does CI work with a monorepo? Do you always have to run all the tests and build all the artifacts? Or are there nice ways to say "just build this part of the repo"?

FartyMcFarter 8y ago
For safe-looking changes, it's OK to only run a subset of the tests (usually including the tests that directly test the changed library). For changes that are more likely to break distant code, you can run all tests (perhaps bundling together several changes in order not to overload the system). Alternatively you can take the risk of breaking tests post-submit... this is not very good citizenship, but in some cases it might be reasonable (when the risk is small).
ebikelaw 8y ago
Dependencies are explicit so the build tool (Bazel, to an approximation) compute the transitive closure of requirements of the desired target. There are more details about testing at [1] 1: https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/45861.pdf https://static.googleusercontent.com/media/research.google.c...
dekhn 8y ago
You specify targets. Just like using bazel: bazel build //tensorflow/blah/.... I maintain a small part of the monorepo, and it's really nice to be able say "Run every test that transitively depends on numpy with my uncommitted changes", so you can know if your changes break anybody who uses numpy when you update the version. Personally I think it would be neat if there was an external "virtual monorepo" that integrated as-close-to-head of all software projects (starting at the root, that's things like absl and icu, with the tails being complex projects like tensorflow), and constantly ran CI to update the base versions of things. Every time I move to the open source world, I basically have to recompile the world from scratch and it's a ton of work.
dlubarov 8y ago
It's flexible; presubmit tests can be configured per-directory. There's also an option to run all tests of packages that could be affected by a change based on the Blaze dependency graph. If you're making changes to a package with tons of dependencies such as Guava, for a risky change you might want to run all affected tests, but for a minor change you might want to run just the standard unit tests. As a compromise, there's also an option to run a random sample of affected tests.

How does CI work with a monorepo? Do you always have to run all the tests and build all the artifacts? Or are there nice ways to say "just build this part of the repo"?

FartyMcFarter 8y ago

For safe-looking changes, it's OK to only run a subset of the tests (usually including the tests that directly test the changed library). For changes that are more likely to break distant code, you can run all tests (perhaps bundling together several changes in order not to overload the system). Alternatively you can take the risk of breaking tests post-submit... this is not very good citizenship, but in some cases it might be reasonable (when the risk is small).

ebikelaw 8y ago

Dependencies are explicit so the build tool (Bazel, to an approximation) compute the transitive closure of requirements of the desired target. There are more details about testing at [1] 1: https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/45861.pdf https://static.googleusercontent.com/media/research.google.c...

dekhn 8y ago

You specify targets. Just like using bazel: bazel build //tensorflow/blah/.... I maintain a small part of the monorepo, and it's really nice to be able say "Run every test that transitively depends on numpy with my uncommitted changes", so you can know if your changes break anybody who uses numpy when you update the version. Personally I think it would be neat if there was an external "virtual monorepo" that integrated as-close-to-head of all software projects (starting at the root, that's things like absl and icu, with the tails being complex projects like tensorflow), and constantly ran CI to update the base versions of things. Every time I move to the open source world, I basically have to recompile the world from scratch and it's a ton of work.

dlubarov 8y ago

It's flexible; presubmit tests can be configured per-directory. There's also an option to run all tests of packages that could be affected by a change based on the Blaze dependency graph. If you're making changes to a package with tons of dependencies such as Guava, for a risky change you might want to run all affected tests, but for a minor change you might want to run just the standard unit tests. As a compromise, there's also an option to run a random sample of affected tests.