5 ms·

How would they do CI/CD if it's all in one big repo? I suppose you could do it if you had a very strict rule where absolutely everything that could affect a "u

by borplk 9y ago

How would they do CI/CD if it's all in one big repo? I suppose you could do it if you had a very strict rule where absolutely everything that could affect a "unit" was inside its own directory (but never and nothing higher up than that "project root"). So you could check which sub-directory is affected within a commit and so on.

tehlike 9y ago
Oh it works really well. One benefit is all tests affected by your target is run as you presubmit. The benefit is everything is at head so things like library bugs, security bugs are all handled naturally as part of the new releases. This usually happens twice a week for most server binaries.
- Xorlev 9y ago
  So long as the presubmits are configured correctly, or someone ignores failures and force submits. :) It's usually quite nice though, and changes that break lots of projects are rolled back quite fast. Usually.
  - tehlike 9y ago
    As much as people hate, force submits are a fact of life.
  - tehlike 9y ago
    Rollback in some cases happen automatically, even.
- krzyk 9y ago
  So, how long does the commit with the checks take take?
  - tehlike 9y ago
    Depends. Mobile tests that run on emulators are the worst. Unit tests finish relatively fast, integration tests that bring up server tend to be slow (30-40mins best case almost for these for projects im working on). The cost of this gets amortized two ways: you can run immediate unit tests manually on command line as the fastest signal. Then when you send for code review, presubmit runs. During code reviews you may choose to run them as you go. Eventually when you submit they run again. If there has not been any changes to your commit/cl and there is an already passing submit, it will just skip and submit.
    - closeparen 9y ago
      What can you possibly be doing in an integration test that takes 40 minutes?
      - Xorlev 9y ago
        In a single integration test? That'd be pretty absurd. At least in our project, each integration test has a certain amount of overhead. Some backends are fakes (when I request X, you provide Y), some are actually booted up with the test, e.g. persistence. Multiply this across N integration tests, have lots of demand for the same CPUs, and you're up to 30-40 minutes of integration test time. Though, that said, some integration tests can be crazy long if they have a lot of "waitFor" style conditions. "Do this, then wait for something to happen in backend Z. Once that's done, do this, and this, and this..."
        Buge 9y ago
        >Multiply this across N integration tests, But in theory with enough servers all the integration tests could be run in parallel. So it would only take as long as the longest single test.
        byroot 9y ago
        The longest single test + the time it takes to setup the test environment (booting servers, etc). Parallelizing tests has diminishing returns unless you manage to dramatically reduce the setup time.
  - saryant 9y ago
    (Not at Google, but Twitter has its own monorepo) Generally submit queue takes 10-20 minutes, longer if you're changing a core library. I find the larger factor is what kind of test has to run. Feature tests can take a while on CI if you're spinning up lots of embedded services (dependent services, MySQL, storage layers, etc).
joshuamorton 9y ago
Blaze/bazel solves this. All build targets (buildable units) explicitly define their direct dependencies. If you modify something, you can then run all tests for all units that transitively depend on your change. For surface level changes this is often quite small. For changes to core libraries, well, you run a lot of tests.
jingwen 9y ago
This is one part of a _very_ big answer, but Bazel (internally Blaze) lets you do reverse dependency querying [0] with the query language, i.e. "Bazel, give me the list of targets that depend on this target that I've just modified" $ bazel query 'rdeps(//foo/my:target, //...)' Of course, this query in the monorepo will take a long time or not work, because the target universe of "//..." is far too large. This is where other systems come in. [0] https://docs.bazel.build/versions/master/query.html#rdeps https://docs.bazel.build/versions/master/query.html#rdeps
- wocram 9y ago
  I imagine this is yet another thing that can be cached, but you can expand on 'other systems'?
  - jingwen 9y ago
    I'm unsure about the confidentiality of those systems, so I'm erring on the side of not expanding. However by deriving from first principles, yes, there is no reason to re-query the transitive closure of unchanged targets' reverse deps, so caching can happen here.

by borplk 9y ago

tehlike 9y ago

Oh it works really well. One benefit is all tests affected by your target is run as you presubmit. The benefit is everything is at head so things like library bugs, security bugs are all handled naturally as part of the new releases. This usually happens twice a week for most server binaries.

Xorlev 9y ago
So long as the presubmits are configured correctly, or someone ignores failures and force submits. :) It's usually quite nice though, and changes that break lots of projects are rolled back quite fast. Usually.
- tehlike 9y ago
  As much as people hate, force submits are a fact of life.
- tehlike 9y ago
  Rollback in some cases happen automatically, even.
krzyk 9y ago
So, how long does the commit with the checks take take?
- tehlike 9y ago
  Depends. Mobile tests that run on emulators are the worst. Unit tests finish relatively fast, integration tests that bring up server tend to be slow (30-40mins best case almost for these for projects im working on). The cost of this gets amortized two ways: you can run immediate unit tests manually on command line as the fastest signal. Then when you send for code review, presubmit runs. During code reviews you may choose to run them as you go. Eventually when you submit they run again. If there has not been any changes to your commit/cl and there is an already passing submit, it will just skip and submit.
  - closeparen 9y ago
    What can you possibly be doing in an integration test that takes 40 minutes?
    - Xorlev 9y ago
      In a single integration test? That'd be pretty absurd. At least in our project, each integration test has a certain amount of overhead. Some backends are fakes (when I request X, you provide Y), some are actually booted up with the test, e.g. persistence. Multiply this across N integration tests, have lots of demand for the same CPUs, and you're up to 30-40 minutes of integration test time. Though, that said, some integration tests can be crazy long if they have a lot of "waitFor" style conditions. "Do this, then wait for something to happen in backend Z. Once that's done, do this, and this, and this..."
      - Buge 9y ago
        >Multiply this across N integration tests, But in theory with enough servers all the integration tests could be run in parallel. So it would only take as long as the longest single test.
        byroot 9y ago
        The longest single test + the time it takes to setup the test environment (booting servers, etc). Parallelizing tests has diminishing returns unless you manage to dramatically reduce the setup time.
- saryant 9y ago
  (Not at Google, but Twitter has its own monorepo) Generally submit queue takes 10-20 minutes, longer if you're changing a core library. I find the larger factor is what kind of test has to run. Feature tests can take a while on CI if you're spinning up lots of embedded services (dependent services, MySQL, storage layers, etc).

joshuamorton 9y ago

Blaze/bazel solves this. All build targets (buildable units) explicitly define their direct dependencies. If you modify something, you can then run all tests for all units that transitively depend on your change. For surface level changes this is often quite small. For changes to core libraries, well, you run a lot of tests.

jingwen 9y ago

This is one part of a _very_ big answer, but Bazel (internally Blaze) lets you do reverse dependency querying [0] with the query language, i.e. "Bazel, give me the list of targets that depend on this target that I've just modified" $ bazel query 'rdeps(//foo/my:target, //...)' Of course, this query in the monorepo will take a long time or not work, because the target universe of "//..." is far too large. This is where other systems come in. [0] https://docs.bazel.build/versions/master/query.html#rdeps https://docs.bazel.build/versions/master/query.html#rdeps

wocram 9y ago
I imagine this is yet another thing that can be cached, but you can expand on 'other systems'?
- jingwen 9y ago
  I'm unsure about the confidentiality of those systems, so I'm erring on the side of not expanding. However by deriving from first principles, yes, there is no reason to re-query the transitive closure of unchanged targets' reverse deps, so caching can happen here.