4 ms·

Oh it works really well. One benefit is all tests affected by your target is run as you presubmit. The benefit is everything is at head so things like library b

Oh it works really well. One benefit is all tests affected by your target is run as you presubmit. The benefit is everything is at head so things like library bugs, security bugs are all handled naturally as part of the new releases. This usually happens twice a week for most server binaries.

Xorlev 9y ago
So long as the presubmits are configured correctly, or someone ignores failures and force submits. :) It's usually quite nice though, and changes that break lots of projects are rolled back quite fast. Usually.
- tehlike 9y ago
  As much as people hate, force submits are a fact of life.
- tehlike 9y ago
  Rollback in some cases happen automatically, even.
krzyk 9y ago
So, how long does the commit with the checks take take?
- tehlike 9y ago
  Depends. Mobile tests that run on emulators are the worst. Unit tests finish relatively fast, integration tests that bring up server tend to be slow (30-40mins best case almost for these for projects im working on). The cost of this gets amortized two ways: you can run immediate unit tests manually on command line as the fastest signal. Then when you send for code review, presubmit runs. During code reviews you may choose to run them as you go. Eventually when you submit they run again. If there has not been any changes to your commit/cl and there is an already passing submit, it will just skip and submit.
  - closeparen 9y ago
    What can you possibly be doing in an integration test that takes 40 minutes?
    - Xorlev 9y ago
      In a single integration test? That'd be pretty absurd. At least in our project, each integration test has a certain amount of overhead. Some backends are fakes (when I request X, you provide Y), some are actually booted up with the test, e.g. persistence. Multiply this across N integration tests, have lots of demand for the same CPUs, and you're up to 30-40 minutes of integration test time. Though, that said, some integration tests can be crazy long if they have a lot of "waitFor" style conditions. "Do this, then wait for something to happen in backend Z. Once that's done, do this, and this, and this..."
      - Buge 9y ago
        >Multiply this across N integration tests, But in theory with enough servers all the integration tests could be run in parallel. So it would only take as long as the longest single test.
        byroot 9y ago
        The longest single test + the time it takes to setup the test environment (booting servers, etc). Parallelizing tests has diminishing returns unless you manage to dramatically reduce the setup time.
    - tehlike 9y ago
      Integration tests bring up servers, and once you have tens or hundreds, this happens.
      - closeparen 9y ago
        Oh wow. We have a test tenancy that's carried throughout production, so you make requests against real backends (read/write data in the test namespace, sometimes read-only production data). There's a proxy in front doing rate limiting, endpoint whitelisting, audit logging, emergency lockdown, etc. I never thought of deploying a whole separate environment just for integration testing. Still, seems you could keep a handful of integration test environments always running? Time spent waiting your turn for one of them could well be less than time spent spinning up a whole bunch of servers.
- saryant 9y ago
  (Not at Google, but Twitter has its own monorepo) Generally submit queue takes 10-20 minutes, longer if you're changing a core library. I find the larger factor is what kind of test has to run. Feature tests can take a while on CI if you're spinning up lots of embedded services (dependent services, MySQL, storage layers, etc).

Xorlev 9y ago

So long as the presubmits are configured correctly, or someone ignores failures and force submits. :) It's usually quite nice though, and changes that break lots of projects are rolled back quite fast. Usually.

tehlike 9y ago
As much as people hate, force submits are a fact of life.
tehlike 9y ago
Rollback in some cases happen automatically, even.

krzyk 9y ago

So, how long does the commit with the checks take take?

tehlike 9y ago
Depends. Mobile tests that run on emulators are the worst. Unit tests finish relatively fast, integration tests that bring up server tend to be slow (30-40mins best case almost for these for projects im working on). The cost of this gets amortized two ways: you can run immediate unit tests manually on command line as the fastest signal. Then when you send for code review, presubmit runs. During code reviews you may choose to run them as you go. Eventually when you submit they run again. If there has not been any changes to your commit/cl and there is an already passing submit, it will just skip and submit.
- closeparen 9y ago
  What can you possibly be doing in an integration test that takes 40 minutes?
  - Xorlev 9y ago
    In a single integration test? That'd be pretty absurd. At least in our project, each integration test has a certain amount of overhead. Some backends are fakes (when I request X, you provide Y), some are actually booted up with the test, e.g. persistence. Multiply this across N integration tests, have lots of demand for the same CPUs, and you're up to 30-40 minutes of integration test time. Though, that said, some integration tests can be crazy long if they have a lot of "waitFor" style conditions. "Do this, then wait for something to happen in backend Z. Once that's done, do this, and this, and this..."
    - Buge 9y ago
      >Multiply this across N integration tests, But in theory with enough servers all the integration tests could be run in parallel. So it would only take as long as the longest single test.
      - byroot 9y ago
        The longest single test + the time it takes to setup the test environment (booting servers, etc). Parallelizing tests has diminishing returns unless you manage to dramatically reduce the setup time.
  - tehlike 9y ago
    Integration tests bring up servers, and once you have tens or hundreds, this happens.
    - closeparen 9y ago
      Oh wow. We have a test tenancy that's carried throughout production, so you make requests against real backends (read/write data in the test namespace, sometimes read-only production data). There's a proxy in front doing rate limiting, endpoint whitelisting, audit logging, emergency lockdown, etc. I never thought of deploying a whole separate environment just for integration testing. Still, seems you could keep a handful of integration test environments always running? Time spent waiting your turn for one of them could well be less than time spent spinning up a whole bunch of servers.
saryant 9y ago
(Not at Google, but Twitter has its own monorepo) Generally submit queue takes 10-20 minutes, longer if you're changing a core library. I find the larger factor is what kind of test has to run. Feature tests can take a while on CI if you're spinning up lots of embedded services (dependent services, MySQL, storage layers, etc).