5 ms·

This is a non-sequitor I think. A bad ci system is independent from the mandate that, functionally "unless we can verify we don't break you, you can't complain

This is a non-sequitor I think. A bad ci system is independent from the mandate that, functionally "unless we can verify we don't break you, you can't complain when we do".

ajross 3y ago
No? The two statements at hand equate to: "Do not add features to the product without adding validation to CI" and "Do not write validation steps CI just to add features to the product". Those are pretty clearly related concepts. And the reason for the disconnect is that the former centers the requirements of integration and not development, which is exactly backwards. CI isn't "the place where you run tests". The place where you run tests is underneath the software developers' fingers. CI is where you integrate tests developers are already running. And to the extent that is not true, CI stops being a useful tool for development and turns into just another roadblock to evade.
- joshuamorton 3y ago
  > No? The two statements at hand equate to: "Do not add features to the product without adding validation to CI" and "Do not write validation steps CI just to add features to the product". Those are pretty clearly related concepts. Only if your CI and developer workflow are different. To a first approximation, at Google, I run `blaze test //foo` to test `foo`, and CI invokes the same command when testing `foo`. There's caveats here, but they don't apply to the beyonce rule in practice, which is centered on unit tests that can be run hermetically on presubmit. Your weird 3-hour integration test won't be run. Edit: Actually it seems like you're saying "once you've tested a feature once, you can release it", which...no. You need some way of ensuring you don't regress, the CI suite is how you do that, otherwise no one will know when the new release of some random open source library that you depend on breaks your application.
  - ajross 3y ago
    Clearly we agree. I'm just saying that this "Beyone Rule", naively applied, leads to cargo cult CI disasters (I've seen this happen both within and outside of Google, FWIW, but more outside than in to be fair). The correct philosophy isn't "all tests must be integrated in CI", it's "CI must be able to integrate all tests".

This is a non-sequitor I think. A bad ci system is independent from the mandate that, functionally "unless we can verify we don't break you, you can't complain when we do".

ajross 3y ago

No? The two statements at hand equate to: "Do not add features to the product without adding validation to CI" and "Do not write validation steps CI just to add features to the product". Those are pretty clearly related concepts. And the reason for the disconnect is that the former centers the requirements of integration and not development, which is exactly backwards. CI isn't "the place where you run tests". The place where you run tests is underneath the software developers' fingers. CI is where you integrate tests developers are already running. And to the extent that is not true, CI stops being a useful tool for development and turns into just another roadblock to evade.

joshuamorton 3y ago
> No? The two statements at hand equate to: "Do not add features to the product without adding validation to CI" and "Do not write validation steps CI just to add features to the product". Those are pretty clearly related concepts. Only if your CI and developer workflow are different. To a first approximation, at Google, I run `blaze test //foo` to test `foo`, and CI invokes the same command when testing `foo`. There's caveats here, but they don't apply to the beyonce rule in practice, which is centered on unit tests that can be run hermetically on presubmit. Your weird 3-hour integration test won't be run. Edit: Actually it seems like you're saying "once you've tested a feature once, you can release it", which...no. You need some way of ensuring you don't regress, the CI suite is how you do that, otherwise no one will know when the new release of some random open source library that you depend on breaks your application.
- ajross 3y ago
  Clearly we agree. I'm just saying that this "Beyone Rule", naively applied, leads to cargo cult CI disasters (I've seen this happen both within and outside of Google, FWIW, but more outside than in to be fair). The correct philosophy isn't "all tests must be integrated in CI", it's "CI must be able to integrate all tests".