3 ms·

Pet peeve: There's an anticorrolary though, which is that tests written solely for the requirements of an elaborate CI system tend to hurt and not help. Tests

by ajross 3y ago

Pet peeve: There's an anticorrolary though, which is that tests written solely for the requirements of an elaborate CI system tend to hurt and not help. Tests need to be trivially executable and inspectable by developers running on their own systems with minimal support. If they aren't, then all you know is that "It's Broken!" and not how to fix it. Good CI is good because it easily integrates "one-off bespoke tests", not because it outlaws them.

joshuamorton 3y ago
This is a non-sequitor I think. A bad ci system is independent from the mandate that, functionally "unless we can verify we don't break you, you can't complain when we do".
- ajross 3y ago
  No? The two statements at hand equate to: "Do not add features to the product without adding validation to CI" and "Do not write validation steps CI just to add features to the product". Those are pretty clearly related concepts. And the reason for the disconnect is that the former centers the requirements of integration and not development, which is exactly backwards. CI isn't "the place where you run tests". The place where you run tests is underneath the software developers' fingers. CI is where you integrate tests developers are already running. And to the extent that is not true, CI stops being a useful tool for development and turns into just another roadblock to evade.
  - joshuamorton 3y ago
    > No? The two statements at hand equate to: "Do not add features to the product without adding validation to CI" and "Do not write validation steps CI just to add features to the product". Those are pretty clearly related concepts. Only if your CI and developer workflow are different. To a first approximation, at Google, I run `blaze test //foo` to test `foo`, and CI invokes the same command when testing `foo`. There's caveats here, but they don't apply to the beyonce rule in practice, which is centered on unit tests that can be run hermetically on presubmit. Your weird 3-hour integration test won't be run. Edit: Actually it seems like you're saying "once you've tested a feature once, you can release it", which...no. You need some way of ensuring you don't regress, the CI suite is how you do that, otherwise no one will know when the new release of some random open source library that you depend on breaks your application.
    - ajross 3y ago
      Clearly we agree. I'm just saying that this "Beyone Rule", naively applied, leads to cargo cult CI disasters (I've seen this happen both within and outside of Google, FWIW, but more outside than in to be fair). The correct philosophy isn't "all tests must be integrated in CI", it's "CI must be able to integrate all tests".

by ajross 3y ago

joshuamorton 3y ago

This is a non-sequitor I think. A bad ci system is independent from the mandate that, functionally "unless we can verify we don't break you, you can't complain when we do".

ajross 3y ago
No? The two statements at hand equate to: "Do not add features to the product without adding validation to CI" and "Do not write validation steps CI just to add features to the product". Those are pretty clearly related concepts. And the reason for the disconnect is that the former centers the requirements of integration and not development, which is exactly backwards. CI isn't "the place where you run tests". The place where you run tests is underneath the software developers' fingers. CI is where you integrate tests developers are already running. And to the extent that is not true, CI stops being a useful tool for development and turns into just another roadblock to evade.
- joshuamorton 3y ago
  > No? The two statements at hand equate to: "Do not add features to the product without adding validation to CI" and "Do not write validation steps CI just to add features to the product". Those are pretty clearly related concepts. Only if your CI and developer workflow are different. To a first approximation, at Google, I run `blaze test //foo` to test `foo`, and CI invokes the same command when testing `foo`. There's caveats here, but they don't apply to the beyonce rule in practice, which is centered on unit tests that can be run hermetically on presubmit. Your weird 3-hour integration test won't be run. Edit: Actually it seems like you're saying "once you've tested a feature once, you can release it", which...no. You need some way of ensuring you don't regress, the CI suite is how you do that, otherwise no one will know when the new release of some random open source library that you depend on breaks your application.
  - ajross 3y ago
    Clearly we agree. I'm just saying that this "Beyone Rule", naively applied, leads to cargo cult CI disasters (I've seen this happen both within and outside of Google, FWIW, but more outside than in to be fair). The correct philosophy isn't "all tests must be integrated in CI", it's "CI must be able to integrate all tests".