5 ms·

I don't understand the utility. Why am I running tests again if nothing has changed? Is this just to speed up multipackage testing use cases?

ben_jones 9y ago
I imagine if you're working on a project with 3 packages and only 1 package changed then the test will automatically skip the other 2 with no flags or changed command needed? Or perhaps if your working on one large package and only one file changed?
- kasey_junk 9y ago
  Yes that’s my understanding.
kjksf 9y ago
Atom's Go integration re-runs tests after save to a .go file. It's very convenient because you can see test failures and test coverage changes immediately. But it's slow if you modified a file that is not involved in tested code paths. This is just one concrete example. In general when the code size grows, very quickly it becomes impossible to tell if a given edit could possibly affect tests or not. So you want to run `go test` just in case so making it faster is a good thing.
- kasey_junk 9y ago
  But this isn’t doing ‘intelligent’ caching, just like go doesn’t do intelligent test checking. If the test binary changes (I think that implies basically any change in the package) the tests rerun.
  - kjksf 9y ago
    In general the bar for making changes is: is it better that it was yesterday? Before 1.10 `go test` used to re-run tests unconditionally. Since 1.10 it'll avoid re-running them in some cases. Neither me nor the announcement claims "intelligence". The rules are simple and effective which is the Go Way. While you could try to make even better, more "intelligent" rules for determining wether to re-run the tests, those would quickly run into engineering complexity chasing diminishing returns. That is not the Go Way.
    - kasey_junk 9y ago
      There are lots of things on my list you could do that were simple & effective that would provide me more value than this. Historically the argument against them have been about adding needless complexity. I wonder who was asking for this and how they got it prioritized.
      - rhencke 9y ago
        It was requested by rsc in 2015: https://github.com/golang/go/issues/11193 https://github.com/golang/go/issues/11193 It was implemented by rsc subsequently, after discussion on the issue: https://go-review.googlesource.com/c/go/+/75631 https://go-review.googlesource.com/c/go/+/75631 I'm not sure what you're trying to insinuate.
        kasey_junk 9y ago
        I’m not insinuating anything. I’m flat saying this seems like a low value feature. Other higher value features are denied on the basis of unneeded complexity.
        maloney 9y ago
        I mean, the project lead wanted a feature and then implemented the feature, clearly it had value to him.
        slrz 9y ago
        The difference is that this feature (ideally) only affects implementation complexity, not interface complexity. It can also be dropped again tomorrow without breaking the world. This makes it wildly different from language changes, which require a lot more consideration.
rafi_kamal 9y ago
Not a Go developer but bazel in Java does the same thing and it's really helpful when writing test cases. Say I am editing two test methods within a test suite of 30 test cases, it will run only those two tests. For the rest 28 it will show the cached output.
- kasey_junk 9y ago
  But thats the thing, this is not incremental compile/test. Its just checking the binary output (at least as far as I can tell).
- paulddraper 9y ago
  That's different. Go will rerun all tests (for that binary)
kevpie 9y ago
Larger projects can now effectively be TDD'd in. Or rather you modify or fix a test in a test file and the tests compile and execute very fast. Or even simply the call to `go run`, `go build` and `go test` will be significantly faster. This is the kind of difference I'm getting GOCACHE=off time go build 16.36 real 37.02 user 6.43 sys GOCACHE=on time go build 0.66 real 0.53 user 0.37 sys Number of files including vendor dependencies (some files may not actually be imported) find . -name '*.go' | wc -l 2031
- kasey_junk 9y ago
  Those 2 runs are between 2 runs where something different change right? Thats effectively just outsourcing the 'did I change anything' memory to the cache system? I'm ok with that I guess and I suppose I see the benefit when running multiple package test runs where their aren't dependencies between them. But I've just not seen the utility in practice, and it seems like it can introduce bugs where the cache is used incorrectly.
  - kevpie 9y ago
    You are right, no changes. With a change, most objects are still valid as cached and can result in similar speedup. I'm also wary of caching introducing bugs and intend to continue to CI without it as a safety net. May even require a no cache build before pushing commits.

ben_jones 9y ago

I imagine if you're working on a project with 3 packages and only 1 package changed then the test will automatically skip the other 2 with no flags or changed command needed? Or perhaps if your working on one large package and only one file changed?

kasey_junk 9y ago
Yes that’s my understanding.

kjksf 9y ago

Atom's Go integration re-runs tests after save to a .go file. It's very convenient because you can see test failures and test coverage changes immediately. But it's slow if you modified a file that is not involved in tested code paths. This is just one concrete example. In general when the code size grows, very quickly it becomes impossible to tell if a given edit could possibly affect tests or not. So you want to run `go test` just in case so making it faster is a good thing.

kasey_junk 9y ago
But this isn’t doing ‘intelligent’ caching, just like go doesn’t do intelligent test checking. If the test binary changes (I think that implies basically any change in the package) the tests rerun.
- kjksf 9y ago
  In general the bar for making changes is: is it better that it was yesterday? Before 1.10 `go test` used to re-run tests unconditionally. Since 1.10 it'll avoid re-running them in some cases. Neither me nor the announcement claims "intelligence". The rules are simple and effective which is the Go Way. While you could try to make even better, more "intelligent" rules for determining wether to re-run the tests, those would quickly run into engineering complexity chasing diminishing returns. That is not the Go Way.
  - kasey_junk 9y ago
    There are lots of things on my list you could do that were simple & effective that would provide me more value than this. Historically the argument against them have been about adding needless complexity. I wonder who was asking for this and how they got it prioritized.
    - rhencke 9y ago
      It was requested by rsc in 2015: https://github.com/golang/go/issues/11193 https://github.com/golang/go/issues/11193 It was implemented by rsc subsequently, after discussion on the issue: https://go-review.googlesource.com/c/go/+/75631 https://go-review.googlesource.com/c/go/+/75631 I'm not sure what you're trying to insinuate.
      - kasey_junk 9y ago
        I’m not insinuating anything. I’m flat saying this seems like a low value feature. Other higher value features are denied on the basis of unneeded complexity.
        maloney 9y ago
        I mean, the project lead wanted a feature and then implemented the feature, clearly it had value to him.
        slrz 9y ago
        The difference is that this feature (ideally) only affects implementation complexity, not interface complexity. It can also be dropped again tomorrow without breaking the world. This makes it wildly different from language changes, which require a lot more consideration.

rafi_kamal 9y ago

Not a Go developer but bazel in Java does the same thing and it's really helpful when writing test cases. Say I am editing two test methods within a test suite of 30 test cases, it will run only those two tests. For the rest 28 it will show the cached output.

kasey_junk 9y ago
But thats the thing, this is not incremental compile/test. Its just checking the binary output (at least as far as I can tell).
paulddraper 9y ago
That's different. Go will rerun all tests (for that binary)

kevpie 9y ago

Larger projects can now effectively be TDD'd in. Or rather you modify or fix a test in a test file and the tests compile and execute very fast. Or even simply the call to `go run`, `go build` and `go test` will be significantly faster. This is the kind of difference I'm getting GOCACHE=off time go build 16.36 real 37.02 user 6.43 sys GOCACHE=on time go build 0.66 real 0.53 user 0.37 sys Number of files including vendor dependencies (some files may not actually be imported) find . -name '*.go' | wc -l 2031

kasey_junk 9y ago
Those 2 runs are between 2 runs where something different change right? Thats effectively just outsourcing the 'did I change anything' memory to the cache system? I'm ok with that I guess and I suppose I see the benefit when running multiple package test runs where their aren't dependencies between them. But I've just not seen the utility in practice, and it seems like it can introduce bugs where the cache is used incorrectly.
- kevpie 9y ago
  You are right, no changes. With a change, most objects are still valid as cached and can result in similar speedup. I'm also wary of caching introducing bugs and intend to continue to CI without it as a safety net. May even require a no cache build before pushing commits.