5 ms·

The go test command now caches test results: if the test executable and command line match a previous run and the files and environment variables consulte

The go test command now caches test results: if the test executable and command line match a previous run and the files and environment variables consulted by that run have not changed either, go test will print the previous test output, replacing the elapsed time with the string “(cached).” This is my favorite part of the release.

kasey_junk 9y ago
I don't understand the utility. Why am I running tests again if nothing has changed? Is this just to speed up multipackage testing use cases?
- ben_jones 9y ago
  I imagine if you're working on a project with 3 packages and only 1 package changed then the test will automatically skip the other 2 with no flags or changed command needed? Or perhaps if your working on one large package and only one file changed?
  - kasey_junk 9y ago
    Yes that’s my understanding.
- kjksf 9y ago
  Atom's Go integration re-runs tests after save to a .go file. It's very convenient because you can see test failures and test coverage changes immediately. But it's slow if you modified a file that is not involved in tested code paths. This is just one concrete example. In general when the code size grows, very quickly it becomes impossible to tell if a given edit could possibly affect tests or not. So you want to run `go test` just in case so making it faster is a good thing.
  - kasey_junk 9y ago
    But this isn’t doing ‘intelligent’ caching, just like go doesn’t do intelligent test checking. If the test binary changes (I think that implies basically any change in the package) the tests rerun.
    - kjksf 9y ago
      In general the bar for making changes is: is it better that it was yesterday? Before 1.10 `go test` used to re-run tests unconditionally. Since 1.10 it'll avoid re-running them in some cases. Neither me nor the announcement claims "intelligence". The rules are simple and effective which is the Go Way. While you could try to make even better, more "intelligent" rules for determining wether to re-run the tests, those would quickly run into engineering complexity chasing diminishing returns. That is not the Go Way.
      - kasey_junk 9y ago
        There are lots of things on my list you could do that were simple & effective that would provide me more value than this. Historically the argument against them have been about adding needless complexity. I wonder who was asking for this and how they got it prioritized.
        rhencke 9y ago
        It was requested by rsc in 2015: https://github.com/golang/go/issues/11193 https://github.com/golang/go/issues/11193 It was implemented by rsc subsequently, after discussion on the issue: https://go-review.googlesource.com/c/go/+/75631 https://go-review.googlesource.com/c/go/+/75631 I'm not sure what you're trying to insinuate.
        kasey_junk 9y ago
        I’m not insinuating anything. I’m flat saying this seems like a low value feature. Other higher value features are denied on the basis of unneeded complexity.
- rafi_kamal 9y ago
  Not a Go developer but bazel in Java does the same thing and it's really helpful when writing test cases. Say I am editing two test methods within a test suite of 30 test cases, it will run only those two tests. For the rest 28 it will show the cached output.
  - kasey_junk 9y ago
    But thats the thing, this is not incremental compile/test. Its just checking the binary output (at least as far as I can tell).
  - paulddraper 9y ago
    That's different. Go will rerun all tests (for that binary)
- kevpie 9y ago
  Larger projects can now effectively be TDD'd in. Or rather you modify or fix a test in a test file and the tests compile and execute very fast. Or even simply the call to `go run`, `go build` and `go test` will be significantly faster. This is the kind of difference I'm getting GOCACHE=off time go build 16.36 real 37.02 user 6.43 sys GOCACHE=on time go build 0.66 real 0.53 user 0.37 sys Number of files including vendor dependencies (some files may not actually be imported) find . -name '*.go' | wc -l 2031
  - kasey_junk 9y ago
    Those 2 runs are between 2 runs where something different change right? Thats effectively just outsourcing the 'did I change anything' memory to the cache system? I'm ok with that I guess and I suppose I see the benefit when running multiple package test runs where their aren't dependencies between them. But I've just not seen the utility in practice, and it seems like it can introduce bugs where the cache is used incorrectly.
    - kevpie 9y ago
      You are right, no changes. With a change, most objects are still valid as cached and can result in similar speedup. I'm also wary of caching introducing bugs and intend to continue to CI without it as a safety net. May even require a no cache build before pushing commits.
lobster_johnson 9y ago
What does "files ... consulted by that run" mean? Does it track file/directory operations that occur during a test? I'm a heavy user of table-driven tests that typically use YAML files that set up the parameters for the test scenarios. These use filepath.Glob(), ioutil.ReadDir(), and so on. For "go test" to invalidate its cache correctly, it would have to intercept those calls and record the dependencies. That'd be awesome.
- Scaevolus 9y ago
  Only source files, unfortunately. You could use a caching build system like Bazel to fork with data-driven tests, but most users don't need that functionality. It would be nice if tests could give hints like t.DisableCache() or t.DependsOn(filename).
  - uluyol 9y ago
    I think this is misinformation. go test does track files read during tests and records a hash of their contents. Bug: https://github.com/golang/go/issues/22593 https://github.com/golang/go/issues/22593 Commit: https://go-review.googlesource.com/c/go/+/81895 https://go-review.googlesource.com/c/go/+/81895
  - lobster_johnson 9y ago
    I looked at the code mentioned in the sibling comment. They do track file access. Incredibly, they've modified the standard library (os.Exec(), Stat() and so on) to log the activity to an internal mechanism used by the testing package. I'm rather flabbergasted by the close coupling (it's a feature that would've been impossible for anyone outside the Go team to implement), though it's certainly a pragmatic solution. Tracking seems to only occur within the current process, so if you call out to a subprocess to generate files, those files won't be taken into account, but I've not gone through the code with a fine comb.
    - HankB99 9y ago
      In other words, there are ways to 'fool' the dependency tree. My first thought was that this might be a contrived situation. Then I realized I have exactly that in a project I'm presently working on. It does a lot of file system operations and I use shell scripts invoked from test code to set up files for testing. I'll have to investigate how to force tests to run if I change test scripts. I suppose changing the `go` test case to result in a test failure and then modifying the script to produce correct results (or a valid failure ;) ) would do the trick. I guess changing the scripts would be detected by `go` so perhaps I don't have a problem. At present none of my scripts invoke other scripts.

kasey_junk 9y ago

I don't understand the utility. Why am I running tests again if nothing has changed? Is this just to speed up multipackage testing use cases?

ben_jones 9y ago
I imagine if you're working on a project with 3 packages and only 1 package changed then the test will automatically skip the other 2 with no flags or changed command needed? Or perhaps if your working on one large package and only one file changed?
- kasey_junk 9y ago
  Yes that’s my understanding.
kjksf 9y ago
Atom's Go integration re-runs tests after save to a .go file. It's very convenient because you can see test failures and test coverage changes immediately. But it's slow if you modified a file that is not involved in tested code paths. This is just one concrete example. In general when the code size grows, very quickly it becomes impossible to tell if a given edit could possibly affect tests or not. So you want to run `go test` just in case so making it faster is a good thing.
- kasey_junk 9y ago
  But this isn’t doing ‘intelligent’ caching, just like go doesn’t do intelligent test checking. If the test binary changes (I think that implies basically any change in the package) the tests rerun.
  - kjksf 9y ago
    In general the bar for making changes is: is it better that it was yesterday? Before 1.10 `go test` used to re-run tests unconditionally. Since 1.10 it'll avoid re-running them in some cases. Neither me nor the announcement claims "intelligence". The rules are simple and effective which is the Go Way. While you could try to make even better, more "intelligent" rules for determining wether to re-run the tests, those would quickly run into engineering complexity chasing diminishing returns. That is not the Go Way.
    - kasey_junk 9y ago
      There are lots of things on my list you could do that were simple & effective that would provide me more value than this. Historically the argument against them have been about adding needless complexity. I wonder who was asking for this and how they got it prioritized.
      - rhencke 9y ago
        It was requested by rsc in 2015: https://github.com/golang/go/issues/11193 https://github.com/golang/go/issues/11193 It was implemented by rsc subsequently, after discussion on the issue: https://go-review.googlesource.com/c/go/+/75631 https://go-review.googlesource.com/c/go/+/75631 I'm not sure what you're trying to insinuate.
        kasey_junk 9y ago
        I’m not insinuating anything. I’m flat saying this seems like a low value feature. Other higher value features are denied on the basis of unneeded complexity.
rafi_kamal 9y ago
Not a Go developer but bazel in Java does the same thing and it's really helpful when writing test cases. Say I am editing two test methods within a test suite of 30 test cases, it will run only those two tests. For the rest 28 it will show the cached output.
- kasey_junk 9y ago
  But thats the thing, this is not incremental compile/test. Its just checking the binary output (at least as far as I can tell).
- paulddraper 9y ago
  That's different. Go will rerun all tests (for that binary)
kevpie 9y ago
Larger projects can now effectively be TDD'd in. Or rather you modify or fix a test in a test file and the tests compile and execute very fast. Or even simply the call to `go run`, `go build` and `go test` will be significantly faster. This is the kind of difference I'm getting GOCACHE=off time go build 16.36 real 37.02 user 6.43 sys GOCACHE=on time go build 0.66 real 0.53 user 0.37 sys Number of files including vendor dependencies (some files may not actually be imported) find . -name '*.go' | wc -l 2031
- kasey_junk 9y ago
  Those 2 runs are between 2 runs where something different change right? Thats effectively just outsourcing the 'did I change anything' memory to the cache system? I'm ok with that I guess and I suppose I see the benefit when running multiple package test runs where their aren't dependencies between them. But I've just not seen the utility in practice, and it seems like it can introduce bugs where the cache is used incorrectly.
  - kevpie 9y ago
    You are right, no changes. With a change, most objects are still valid as cached and can result in similar speedup. I'm also wary of caching introducing bugs and intend to continue to CI without it as a safety net. May even require a no cache build before pushing commits.

lobster_johnson 9y ago

What does "files ... consulted by that run" mean? Does it track file/directory operations that occur during a test? I'm a heavy user of table-driven tests that typically use YAML files that set up the parameters for the test scenarios. These use filepath.Glob(), ioutil.ReadDir(), and so on. For "go test" to invalidate its cache correctly, it would have to intercept those calls and record the dependencies. That'd be awesome.

Scaevolus 9y ago
Only source files, unfortunately. You could use a caching build system like Bazel to fork with data-driven tests, but most users don't need that functionality. It would be nice if tests could give hints like t.DisableCache() or t.DependsOn(filename).
- uluyol 9y ago
  I think this is misinformation. go test does track files read during tests and records a hash of their contents. Bug: https://github.com/golang/go/issues/22593 https://github.com/golang/go/issues/22593 Commit: https://go-review.googlesource.com/c/go/+/81895 https://go-review.googlesource.com/c/go/+/81895
- lobster_johnson 9y ago
  I looked at the code mentioned in the sibling comment. They do track file access. Incredibly, they've modified the standard library (os.Exec(), Stat() and so on) to log the activity to an internal mechanism used by the testing package. I'm rather flabbergasted by the close coupling (it's a feature that would've been impossible for anyone outside the Go team to implement), though it's certainly a pragmatic solution. Tracking seems to only occur within the current process, so if you call out to a subprocess to generate files, those files won't be taken into account, but I've not gone through the code with a fine comb.
  - HankB99 9y ago
    In other words, there are ways to 'fool' the dependency tree. My first thought was that this might be a contrived situation. Then I realized I have exactly that in a project I'm presently working on. It does a lot of file system operations and I use shell scripts invoked from test code to set up files for testing. I'll have to investigate how to force tests to run if I change test scripts. I suppose changing the `go` test case to result in a test failure and then modifying the script to produce correct results (or a valid failure ;) ) would do the trick. I guess changing the scripts would be detected by `go` so perhaps I don't have a problem. At present none of my scripts invoke other scripts.