5 ms·

Code coverage for Go integration tests

162 points by ingve 4y ago

eatonphil 4y ago
There was a technique that existed already where you could use `go test -cover` and the `-o` flag to produce a binary from `go test` rather than actually running tests. So you could build a binary that had coverage enabled. Then when you exercised code paths by running that binary with whatever inputs, coverage would be collected and dumped into some local file. Here's an example of the build step: https://github.com/multiprocessio/datastation/blob/main/runner/scripts/runner_test.build#L3 https://github.com/multiprocessio/datastation/blob/main/runn.... And here's how I'd analyze coverage after using the binary: https://github.com/multiprocessio/datastation/blob/main/runner/scripts/test_coverage.sh https://github.com/multiprocessio/datastation/blob/main/runn.... This analysis actually combined both unit test coverage files with integration test coverage so it gave a pretty good overall indication of code paths covered by any tests. I can't remember where I found this technique but it's been around for a while. This new option is the same thing but a way to `go build` with `-cover` instead of `go test -cover -o $out`? Do I have that right? Another example of doing this from 2019: https://stackoverflow.com/a/55640874/1507139 https://stackoverflow.com/a/55640874/1507139. I'm sure I learned about this from somebody's blog but I can't remember where.
- FiloSottile 4y ago
  No idea if the technique was original, it’s been too long, but I wrote about it on the Cloudflare blog in 2016. https://blog.cloudflare.com/go-coverage-with-external-tests/ https://blog.cloudflare.com/go-coverage-with-external-tests/ Anyway, very happy to see the team ship first class support for it. It immediately made coverage support in testscript much more natural, too. https://github.com/rogpeppe/go-internal/pull/201 https://github.com/rogpeppe/go-internal/pull/201
  - aleksi 4y ago
    Elastic wrote about that in 2015: https://www.elastic.co/blog/code-coverage-for-your-golang-system-tests https://www.elastic.co/blog/code-coverage-for-your-golang-sy.... I guess multiple people figured it out independently.
- eliben 4y ago
  Yes, technically you could wrap every `main` in a test and build everything just right, but it's nice to be able to do this without any hacks if you already have an integration test suite and you want to know what coverage it gets you.
amenghra 4y ago
In case you have never used it, https://app.codecov.io/ https://app.codecov.io/ can watch your GitHub repo and run tests + coverage. There's a nice UI to then see the coverage % for each folder and also whether a PR or commit is increasing or decreasing coverage. If I recall correctly, there's zero configuration needed for Go projects. You can be up and running in minutes. I think Codecov is free for open source projects and paid for private repos, someone more knowledgeable can chime in.
- zeeg 4y ago
  ♥ thank you! If y’all have feedback for what we can do better with Codecov let us know.
  - aleksi 4y ago
    I wish your “redesigns” did not remove useful features: https://app.codecov.io/gh/feedback/p/view-source-with-flags https://app.codecov.io/gh/feedback/p/view-source-with-flags
- deleted 4y ago
  [deleted]
adamch 4y ago
This is a really cool feature, thanks very much for adding it. I'd love to see this in more languages. I'd love to use it in my Rust integration tests.
pdubouilh 4y ago
A bit more convoluted, but it's also possible in Rust https://github.com/pldubouilh/rust-coverage-integration-test https://github.com/pldubouilh/rust-coverage-integration-test
deleted 4y ago
[deleted]
qaq 4y ago
Go tooling keeps getting better
nitwit005 4y ago
Cool. I definitely felt the lack of this.
bhawks 4y ago
I find the value of code coverage goes down when the size of the instrumented codebase goes up. It's helpful for smaller tests as you can eyeball where you may be missing cases but on integration tests you end up getting a high coverage percentage just by bringing up the system and not even asserting anything (I guess it doesn't crash :) ). If you want to methodically measure integration test coverage it seems more valuable to enumerate all your critical features or user flows and attribute test cases to them.
- gabereiser 4y ago
  Proper testing is your issue, not coverage. Coverage is to know where it’s missing. Proper testing (positive AND negative cases) is extremely valuable. Not only to ensure your code works properly, but also as documentation on how to use your APIs. Integration testing is where most testing cycles fail. It’s also more difficult for the developer because now they need an emulator, a container, or some sandbox to test the integration. Feature testing is one thing but I’d like to know if the foundation works, that the plumbing is there.
janosdebugs 4y ago
A colleague of mine wrote a tool to make the process of dealing with coverage data a bit simpler: https://github.com/gotesttools/limgo https://github.com/gotesttools/limgo
aleksi 4y ago
It is (non-sarcastically) good to see the Go team changing their opinion on the usefulness of it: https://github.com/golang/go/issues/16537 https://github.com/golang/go/issues/16537
bilinguliar 4y ago
I guess this can be used to eliminate dead code from your projects as well. Cases that you thought were executed but, in reality, are not. I am curious if the overhead of this feature allows for running it in production for some time, let’s say a day.
drewcoo 4y ago
A few years ago, I used goc to collect coverage from multiple services at the same time when they were used in E2E tests. I was very happy with it. Check it out! https://github.com/qiniu/goc https://github.com/qiniu/goc
donatj 4y ago
Oh, interesting. Full on integration testing for Go CLI apps is an area in which I am sorely lacking. Does anyone have recommendations of tooling to generally help instrument testing CLI tools? Hopefully better than what I’ve done thus far which is basically a shell script of ./mybinary <<< $INPUT > output.txt 2> error.txt diff expected-output.txt output.txt diff expected-error.txt error.txt
nzach 4y ago
Could we hack this to better understand what your code is doing in prod? Let me describe a scenario: Let's say you just inherited some service that is already running in production but it don't have a great documentation nor a good test suite. To make matters worse you don't really know how this service is being used and what are the critical paths inside it. What I'm thinking is if it would be possible to use this feature to collect some real world usage data for this application. And this data could give you a better understanding about what is actually important in the service. For APIs we already have a part of this data, as most server in Go have the number of requests for each endpoint. But I think in many cases a more granular data could be really helpful. Yes, I know that the number of times some function was called isn't a reliable indicator of the importance of said function. But it should be better than nothing. And I also know that something like this would have a negative performance effect that would not be acceptable for some situations.

162 points by ingve 4y ago

There was a technique that existed already where you could use `go test -cover` and the `-o` flag to produce a binary from `go test` rather than actually running tests. So you could build a binary that had coverage enabled. Then when you exercised code paths by running that binary with whatever inputs, coverage would be collected and dumped into some local file. Here's an example of the build step: https://github.com/multiprocessio/datastation/blob/main/runner/scripts/runner_test.build#L3 https://github.com/multiprocessio/datastation/blob/main/runn.... And here's how I'd analyze coverage after using the binary: https://github.com/multiprocessio/datastation/blob/main/runner/scripts/test_coverage.sh https://github.com/multiprocessio/datastation/blob/main/runn.... This analysis actually combined both unit test coverage files with integration test coverage so it gave a pretty good overall indication of code paths covered by any tests. I can't remember where I found this technique but it's been around for a while. This new option is the same thing but a way to `go build` with `-cover` instead of `go test -cover -o $out`? Do I have that right? Another example of doing this from 2019: https://stackoverflow.com/a/55640874/1507139 https://stackoverflow.com/a/55640874/1507139. I'm sure I learned about this from somebody's blog but I can't remember where.

FiloSottile 4y ago
No idea if the technique was original, it’s been too long, but I wrote about it on the Cloudflare blog in 2016. https://blog.cloudflare.com/go-coverage-with-external-tests/ https://blog.cloudflare.com/go-coverage-with-external-tests/ Anyway, very happy to see the team ship first class support for it. It immediately made coverage support in testscript much more natural, too. https://github.com/rogpeppe/go-internal/pull/201 https://github.com/rogpeppe/go-internal/pull/201
- aleksi 4y ago
  Elastic wrote about that in 2015: https://www.elastic.co/blog/code-coverage-for-your-golang-system-tests https://www.elastic.co/blog/code-coverage-for-your-golang-sy.... I guess multiple people figured it out independently.
eliben 4y ago
Yes, technically you could wrap every `main` in a test and build everything just right, but it's nice to be able to do this without any hacks if you already have an integration test suite and you want to know what coverage it gets you.

amenghra 4y ago

In case you have never used it, https://app.codecov.io/ https://app.codecov.io/ can watch your GitHub repo and run tests + coverage. There's a nice UI to then see the coverage % for each folder and also whether a PR or commit is increasing or decreasing coverage. If I recall correctly, there's zero configuration needed for Go projects. You can be up and running in minutes. I think Codecov is free for open source projects and paid for private repos, someone more knowledgeable can chime in.

zeeg 4y ago
♥ thank you! If y’all have feedback for what we can do better with Codecov let us know.
- aleksi 4y ago
  I wish your “redesigns” did not remove useful features: https://app.codecov.io/gh/feedback/p/view-source-with-flags https://app.codecov.io/gh/feedback/p/view-source-with-flags
deleted 4y ago
[deleted]

adamch 4y ago

This is a really cool feature, thanks very much for adding it. I'd love to see this in more languages. I'd love to use it in my Rust integration tests.

pdubouilh 4y ago

A bit more convoluted, but it's also possible in Rust https://github.com/pldubouilh/rust-coverage-integration-test https://github.com/pldubouilh/rust-coverage-integration-test

deleted 4y ago

[deleted]

qaq 4y ago

Go tooling keeps getting better

nitwit005 4y ago

Cool. I definitely felt the lack of this.

bhawks 4y ago

I find the value of code coverage goes down when the size of the instrumented codebase goes up. It's helpful for smaller tests as you can eyeball where you may be missing cases but on integration tests you end up getting a high coverage percentage just by bringing up the system and not even asserting anything (I guess it doesn't crash :) ). If you want to methodically measure integration test coverage it seems more valuable to enumerate all your critical features or user flows and attribute test cases to them.

gabereiser 4y ago
Proper testing is your issue, not coverage. Coverage is to know where it’s missing. Proper testing (positive AND negative cases) is extremely valuable. Not only to ensure your code works properly, but also as documentation on how to use your APIs. Integration testing is where most testing cycles fail. It’s also more difficult for the developer because now they need an emulator, a container, or some sandbox to test the integration. Feature testing is one thing but I’d like to know if the foundation works, that the plumbing is there.

janosdebugs 4y ago

A colleague of mine wrote a tool to make the process of dealing with coverage data a bit simpler: https://github.com/gotesttools/limgo https://github.com/gotesttools/limgo

aleksi 4y ago

It is (non-sarcastically) good to see the Go team changing their opinion on the usefulness of it: https://github.com/golang/go/issues/16537 https://github.com/golang/go/issues/16537

bilinguliar 4y ago

I guess this can be used to eliminate dead code from your projects as well. Cases that you thought were executed but, in reality, are not. I am curious if the overhead of this feature allows for running it in production for some time, let’s say a day.

drewcoo 4y ago

A few years ago, I used goc to collect coverage from multiple services at the same time when they were used in E2E tests. I was very happy with it. Check it out! https://github.com/qiniu/goc https://github.com/qiniu/goc

donatj 4y ago

Oh, interesting. Full on integration testing for Go CLI apps is an area in which I am sorely lacking. Does anyone have recommendations of tooling to generally help instrument testing CLI tools? Hopefully better than what I’ve done thus far which is basically a shell script of ./mybinary <<< $INPUT > output.txt 2> error.txt diff expected-output.txt output.txt diff expected-error.txt error.txt

nzach 4y ago

Could we hack this to better understand what your code is doing in prod? Let me describe a scenario: Let's say you just inherited some service that is already running in production but it don't have a great documentation nor a good test suite. To make matters worse you don't really know how this service is being used and what are the critical paths inside it. What I'm thinking is if it would be possible to use this feature to collect some real world usage data for this application. And this data could give you a better understanding about what is actually important in the service. For APIs we already have a part of this data, as most server in Go have the number of requests for each endpoint. But I think in many cases a more granular data could be really helpful. Yes, I know that the number of times some function was called isn't a reliable indicator of the importance of said function. But it should be better than nothing. And I also know that something like this would have a negative performance effect that would not be acceptable for some situations.