5 ms·

Question for an insider. Bazel code coverage seems to be one of the roughest edges. At least when running C++ tests. Only lcov is supported, and due to the san

by carlmr 4y ago

Question for an insider. Bazel code coverage seems to be one of the roughest edges. At least when running C++ tests. Only lcov is supported, and due to the sandboxing it's impossible to get other tools working since they always spit out extra instrumentation data that's neither cached nor available on the next sandbox. Does blaze support code coverage properly? Does Google just not care about code coverage? Am I just missing some obvious thing in the documentation?

bjackman 4y ago
Hmm sorry I'm quite ignorant about that. Code coverage is pretty well supported in the internal ecosystem but I would not be surprised to hear this is because we designed Bazel around the system Google was already using for coverage.
- simplotek 4y ago
  > Code coverage is pretty well supported in the internal ecosystem (...) What you're actually saying is that internally Google managed to get a single and very specific use case to work and forced all projects to either adopt it or have no code coverage.
Game_Ender 4y ago
You can get coverage working for almost anything by having each test convert its coverage data to LCOV before stopping, in the same sandbox as the test. I forget the exact details but I think you can do this with a custom run_under test wrapper used during coverage. The wrapper runs your test, with whatever custom coverage system you have, then afterwards it converts to LCOV. Sometimes this might require updates to language specific rules because the test runner itself needs to be configured. The bottom line is that Bazel upstream does not put a lot of attention into this, so it’s under developed outside of Google, because Google internal does not use the OSS rules.
- carlmr 4y ago
  >I forget the exact details but I think you can do this with a custom run_under test wrapper used during coverage. The wrapper runs your test, with whatever custom coverage system you have, then afterwards it converts to LCOV. The testing is not all you need to modify. You need to modify every compile and link step, since you get byproducts of the instrumentation on every step. >Sometimes this might require updates to language specific rules because the test runner itself needs to be configured. That's what I found, too, however I feel like rewriting the C++ rules is a huge undertaking, since they're not defined in Starlark.
- simplotek 4y ago
  > You can get coverage working for almost anything by having each test convert its coverage data to LCOV before stopping, in the same sandbox as the test. I have to call bullshit on this. With C or C++ you need to build the whole project with specific compiler flags to be able to generate code coverage reports. This is not something you implement by tweaking a setting in unit tests.
sam_bristow 4y ago
I've also struggled with the C++ coverage support in Bazel. It looks unloved since around 2018. The most useful leads I've found so far are: * The design doc for the (abandoned?) coverage framework [1] * A prototype of GCOVR support [2] * Comment on getting llvm-profdata working [3] [1] https://docs.google.com/document/d/1-ZWHF-Q-qCKf19ik-t33ie58BkNurrYYzKR4OLtcilY/edit https://docs.google.com/document/d/1-ZWHF-Q-qCKf19ik-t33ie58... [2] https://github.com/ulfjack/bazel/commit/e9f21bbf562c1f6006eb8615569359755a57dd11 https://github.com/ulfjack/bazel/commit/e9f21bbf562c1f6006eb... [3] https://github.com/bazelbuild/bazel/issues/8178#issuecomment-488390198 https://github.com/bazelbuild/bazel/issues/8178#issuecomment...

by carlmr 4y ago

bjackman 4y ago

Hmm sorry I'm quite ignorant about that. Code coverage is pretty well supported in the internal ecosystem but I would not be surprised to hear this is because we designed Bazel around the system Google was already using for coverage.

simplotek 4y ago
> Code coverage is pretty well supported in the internal ecosystem (...) What you're actually saying is that internally Google managed to get a single and very specific use case to work and forced all projects to either adopt it or have no code coverage.

Game_Ender 4y ago

You can get coverage working for almost anything by having each test convert its coverage data to LCOV before stopping, in the same sandbox as the test. I forget the exact details but I think you can do this with a custom run_under test wrapper used during coverage. The wrapper runs your test, with whatever custom coverage system you have, then afterwards it converts to LCOV. Sometimes this might require updates to language specific rules because the test runner itself needs to be configured. The bottom line is that Bazel upstream does not put a lot of attention into this, so it’s under developed outside of Google, because Google internal does not use the OSS rules.

carlmr 4y ago
>I forget the exact details but I think you can do this with a custom run_under test wrapper used during coverage. The wrapper runs your test, with whatever custom coverage system you have, then afterwards it converts to LCOV. The testing is not all you need to modify. You need to modify every compile and link step, since you get byproducts of the instrumentation on every step. >Sometimes this might require updates to language specific rules because the test runner itself needs to be configured. That's what I found, too, however I feel like rewriting the C++ rules is a huge undertaking, since they're not defined in Starlark.
simplotek 4y ago
> You can get coverage working for almost anything by having each test convert its coverage data to LCOV before stopping, in the same sandbox as the test. I have to call bullshit on this. With C or C++ you need to build the whole project with specific compiler flags to be able to generate code coverage reports. This is not something you implement by tweaking a setting in unit tests.

sam_bristow 4y ago

I've also struggled with the C++ coverage support in Bazel. It looks unloved since around 2018. The most useful leads I've found so far are: * The design doc for the (abandoned?) coverage framework [1] * A prototype of GCOVR support [2] * Comment on getting llvm-profdata working [3] [1] https://docs.google.com/document/d/1-ZWHF-Q-qCKf19ik-t33ie58BkNurrYYzKR4OLtcilY/edit https://docs.google.com/document/d/1-ZWHF-Q-qCKf19ik-t33ie58... [2] https://github.com/ulfjack/bazel/commit/e9f21bbf562c1f6006eb8615569359755a57dd11 https://github.com/ulfjack/bazel/commit/e9f21bbf562c1f6006eb... [3] https://github.com/bazelbuild/bazel/issues/8178#issuecomment-488390198 https://github.com/bazelbuild/bazel/issues/8178#issuecomment...