4 ms·

On the density of SQLite's tests, Hwaci owns a test suite with MC/DC coverage[1][2], that Richard kindly loaned to John for this experiment. MC/DC coverage is u

On the density of SQLite's tests, Hwaci owns a test suite with MC/DC coverage[1][2], that Richard kindly loaned to John for this experiment. MC/DC coverage is used as a criterion for safety-critical software testing. Before hearing about SQLite, I had never seen anyone commit to MC/DC coverage without being forced to by a safety standard. This test suite is how John found so much stuff, and that's also how one can trust that the binaries produced by compilers tested by Richard Hipp work correctly even if there is some undefined behavior in the source code. [1] https://en.wikipedia.org/wiki/Modified_condition/decision_coverage https://en.wikipedia.org/wiki/Modified_condition/decision_co... [2] https://www.sqlite.org/testing.html#mcdc https://www.sqlite.org/testing.html#mcdc

nkurz 11y ago
I'm amazed by the quality of their tests, but still wonder if it's enough. The question I had going in was whether their test coverage looked all the possible branches in the source (ie, what the programmer intended) or all the branches in the binary (ie, post compiler "optimizations"). Looking at your second link, it ends up being the less desired second option, but since they are using -fprofile-arcs it may be the added profiling code defeats the unwanted optimizations. But still, a potential blind spot occurs when both the production and instrumented code have the same omissions. So while their level of testing is truly heroic, since the effect of UB optimization can be to omit branches that the compiler reasons can never happen, it seems like there still may be cases where source level coverage does not match up with binary coverage. Maybe they have some external way of noticing that this is happening? Edit: Just saw that there are some new comments from both John and Richard at the bottom of the blog post.
- kazinator 11y ago
  If some necessary code is optimized away, this can surely translate to the code missing some requirement or failing, which can be covered by a regression test case. Anyway, it sounds like the testing is better than what is applied to the compilers they are using.

nkurz 11y ago

I'm amazed by the quality of their tests, but still wonder if it's enough. The question I had going in was whether their test coverage looked all the possible branches in the source (ie, what the programmer intended) or all the branches in the binary (ie, post compiler "optimizations"). Looking at your second link, it ends up being the less desired second option, but since they are using -fprofile-arcs it may be the added profiling code defeats the unwanted optimizations. But still, a potential blind spot occurs when both the production and instrumented code have the same omissions. So while their level of testing is truly heroic, since the effect of UB optimization can be to omit branches that the compiler reasons can never happen, it seems like there still may be cases where source level coverage does not match up with binary coverage. Maybe they have some external way of noticing that this is happening? Edit: Just saw that there are some new comments from both John and Richard at the bottom of the blog post.

kazinator 11y ago
If some necessary code is optimized away, this can surely translate to the code missing some requirement or failing, which can be covered by a regression test case. Anyway, it sounds like the testing is better than what is applied to the compilers they are using.