4 ms·

I've tried using this to test my tests— take some code that has (presumably comprehensive tests) and C-reduce it to the smallest program that passes the test..

by nullc 12y ago

I've tried using this to test my tests— take some code that has (presumably comprehensive tests) and C-reduce it to the smallest program that passes the test... Then go see what important code it removed and fix the tests until it doesn't pass. Kinda fun, but tedious because it ruthlessly trashes the formatting, so matching up the before and after code is hard.

eridius 12y ago
I wonder how hard it is to write a program that will parse two C source files into AST's, diff the AST's, and then map the AST back to the original files and highlight where all the differences are?
- JoshTriplett 12y ago
  Quite difficult if you need to handle arbitrary C, primarily because of the preprocessor; preprocessing is a textual step, and before preprocessing, C code can't necessarily be turned into a sensible AST. You'd likely get better results by running both source files through an indenter and diffing the results. That's a strong argument for enforcing a canonical style via tools like "go fmt" or "indent".
dbaupp 12y ago
That sounds like you're effectively using it as a test coverage tool to find code paths that aren't being tested, would that be an accurate summary?
- nullc 12y ago
  Not quite, because code can still get run by a test but not be tested. This isn't all that interesting to do except on code that already has ~100% branch coverage. (Since obviously any unexecuted code is going to get removed, and lcov is a lot less work to use :) ) There aren't really any good mutation testing tools for C— e.g. tools to test your tests by breaking your code and making sure your tests fail. C-reduce can be used as a limited kind of mutation test— one that only tests reductions. One nice thing is that for code with 100% coverage any successful reduction is quite often a shortcoming in the tests or a missed optimization in the code, which isn't something you can say for most other kinds of mutations. (except for the fact that c-reduce does a lot of formatting garbling that doesn't really change the code)
  - dbaupp 12y ago
    Ah, nice. So it's more like a "semantic coverage" tool, i.e. finding the code paths that don't have any effect on the output.
jevinskie 12y ago
Try running the reduced code through clang-format.
rurounijones 12y ago
In the ruby world there is the "mutant" ( https://github.com/mbj/mutant https://github.com/mbj/mutant ) gem that does stuff like this automatically. It runs your tests and makes sure they all pass then it mutates the code under test (changing a == b to a != b or a <= 10 to a < 10) and makes sure that at least one of your tests fails. If your tests still pass after the mutation then you know you are not covering something. Tis a great idea but making a C version with the equivalent functionality would be a lot harder I think.
- michaelmior 12y ago
  Curious why you're thinking doing this in C would be a lot harder? All you really need to do is define what some valid transformations are. There are existing mutation testing tools for C[1]. [1] http://stackoverflow.com/questions/4715890/what-mutation-testing-frameworks-exist-for-c-c http://stackoverflow.com/questions/4715890/what-mutation-tes...

by nullc 12y ago

eridius 12y ago

I wonder how hard it is to write a program that will parse two C source files into AST's, diff the AST's, and then map the AST back to the original files and highlight where all the differences are?

JoshTriplett 12y ago
Quite difficult if you need to handle arbitrary C, primarily because of the preprocessor; preprocessing is a textual step, and before preprocessing, C code can't necessarily be turned into a sensible AST. You'd likely get better results by running both source files through an indenter and diffing the results. That's a strong argument for enforcing a canonical style via tools like "go fmt" or "indent".

dbaupp 12y ago

That sounds like you're effectively using it as a test coverage tool to find code paths that aren't being tested, would that be an accurate summary?

nullc 12y ago
Not quite, because code can still get run by a test but not be tested. This isn't all that interesting to do except on code that already has ~100% branch coverage. (Since obviously any unexecuted code is going to get removed, and lcov is a lot less work to use :) ) There aren't really any good mutation testing tools for C— e.g. tools to test your tests by breaking your code and making sure your tests fail. C-reduce can be used as a limited kind of mutation test— one that only tests reductions. One nice thing is that for code with 100% coverage any successful reduction is quite often a shortcoming in the tests or a missed optimization in the code, which isn't something you can say for most other kinds of mutations. (except for the fact that c-reduce does a lot of formatting garbling that doesn't really change the code)
- dbaupp 12y ago
  Ah, nice. So it's more like a "semantic coverage" tool, i.e. finding the code paths that don't have any effect on the output.

jevinskie 12y ago

Try running the reduced code through clang-format.

rurounijones 12y ago

In the ruby world there is the "mutant" ( https://github.com/mbj/mutant https://github.com/mbj/mutant ) gem that does stuff like this automatically. It runs your tests and makes sure they all pass then it mutates the code under test (changing a == b to a != b or a <= 10 to a < 10) and makes sure that at least one of your tests fails. If your tests still pass after the mutation then you know you are not covering something. Tis a great idea but making a C version with the equivalent functionality would be a lot harder I think.

michaelmior 12y ago
Curious why you're thinking doing this in C would be a lot harder? All you really need to do is define what some valid transformations are. There are existing mutation testing tools for C[1]. [1] http://stackoverflow.com/questions/4715890/what-mutation-testing-frameworks-exist-for-c-c http://stackoverflow.com/questions/4715890/what-mutation-tes...