4 ms·

It's an interesting method. Does anyone know if these kinds of torture tests get collected into a common library to help any future compilers or if they just re

by pedrocr 12y ago

It's an interesting method. Does anyone know if these kinds of torture tests get collected into a common library to help any future compilers or if they just result in a paper and bug reports to current ones?

TheLoneWolfling 12y ago
Good question. Another idea: taking bug reports for one compiler for a language and using that as fuzzing input for another compiler for the same language.
- pascal_cuoq 12y ago
  This gives very poor coverage. The testcases in one compiler's regressions suites are usually reduced to the shortest program that causes a previous bug, and another compiler that does not share any source code with the first one will have no particular reason to contain bugs that are revealed by the same programs. Sure, you may find some this way, but it is not likely to be the best use of your time. Invent something like Csmith/EMI instead, that will be more efficient! The C language needs a good test suite with known expected results and good coverage of the language's definition, but one way to build this suite is not to catenate all the regression tests of existing compilers. Actually, part of the reasons why we do not have such a reference testsuite is that it is not as easy as putting together regression tests from various origins.
  - regehr 12y ago
    I'm still interested in somehow creating such a test suite, Pascal -- one that is designed to test corner cases in the standard rather than testing corner cases in the optimizer. It would somehow be derived from various formalizations of the standard such as yours, Xavier's, Chucky's.
    - pascal_cuoq 12y ago
      1- secure all the SSL. 2- ??? 3- get around to making a conformity assessment test suite for C.
      - regehr 12y ago
        4. PROFIT
pascal_cuoq 12y ago
The generated tests themselves are hardly worth the bytes they occupy. The value is in the compact program that can generate as many of them as desired very quickly, with varying options to focus on varying aspects of compilation (with or without bit-fields, with or without deep pointer nesting, …) We forced Xuejun Yang (who turned Randprog, the prototype that came before Csmith into Csmith) to fix more bugs than were necessary for his PhD or than you can expect a research prototype to have bugs fixed (I am one of the developers of Frama-C, pitting Frama-C “against” Csmith was my hobby for a summer, and we found and reported as many bugs in Csmith as we found bugs in Frama-C). The sentence “over the last couple of years I’ve slacked off on reporting compiler bugs” near the post's conclusion is telling. You can expect the same story to unroll for EMI. Researchers are not rewarded for maintaining software ad vitam æternam, even if the software is useful, but hopefully they have or will soon release the generator as open-source, and then if you find it too useful not to use further, you can fix or work around bugs as you find them. As an example, I think that there is still one bug in Csmith that we work around by ignoring programs that have the symptoms that usually indicate that bug (we still use Csmith to test Frama-C after we have finished a major feature that could introduce the sort of bug it can detect). See also http://blog.regehr.org/archives/1058 http://blog.regehr.org/archives/1058 on the same blog for additional thoughts on the fate of academic software.
- regehr 12y ago
  Pascal, have you reported the bug? Xuejun is doing some Csmith work right now and we can probably get him to fix it.
  - pascal_cuoq 12y ago
    It's one from the last batch, that Xuejun hoped to have fixed but still comes up occasionaly. If there is a window of opportunity, I will try to get more information and re-report it.
    - regehr 12y ago
      Please do, if it is not too much trouble.

by pedrocr 12y ago

TheLoneWolfling 12y ago

Good question. Another idea: taking bug reports for one compiler for a language and using that as fuzzing input for another compiler for the same language.

pascal_cuoq 12y ago
This gives very poor coverage. The testcases in one compiler's regressions suites are usually reduced to the shortest program that causes a previous bug, and another compiler that does not share any source code with the first one will have no particular reason to contain bugs that are revealed by the same programs. Sure, you may find some this way, but it is not likely to be the best use of your time. Invent something like Csmith/EMI instead, that will be more efficient! The C language needs a good test suite with known expected results and good coverage of the language's definition, but one way to build this suite is not to catenate all the regression tests of existing compilers. Actually, part of the reasons why we do not have such a reference testsuite is that it is not as easy as putting together regression tests from various origins.
- regehr 12y ago
  I'm still interested in somehow creating such a test suite, Pascal -- one that is designed to test corner cases in the standard rather than testing corner cases in the optimizer. It would somehow be derived from various formalizations of the standard such as yours, Xavier's, Chucky's.
  - pascal_cuoq 12y ago
    1- secure all the SSL. 2- ??? 3- get around to making a conformity assessment test suite for C.
    - regehr 12y ago
      4. PROFIT

pascal_cuoq 12y ago

The generated tests themselves are hardly worth the bytes they occupy. The value is in the compact program that can generate as many of them as desired very quickly, with varying options to focus on varying aspects of compilation (with or without bit-fields, with or without deep pointer nesting, …) We forced Xuejun Yang (who turned Randprog, the prototype that came before Csmith into Csmith) to fix more bugs than were necessary for his PhD or than you can expect a research prototype to have bugs fixed (I am one of the developers of Frama-C, pitting Frama-C “against” Csmith was my hobby for a summer, and we found and reported as many bugs in Csmith as we found bugs in Frama-C). The sentence “over the last couple of years I’ve slacked off on reporting compiler bugs” near the post's conclusion is telling. You can expect the same story to unroll for EMI. Researchers are not rewarded for maintaining software ad vitam æternam, even if the software is useful, but hopefully they have or will soon release the generator as open-source, and then if you find it too useful not to use further, you can fix or work around bugs as you find them. As an example, I think that there is still one bug in Csmith that we work around by ignoring programs that have the symptoms that usually indicate that bug (we still use Csmith to test Frama-C after we have finished a major feature that could introduce the sort of bug it can detect). See also http://blog.regehr.org/archives/1058 http://blog.regehr.org/archives/1058 on the same blog for additional thoughts on the fate of academic software.

regehr 12y ago
Pascal, have you reported the bug? Xuejun is doing some Csmith work right now and we can probably get him to fix it.
- pascal_cuoq 12y ago
  It's one from the last batch, that Xuejun hoped to have fixed but still comes up occasionaly. If there is a window of opportunity, I will try to get more information and re-report it.
  - regehr 12y ago
    Please do, if it is not too much trouble.