7 ms·

Testing GCC

84 points by dmalcolm 10y ago

saurik 10y ago
> So, in GCC 7, we’ve extended the C frontend so that we can embed fragments of GIMPLE and RTL dumps as the bodies of functions. This additionally sounds extremely interesting for reasons that have nothing to do with testing (though, I bet most of the things I'd want to do this with this will continue to require the expanded platform support for __attribute__((__naked__)) that has blocked a lot of my use cases for inline assembly and which does not seem to be something that is "wanted" by GCC, though I should verify it isn't just due to no one providing a patch... it isn't as if there aren't other people asking for it).
monochromatic 10y ago
Took forever to load, then gave me "Page Does Not Exist." Anybody else having issues?
- pirocks 10y ago
  I too am having issues.
  - twiss 10y ago
    I filed a report through the "report a website issue" button at the blog homepage. Hope they fix it soon.
- smhenderson 10y ago
  I am getting the same. Tried about 15 minutes ago and then again just now. Still showing the Page Does Not Exist error.
- ParadoxOryx 10y ago
  Here's the Google cached version - http://webcache.googleusercontent.com/search?q=cache:https://developers.redhat.com/blog/2017/02/13/testing-testing-gcc/ http://webcache.googleusercontent.com/search?q=cache:https:/...
  - egwynn 10y ago
    For me, that version refreshes indefinitely. I recommend reading the plain-text version: http://webcache.googleusercontent.com/search?q=cache:https://developers.redhat.com/blog/2017/02/13/testing-testing-gcc/&num=1&prmd=ivns&strip=1&vwsrc=0 http://webcache.googleusercontent.com/search?q=cache:https:/...
- dmalcolm 10y ago
  [author of the post here] Thanks; am chasing it up at my end
  - i336_ 10y ago
    Thanks for chasing it up! Would I be right in guessing that we're looking at a well-built security system responding to high page load? Saying the page doesn't exist in these types of situations is interesting. Users consider 404s permanent, so some proportion is unlikely to come back. Furthermore, because each user is hitting the page individually, there's going to be very little out-of-band "no it doesn't work", so it won't reduce server load by much. From a security standpoint, if high load like this is causing "lp0 on fire"-type issues, a generic 404 may be the correct answer, depending on the situation. A 404 is quite generic, but I'm sure it could still leak useful information depending on the scenario. Whatever's going on, hearing what actually happened would be very cool (and educational, most people here are Web developers!).
    - dmalcolm 10y ago
      It seems to be working again. Sorry about the snafu. I've been exchanging emails with the team on the hosting side, but I don't yet have info on what happened.
      - i336_ 10y ago
        Yay! Yep, it's back up. Thanks again! In case it's useful, some of the page resources were 503ing/404ing before: http://i.imgur.com/5JUyphf.png http://i.imgur.com/5JUyphf.png (the intermixed connection_refused errors are from my /etc/hosts blocking)
drfuchs 10y ago
Great, but now how will you find out that a particular optimization pass hasn't been completely obviated by earlier passes, since you won't be looking for C code that actually triggers it?
- correnos 10y ago
  Well, you could embed a machine-checked proof that instruction sequences of the targeted form will sometimes survive previous optimization passes and therefore the given optimization will always be contributing something. But that's hard, so every compiler I've used leaves it to human engineering instead.
  - astrange 10y ago
    You can emit stats for time and # of instructions changed by each pass. gcc certainly does the first, don't remember the second.
- bonzini 10y ago
  You can do both kinds of tests: check that a pass still optimizes some GIMPLE or RTL input the same way, and check that the C code is optimized the same way. In turn, the latter can either look at per-pass dumps, or at the final assembly code for one or more architectures.
- greglindahl 10y ago
  I never got around to actually implementing the following in the PathScale compiler, but my idea was to add logging in all the places where allocations do something, and then create lists of optimizations that I expect to have successfully fired in various functions or programs. This would be especially handy for checking that new optimizations didn't disable important things for various SPECcpu benchmarks.
  - greglindahl 10y ago
    Sorry, "the places where optimizations do something". Didn't notice until after the edit window was long passed. I blame autocorrect :-)
deleted 10y ago
[deleted]
wolf550e 10y ago
So is GCC adding a toolchain similar to the one LLVM has around its bitcode? In this instance, a way to write a text representation of an internal data structure that represents code, run a single optimization pass over that data structure, dump the result, compare the result with serialized text representation of "expected value"? Will they add the other parts, to extract an llvm-like core out of gcc and implement the C and C++ compilers and link time optimizer as users of that core?
- ori_b 10y ago
  > So is GCC adding a toolchain similar to the one LLVM has around its bitcode? Not as far as I can tell. > In this instance, a way to write a text representation of an internal data structure that represents code, run a single optimization pass over that data structure, dump the result, compare the result with serialized text representation of "expected value"? That's already existed for ages. See the -fdump-tree-* family of functions. It also doesn't do anything llvm-like.
  - wolf550e 10y ago
    So all that hasn't existed for ages. Only dumping the intermediate representation has existed for ages, but not loading it and not running the middle-end of the compiler on the loaded IR.
    - dmalcolm 10y ago
      [author of the post here] Correct: loading the IR dumps back in is new.
    - DannyBee 10y ago
      Right, the historical problem with loading GIMPLE was that it still required all sorts of frontend specific bits. Also, tons of things reused tons of bits for different purposes :) Way the heck back when (pre-c++), i started splitting up just the decl hierarchy in trees to try to get things to stop reusing bits, and to expose what would be required to make it independent of the frontendy bits.
namelezz 10y ago
> As a relative newcomer to the project, one of my “pain points” learning GCC’s internals was the custom garbage collector it uses to manage memory It's interesting that GCC has a custom GC.
- wolf550e 10y ago
  https://en.wikipedia.org/wiki/Greenspun's_tenth_rule https://en.wikipedia.org/wiki/Greenspun's_tenth_rule ?
  - dmalcolm 10y ago
    [author of the article] The RTL dump format does look a lot like Lisp, FWIW; see: https://gcc.gnu.org/onlinedocs/gccint/RTL.html https://gcc.gnu.org/onlinedocs/gccint/RTL.html or this article I wrote a while back on it: http://developerblog.redhat.com/2014/09/16/improving-gccs-internals/ http://developerblog.redhat.com/2014/09/16/improving-gccs-in... (This aspect of GCC is probably the oldest, written by RMS back in the late 1980s, I believe).
  - pekk 10y ago
    Middlebrow dismissal? Memory management is a common concern across languages, and languages which do not include garbage collection tend to host a variety of approaches depending on the author and what sort of thing they were building. Writing C code that uses garbage collection does not equal Common Lisp.
    - wolf550e 10y ago
      Not meant as a dismissal at all. I just noted that the phenomenon in which big complex C programs evolve into using features of higher level languages (like garbage collection instead of the regular malloc+free approach) is well known. Games are famous for using frame-bound arenas. GCC wanted to run on anything that had enough RAM, so it's written in C (and now I think C++). I think a language with garbage collection and ML style pattern matching would be nicer for writing a compiler, and I don't think porting the GC runtime to every platform is that much work, but Microsoft didn't rewrite their compilers in F# and Oracle didn't rewrite the Java compilers in Scala, so maybe I'm missing something.
raverbashing 10y ago
Well, besides the big testing of compiling an entire linux distribution, yes, it is good to have specific tests for some particular issues in the compiler (manageable pass/fail tests that can identify a problem quickly)
- greglindahl 10y ago
  gcc has always had a huge unit test suite. Not a new thing.
  - sanxiyn 10y ago
    gcc had a huge integration test suite, which is different from unit test suite.
- hermitdev 10y ago
  Yeah...I got hit by a regression in a bug fix in RH's GCC back in the 4.1.2 era. The revision/build number only moved by a dozen or 2, but they completely broke C++ anonymous namespaces at global scope (caused an ICE). Glad I didn't let Ops upgrade all of my boxes at the same time like they wanted...
boris 10y ago
Interesting to see complex pieces of software including features that are primarily used for testing. I guess this is a natural next step after having testing micro-formats[1]. One thing I don't like about GCC's testing framework is the number of files. If you need to test 10 related things (say, diagnostics for a particular language construct), then you will have 10 different test files. I would much prefer for them to be in a single file. This is one of the goals of Testscript[2]. [1] https://blog.nelhage.com/2016/12/how-i-test/ https://blog.nelhage.com/2016/12/how-i-test/ [2] https://build2.org/build2/doc/build2-testscript-manual.xhtml https://build2.org/build2/doc/build2-testscript-manual.xhtml
- dmalcolm 10y ago
  [author of the post] Thanks for the links; a lot of interesting material in there. FWIW, DejaGnu gives some flexibility over how granular the test files can be. For example, here's part of the test case for the -Wmisleading-indentation warning I added in gcc 6: https://github.com/gcc-mirror/gcc/blob/master/gcc/testsuite/c-c%2B%2B-common/Wmisleading-indentation.c https://github.com/gcc-mirror/gcc/blob/master/gcc/testsuite/... i.e. that's one 700 line test source file, expressing most of the test coverage for one feature. I realize now that the post should have had an example of what such a test case has; see the "dg-warning" directives in that file. So that's one "single file" test case. That said, I had to put some of the other test cases for that feature into different test files: some of them interact with the preprocessor in "interesting" ways, so it was simplest to split them out. However, when a test starts failing, it's handy if the failing test is small (especially when stepping through the latter in the debugger...). So in our testing we have a mix of both big test files covering a lot of material, and small test files, covering very specific corner cases. Another aspect of granularity is how long the testsuite takes to run: each test source file in the DejaGnu suite involves exec-ing the compiler, which adds a fixed amount of overhead. It parallelizes well, but it's still advantageous to stuff more testing into fewer files (with the obvious tradeoff w.r.t. debugability if something starts failing). This was another reason for adding the unit-testing suite: this part is all done in-process, and so these ~30,000 tests run in less than a second.
pizlonator 10y ago
I'm super impressed by the size of their test suite. I wish I had a test suite like that!

84 points by dmalcolm 10y ago

> So, in GCC 7, we’ve extended the C frontend so that we can embed fragments of GIMPLE and RTL dumps as the bodies of functions. This additionally sounds extremely interesting for reasons that have nothing to do with testing (though, I bet most of the things I'd want to do this with this will continue to require the expanded platform support for __attribute__((__naked__)) that has blocked a lot of my use cases for inline assembly and which does not seem to be something that is "wanted" by GCC, though I should verify it isn't just due to no one providing a patch... it isn't as if there aren't other people asking for it).

monochromatic 10y ago

Took forever to load, then gave me "Page Does Not Exist." Anybody else having issues?

pirocks 10y ago
I too am having issues.
- twiss 10y ago
  I filed a report through the "report a website issue" button at the blog homepage. Hope they fix it soon.
smhenderson 10y ago
I am getting the same. Tried about 15 minutes ago and then again just now. Still showing the Page Does Not Exist error.
ParadoxOryx 10y ago
Here's the Google cached version - http://webcache.googleusercontent.com/search?q=cache:https://developers.redhat.com/blog/2017/02/13/testing-testing-gcc/ http://webcache.googleusercontent.com/search?q=cache:https:/...
- egwynn 10y ago
  For me, that version refreshes indefinitely. I recommend reading the plain-text version: http://webcache.googleusercontent.com/search?q=cache:https://developers.redhat.com/blog/2017/02/13/testing-testing-gcc/&num=1&prmd=ivns&strip=1&vwsrc=0 http://webcache.googleusercontent.com/search?q=cache:https:/...
dmalcolm 10y ago
[author of the post here] Thanks; am chasing it up at my end
- i336_ 10y ago
  Thanks for chasing it up! Would I be right in guessing that we're looking at a well-built security system responding to high page load? Saying the page doesn't exist in these types of situations is interesting. Users consider 404s permanent, so some proportion is unlikely to come back. Furthermore, because each user is hitting the page individually, there's going to be very little out-of-band "no it doesn't work", so it won't reduce server load by much. From a security standpoint, if high load like this is causing "lp0 on fire"-type issues, a generic 404 may be the correct answer, depending on the situation. A 404 is quite generic, but I'm sure it could still leak useful information depending on the scenario. Whatever's going on, hearing what actually happened would be very cool (and educational, most people here are Web developers!).
  - dmalcolm 10y ago
    It seems to be working again. Sorry about the snafu. I've been exchanging emails with the team on the hosting side, but I don't yet have info on what happened.
    - i336_ 10y ago
      Yay! Yep, it's back up. Thanks again! In case it's useful, some of the page resources were 503ing/404ing before: http://i.imgur.com/5JUyphf.png http://i.imgur.com/5JUyphf.png (the intermixed connection_refused errors are from my /etc/hosts blocking)

drfuchs 10y ago

Great, but now how will you find out that a particular optimization pass hasn't been completely obviated by earlier passes, since you won't be looking for C code that actually triggers it?

correnos 10y ago
Well, you could embed a machine-checked proof that instruction sequences of the targeted form will sometimes survive previous optimization passes and therefore the given optimization will always be contributing something. But that's hard, so every compiler I've used leaves it to human engineering instead.
- astrange 10y ago
  You can emit stats for time and # of instructions changed by each pass. gcc certainly does the first, don't remember the second.
bonzini 10y ago
You can do both kinds of tests: check that a pass still optimizes some GIMPLE or RTL input the same way, and check that the C code is optimized the same way. In turn, the latter can either look at per-pass dumps, or at the final assembly code for one or more architectures.
greglindahl 10y ago
I never got around to actually implementing the following in the PathScale compiler, but my idea was to add logging in all the places where allocations do something, and then create lists of optimizations that I expect to have successfully fired in various functions or programs. This would be especially handy for checking that new optimizations didn't disable important things for various SPECcpu benchmarks.
- greglindahl 10y ago
  Sorry, "the places where optimizations do something". Didn't notice until after the edit window was long passed. I blame autocorrect :-)

deleted 10y ago

[deleted]

wolf550e 10y ago

So is GCC adding a toolchain similar to the one LLVM has around its bitcode? In this instance, a way to write a text representation of an internal data structure that represents code, run a single optimization pass over that data structure, dump the result, compare the result with serialized text representation of "expected value"? Will they add the other parts, to extract an llvm-like core out of gcc and implement the C and C++ compilers and link time optimizer as users of that core?

ori_b 10y ago
> So is GCC adding a toolchain similar to the one LLVM has around its bitcode? Not as far as I can tell. > In this instance, a way to write a text representation of an internal data structure that represents code, run a single optimization pass over that data structure, dump the result, compare the result with serialized text representation of "expected value"? That's already existed for ages. See the -fdump-tree-* family of functions. It also doesn't do anything llvm-like.
- wolf550e 10y ago
  So all that hasn't existed for ages. Only dumping the intermediate representation has existed for ages, but not loading it and not running the middle-end of the compiler on the loaded IR.
  - dmalcolm 10y ago
    [author of the post here] Correct: loading the IR dumps back in is new.
  - DannyBee 10y ago
    Right, the historical problem with loading GIMPLE was that it still required all sorts of frontend specific bits. Also, tons of things reused tons of bits for different purposes :) Way the heck back when (pre-c++), i started splitting up just the decl hierarchy in trees to try to get things to stop reusing bits, and to expose what would be required to make it independent of the frontendy bits.

namelezz 10y ago

> As a relative newcomer to the project, one of my “pain points” learning GCC’s internals was the custom garbage collector it uses to manage memory It's interesting that GCC has a custom GC.

wolf550e 10y ago
https://en.wikipedia.org/wiki/Greenspun's_tenth_rule https://en.wikipedia.org/wiki/Greenspun's_tenth_rule ?
- dmalcolm 10y ago
  [author of the article] The RTL dump format does look a lot like Lisp, FWIW; see: https://gcc.gnu.org/onlinedocs/gccint/RTL.html https://gcc.gnu.org/onlinedocs/gccint/RTL.html or this article I wrote a while back on it: http://developerblog.redhat.com/2014/09/16/improving-gccs-internals/ http://developerblog.redhat.com/2014/09/16/improving-gccs-in... (This aspect of GCC is probably the oldest, written by RMS back in the late 1980s, I believe).
- pekk 10y ago
  Middlebrow dismissal? Memory management is a common concern across languages, and languages which do not include garbage collection tend to host a variety of approaches depending on the author and what sort of thing they were building. Writing C code that uses garbage collection does not equal Common Lisp.
  - wolf550e 10y ago
    Not meant as a dismissal at all. I just noted that the phenomenon in which big complex C programs evolve into using features of higher level languages (like garbage collection instead of the regular malloc+free approach) is well known. Games are famous for using frame-bound arenas. GCC wanted to run on anything that had enough RAM, so it's written in C (and now I think C++). I think a language with garbage collection and ML style pattern matching would be nicer for writing a compiler, and I don't think porting the GC runtime to every platform is that much work, but Microsoft didn't rewrite their compilers in F# and Oracle didn't rewrite the Java compilers in Scala, so maybe I'm missing something.

raverbashing 10y ago

Well, besides the big testing of compiling an entire linux distribution, yes, it is good to have specific tests for some particular issues in the compiler (manageable pass/fail tests that can identify a problem quickly)

greglindahl 10y ago
gcc has always had a huge unit test suite. Not a new thing.
- sanxiyn 10y ago
  gcc had a huge integration test suite, which is different from unit test suite.
hermitdev 10y ago
Yeah...I got hit by a regression in a bug fix in RH's GCC back in the 4.1.2 era. The revision/build number only moved by a dozen or 2, but they completely broke C++ anonymous namespaces at global scope (caused an ICE). Glad I didn't let Ops upgrade all of my boxes at the same time like they wanted...

boris 10y ago

Interesting to see complex pieces of software including features that are primarily used for testing. I guess this is a natural next step after having testing micro-formats[1]. One thing I don't like about GCC's testing framework is the number of files. If you need to test 10 related things (say, diagnostics for a particular language construct), then you will have 10 different test files. I would much prefer for them to be in a single file. This is one of the goals of Testscript[2]. [1] https://blog.nelhage.com/2016/12/how-i-test/ https://blog.nelhage.com/2016/12/how-i-test/ [2] https://build2.org/build2/doc/build2-testscript-manual.xhtml https://build2.org/build2/doc/build2-testscript-manual.xhtml

dmalcolm 10y ago
[author of the post] Thanks for the links; a lot of interesting material in there. FWIW, DejaGnu gives some flexibility over how granular the test files can be. For example, here's part of the test case for the -Wmisleading-indentation warning I added in gcc 6: https://github.com/gcc-mirror/gcc/blob/master/gcc/testsuite/c-c%2B%2B-common/Wmisleading-indentation.c https://github.com/gcc-mirror/gcc/blob/master/gcc/testsuite/... i.e. that's one 700 line test source file, expressing most of the test coverage for one feature. I realize now that the post should have had an example of what such a test case has; see the "dg-warning" directives in that file. So that's one "single file" test case. That said, I had to put some of the other test cases for that feature into different test files: some of them interact with the preprocessor in "interesting" ways, so it was simplest to split them out. However, when a test starts failing, it's handy if the failing test is small (especially when stepping through the latter in the debugger...). So in our testing we have a mix of both big test files covering a lot of material, and small test files, covering very specific corner cases. Another aspect of granularity is how long the testsuite takes to run: each test source file in the DejaGnu suite involves exec-ing the compiler, which adds a fixed amount of overhead. It parallelizes well, but it's still advantageous to stuff more testing into fewer files (with the obvious tradeoff w.r.t. debugability if something starts failing). This was another reason for adding the unit-testing suite: this part is all done in-process, and so these ~30,000 tests run in less than a second.

pizlonator 10y ago

I'm super impressed by the size of their test suite. I wish I had a test suite like that!