3 ms·

More to the point, anyone doing compiler validation work would a) always have a full test suite as their first line of attack, b) always have a well understood

More to the point, anyone doing compiler validation work would a) always have a full test suite as their first line of attack, b) always have a well understood developer pipeline in place, to ensure a consistent understanding - coding rules, code review processes, etc. and c) always looks at what the compiler is actually doing, compared to what they think its doing - i.e. is able to navigate code -> assembly and all the stages in between, with competence. Anything less is asking for trouble.

joosters 6y ago
A ‘full’ test suite? I’m not sure that one of those has ever existed for any large program, let alone one under active development. We have to work in the real world here!
- fit2rule 6y ago
  Yes, a full test suite. 100% code coverage is a thing, even in compiler validation. Also, every test that was ever written for a compiler is still out there, being run, somewhere. And, for the record, the real world is pretty big.
  - andrewaylett 6y ago
    But 100% code coverage isn't necessarily a useful thing in this context, as different code (and different options) mean different interactions between passes. And you're never going to cover all the different paths through the compiler. Back when I worked on a commercial compiler, we had a smoke test which we ran before committing. It would take ~30 minutes to run. We had a longer, roughly eight hour, set of tests which we ran daily. And we had machines running tests 24/7, which managed to clock up roughly 20 million distinct test cases (combinations of code and options) over the course of a year. Every single compiler bug we'd ever seen was present (in a reduced form) in the conformance tests we'd run before release. That took about a week.
    - fit2rule 6y ago
      >And you're never going to cover all the different paths through the compiler. Depends who you work for and how willing they are to let such justifications stand in a court of law.
  - joosters 6y ago
    100% code coverage does not mean a full test suite. You can still find huge numbers of bugs in code that has 100% coverage in a test suite. The two are not the same!
    - fit2rule 6y ago
      I'm not offering 100% coverage as anything but a level playing field, that things are tested. Also, code-path analysis is pretty much more advanced than you might think, especially when you get auto/generated testing rigs configured for particular tasks - and especially if you have a compiler to test, as well as all the (reproducible) applications it may produce ..

joosters 6y ago

A ‘full’ test suite? I’m not sure that one of those has ever existed for any large program, let alone one under active development. We have to work in the real world here!

fit2rule 6y ago
Yes, a full test suite. 100% code coverage is a thing, even in compiler validation. Also, every test that was ever written for a compiler is still out there, being run, somewhere. And, for the record, the real world is pretty big.
- andrewaylett 6y ago
  But 100% code coverage isn't necessarily a useful thing in this context, as different code (and different options) mean different interactions between passes. And you're never going to cover all the different paths through the compiler. Back when I worked on a commercial compiler, we had a smoke test which we ran before committing. It would take ~30 minutes to run. We had a longer, roughly eight hour, set of tests which we ran daily. And we had machines running tests 24/7, which managed to clock up roughly 20 million distinct test cases (combinations of code and options) over the course of a year. Every single compiler bug we'd ever seen was present (in a reduced form) in the conformance tests we'd run before release. That took about a week.
  - fit2rule 6y ago
    >And you're never going to cover all the different paths through the compiler. Depends who you work for and how willing they are to let such justifications stand in a court of law.
- joosters 6y ago
  100% code coverage does not mean a full test suite. You can still find huge numbers of bugs in code that has 100% coverage in a test suite. The two are not the same!
  - fit2rule 6y ago
    I'm not offering 100% coverage as anything but a level playing field, that things are tested. Also, code-path analysis is pretty much more advanced than you might think, especially when you get auto/generated testing rigs configured for particular tasks - and especially if you have a compiler to test, as well as all the (reproducible) applications it may produce ..