4 ms·

Would your aarch64 test suite have caught it? Testing is fine and all, but it can only find bugs the test authors predicted. There will always be some issues th

Would your aarch64 test suite have caught it? Testing is fine and all, but it can only find bugs the test authors predicted. There will always be some issues that test suites miss. It's obvious to add regression tests for this and related issues now that they've been found. And great! That's what regression tests are for. But testing is always going to be fundamentally incomplete in that it can only cover non-obvious things after they've already been encountered.

mort96 4y ago
I would've expected a compiler to have a huge integration/regression test suite which just consists of other pieces of software's test suites. Grab the N most popular Haskell programs with test suites, compile them, run their test suites, verify that they still pass on the new version.
- sandermvanvliet 4y ago
  Isn’t rust doing something like that? Can’t seem to find the article right now but I remember reading something about this approach.
  - steveklabnik 4y ago
    Yes, “crater” is the name of the tool.
  - estebank 4y ago
    > Crater is a tool for compiling and running tests for every crate on crates.io (and a few on GitHub). It is mainly used for checking the extent of breakage when implementing potentially breaking changes and ensuring lack of breakage by running beta vs stable compiler versions. https://rustc-dev-guide.rust-lang.org/tests/crater.html https://rustc-dev-guide.rust-lang.org/tests/crater.html https://github.com/rust-lang/crater https://github.com/rust-lang/crater https://crater.rust-lang.org/ https://crater.rust-lang.org/ Even then such a bug as GHC's can conceivably not be detected.
    - moomin 4y ago
      It would be much harder to achieve this with GHC since the package manager is far from deterministic.
      - bgamari 4y ago
        I'm not sure what you mean by this. `cabal-install` is quite deterministic; you can even freeze the index date for full build plan determinism. The largest challenge in this area is the fact that core libraries' interface sometimes change. Our [head.hackage](https://gitlab.haskell.org/ghc/head.hackage/ https://gitlab.haskell.org/ghc/head.hackage/) infrastructure is one way of addressing this. I agree that it would be great to leverage `head.hackage` to test GHC against a larger body of packages.
        moomin 4y ago
        It definitely _can_ be deterministic, but my experience of pulling down packages without a frozen index is that it rarely compiles. This, inevitably is because the plan depends on the current state of hackage. A combination of “it might pick a different minor version”, “different versions can cause cascading effects”, “many libraries have breaking changes” and “Haskell is extremely unforgiving of potential breaks” makes it very difficult to use without either a) being really good or b) downloading a freeze index. (I download freeze indexes.) The whole core libraries thing is difficult, in that I can see it’s a real problem for everyone constantly updating their packages (and consuming them), but also that most of the changes are actually good ideas.
- whateveracct 4y ago
  nixpkgs
remram 4y ago
The issue is that basic math doesn't work. I don't dispute that no test suite can catch everything, but yes I do expect that any reasonable test suite that compiles and runs code would have caught this.
- Dylan16807 4y ago
  Basic math that does multiple operations on the same 8 or 16 bit value.
  - gus_massa 4y ago
    I agree. I'm still confused, but I hope this is correct. IIUC the problem here is that they do something like (a // b ) * b where // is the integer division. They are not breaking all math operations, just math operations that have a very specific pattern. Also, I'm not sure if it's important that 128 <= b <= 255, because the main problem seams to be they are extending a number and the leading 1 in the binary representation is a problem (???). So you must test a very specific math pattern, with very specific numbers, and then hope that other parts of the optimizer don't optimize the problem away before the bug in the compiler makes a mistake. I don't know too much about the automatic test suit in Haskell, but I know more about the automatic test suit in Racket. In Racket there are a lot of test written manually, and some randomized test, and also tests for many of the packages. But I really doubt there is a test somewhere for this specific case. [Is there a cross language regression test collection? Something that collects weird cases, so other languages maintainers can take a look, translate the code and add it to the tests.]
    - bgamari 4y ago
      That is correct. The problem is only triggered by very particular patterns of integer operations which happen to get lowered to C-- where an operand is bound to a local register; this register then gets inappropriately mutated by the code generated by the backend. As you say, GHC, like most language implementations, has an extensive [testsuite](https://gitlab.haskell.org/ghc/ghc/-/tree/master/testsuite https://gitlab.haskell.org/ghc/ghc/-/tree/master/testsuite) (10k tests which result in over 30k testcases). Sadly, none of these tests covered this particular case. Moreover, we also have a systematic [property testing framework](https://gitlab.haskell.org/ghc/test-primops https://gitlab.haskell.org/ghc/test-primops) testing the backends and primitive operations provided by the compiler. This testsuite also didn't test this form of program since it focuses on testing the expression fragment of the Cmm language (and therefore won't generate local register bindings). This is a blind spot that I am going to look into addressing.

mort96 4y ago

I would've expected a compiler to have a huge integration/regression test suite which just consists of other pieces of software's test suites. Grab the N most popular Haskell programs with test suites, compile them, run their test suites, verify that they still pass on the new version.

sandermvanvliet 4y ago
Isn’t rust doing something like that? Can’t seem to find the article right now but I remember reading something about this approach.
- steveklabnik 4y ago
  Yes, “crater” is the name of the tool.
- estebank 4y ago
  > Crater is a tool for compiling and running tests for every crate on crates.io (and a few on GitHub). It is mainly used for checking the extent of breakage when implementing potentially breaking changes and ensuring lack of breakage by running beta vs stable compiler versions. https://rustc-dev-guide.rust-lang.org/tests/crater.html https://rustc-dev-guide.rust-lang.org/tests/crater.html https://github.com/rust-lang/crater https://github.com/rust-lang/crater https://crater.rust-lang.org/ https://crater.rust-lang.org/ Even then such a bug as GHC's can conceivably not be detected.
  - moomin 4y ago
    It would be much harder to achieve this with GHC since the package manager is far from deterministic.
    - bgamari 4y ago
      I'm not sure what you mean by this. `cabal-install` is quite deterministic; you can even freeze the index date for full build plan determinism. The largest challenge in this area is the fact that core libraries' interface sometimes change. Our [head.hackage](https://gitlab.haskell.org/ghc/head.hackage/ https://gitlab.haskell.org/ghc/head.hackage/) infrastructure is one way of addressing this. I agree that it would be great to leverage `head.hackage` to test GHC against a larger body of packages.
      - moomin 4y ago
        It definitely _can_ be deterministic, but my experience of pulling down packages without a frozen index is that it rarely compiles. This, inevitably is because the plan depends on the current state of hackage. A combination of “it might pick a different minor version”, “different versions can cause cascading effects”, “many libraries have breaking changes” and “Haskell is extremely unforgiving of potential breaks” makes it very difficult to use without either a) being really good or b) downloading a freeze index. (I download freeze indexes.) The whole core libraries thing is difficult, in that I can see it’s a real problem for everyone constantly updating their packages (and consuming them), but also that most of the changes are actually good ideas.
whateveracct 4y ago
nixpkgs

remram 4y ago

The issue is that basic math doesn't work. I don't dispute that no test suite can catch everything, but yes I do expect that any reasonable test suite that compiles and runs code would have caught this.

Dylan16807 4y ago
Basic math that does multiple operations on the same 8 or 16 bit value.
- gus_massa 4y ago
  I agree. I'm still confused, but I hope this is correct. IIUC the problem here is that they do something like (a // b ) * b where // is the integer division. They are not breaking all math operations, just math operations that have a very specific pattern. Also, I'm not sure if it's important that 128 <= b <= 255, because the main problem seams to be they are extending a number and the leading 1 in the binary representation is a problem (???). So you must test a very specific math pattern, with very specific numbers, and then hope that other parts of the optimizer don't optimize the problem away before the bug in the compiler makes a mistake. I don't know too much about the automatic test suit in Haskell, but I know more about the automatic test suit in Racket. In Racket there are a lot of test written manually, and some randomized test, and also tests for many of the packages. But I really doubt there is a test somewhere for this specific case. [Is there a cross language regression test collection? Something that collects weird cases, so other languages maintainers can take a look, translate the code and add it to the tests.]
  - bgamari 4y ago
    That is correct. The problem is only triggered by very particular patterns of integer operations which happen to get lowered to C-- where an operand is bound to a local register; this register then gets inappropriately mutated by the code generated by the backend. As you say, GHC, like most language implementations, has an extensive [testsuite](https://gitlab.haskell.org/ghc/ghc/-/tree/master/testsuite https://gitlab.haskell.org/ghc/ghc/-/tree/master/testsuite) (10k tests which result in over 30k testcases). Sadly, none of these tests covered this particular case. Moreover, we also have a systematic [property testing framework](https://gitlab.haskell.org/ghc/test-primops https://gitlab.haskell.org/ghc/test-primops) testing the backends and primitive operations provided by the compiler. This testsuite also didn't test this form of program since it focuses on testing the expression fragment of the Cmm language (and therefore won't generate local register bindings). This is a blind spot that I am going to look into addressing.