4 ms·

A regression when the LLVM backend updated from 8 to 9. So arguably a regression in LLVM! Reminds me of a Zig stream I watched recently where Andy was updating

A regression when the LLVM backend updated from 8 to 9. So arguably a regression in LLVM! Reminds me of a Zig stream I watched recently where Andy was updating the compiler backend to use an LLVM 12 release candidate. He ended up filling a couple bug reports with LLVM, using Zig compiler issues to generate reduced case LLVM IR examples that compiled differently with LLVM 11. He mentioned that he had to try this upgrade with the release candidate because right now they're LLVM regressions, but once 12 ships they become Zig regressions!

jedisct1 6y ago
Zig does a really good job at finding bugs in LLVM every time the LLVM version is updated. And Zig developers also do a very good job at reporting and fixing them. Which is why contributing to Zig also means contributing to LLVM.
- rob74 6y ago
  Language developers finding bugs in LLVM is ok, but if there are already several instances of regressions in LLVM found by developers of (maybe "exotic") languages, then the reaction should be less "hey, well done, you found a bug" and more "how can we avoid such regressions in the future?"...
  - MaxBarraclough 6y ago
    I imagine Clang and GCC have growing test-suites to defend against regressions. I wonder if they share them.
    - wyldfire 6y ago
      They both do, yes. LLVM's [1] includes (some of?) gcc's, the reverse may also be true. LLVM also has extensive unit tests that can exercise codegen and backend functionality. [1] https://github.com/llvm/llvm-test-suite https://github.com/llvm/llvm-test-suite
    - sesuximo 6y ago
      Clang writes a subset of LLVM IR, and other types of IR are supported but very poorly optimized/tested/etc. The LLVM docs say you should generate what Clang generates to avoid regressions.
  - PoignardAzur 6y ago
    Separate the backend into functional subcomponents and fuzz them as much as possible? It's what cranelift does.
    - wyldfire 6y ago
      Fuzzing is great for finding crashes and other catastrophic misbehavior. But suboptimal codegen like this would be difficult to reveal with fuzzing. You could do it with a reference compiler (this has in fact been done before) but finding suboptimal codegen like this case would still be kinda tricky.
      - tom_mellior 6y ago
        Based on other comments this seems to be a regression in LLVM. LLVM does have a reference compiler for regressions: It's the previous version of LLVM. (Not saying that that makes this trivial.)
        wyldfire 6y ago
        But if this next release makes codegen improvements, those would all show up as differences. Separating the improvements from the regressions is difficult enough -- fuzzing doesn't really help here, it makes it much harder to determine what code should have been generated. Generally, codegen issues are exposed by benchmarks or someone who is curious enough to examine and analyze the code generated from multiple compilers or multiple releases. The latter is much rarer. But as these happen, the bar keeps getting raised and we do grow the test suite.
Google234 6y ago
For those reading now it’s not LLVM’s fault. It was a rust change that disabled it

jedisct1 6y ago

Zig does a really good job at finding bugs in LLVM every time the LLVM version is updated. And Zig developers also do a very good job at reporting and fixing them. Which is why contributing to Zig also means contributing to LLVM.

rob74 6y ago
Language developers finding bugs in LLVM is ok, but if there are already several instances of regressions in LLVM found by developers of (maybe "exotic") languages, then the reaction should be less "hey, well done, you found a bug" and more "how can we avoid such regressions in the future?"...
- MaxBarraclough 6y ago
  I imagine Clang and GCC have growing test-suites to defend against regressions. I wonder if they share them.
  - wyldfire 6y ago
    They both do, yes. LLVM's [1] includes (some of?) gcc's, the reverse may also be true. LLVM also has extensive unit tests that can exercise codegen and backend functionality. [1] https://github.com/llvm/llvm-test-suite https://github.com/llvm/llvm-test-suite
  - sesuximo 6y ago
    Clang writes a subset of LLVM IR, and other types of IR are supported but very poorly optimized/tested/etc. The LLVM docs say you should generate what Clang generates to avoid regressions.
- PoignardAzur 6y ago
  Separate the backend into functional subcomponents and fuzz them as much as possible? It's what cranelift does.
  - wyldfire 6y ago
    Fuzzing is great for finding crashes and other catastrophic misbehavior. But suboptimal codegen like this would be difficult to reveal with fuzzing. You could do it with a reference compiler (this has in fact been done before) but finding suboptimal codegen like this case would still be kinda tricky.
    - tom_mellior 6y ago
      Based on other comments this seems to be a regression in LLVM. LLVM does have a reference compiler for regressions: It's the previous version of LLVM. (Not saying that that makes this trivial.)
      - wyldfire 6y ago
        But if this next release makes codegen improvements, those would all show up as differences. Separating the improvements from the regressions is difficult enough -- fuzzing doesn't really help here, it makes it much harder to determine what code should have been generated. Generally, codegen issues are exposed by benchmarks or someone who is curious enough to examine and analyze the code generated from multiple compilers or multiple releases. The latter is much rarer. But as these happen, the bar keeps getting raised and we do grow the test suite.

Google234 6y ago

For those reading now it’s not LLVM’s fault. It was a rust change that disabled it