4 ms·

This was a rough release cycle for downstream users of LLVM. During testing of the release candidates, we found and reported 7 regressions from LLVM 11. However

This was a rough release cycle for downstream users of LLVM. During testing of the release candidates, we found and reported 7 regressions from LLVM 11. However, despite having reproducible regressions reported as release blockers, the LLVM project tagged release 12.0.0. Not only were there open regressions at this time, but the 12.0.0 tag did not even tag the tip of the release/12.x branch - so there were fixes already committed into the release branch that did not make it into the tag. I don't follow LLVM development. Is this normal for the project?

coldtea 5y ago
>we found and reported 7 regressions from LLVM 11. However, despite having reproducible regressions reported as release blockers Well, this says that zig team reported those bugs as "release blockers". Doesn't mean the LLVM release team also considered them release blockers (or that it had too).
- mkishi 5y ago
  Could you elaborate on when they wouldn't treat regressions as blockers? I'm not very familiar with LLVM and couldn't gather that from the "How To Release LLVM To The Public" docs [1]. [1] https://llvm.org/docs/HowToReleaseLLVM.html https://llvm.org/docs/HowToReleaseLLVM.html
  - coldtea 5y ago
    Well, e.g. they say: "Only critical bugs found during this testing phase will be fixed." - so e.g. not all regressions are necessarily release blockers. Besides, whatever the "process" says, isn't it ultimate up to the release manager?
slimsag 5y ago
Accidents happen, and LLVM is a really large complex project with tons of downstream users (like Zig and Rust) discovering issues not caught by LLVM's test suite / clang. It's definitely a real problem[0][1][2][3], I would argue LLVM needs a longer release-candidate cycle before a release is cut - to give downstream users more to pick up and report regressions, as well as LLVM contributors to fix them. A higher quality bar would be great. [0] https://twitter.com/andy_kelley/status/1365186310088974336 https://twitter.com/andy_kelley/status/1365186310088974336 [1] https://twitter.com/andy_kelley/status/1303046926430908416 https://twitter.com/andy_kelley/status/1303046926430908416 [2] https://www.reddit.com/r/rust/comments/gh3thc/llvm_10_has_performance_regressions_with_rust/ https://www.reddit.com/r/rust/comments/gh3thc/llvm_10_has_pe... [3] https://www.reddit.com/r/rust/comments/n5ne5i/regression_miscompilation_due_to_bug_in_mutable/ https://www.reddit.com/r/rust/comments/n5ne5i/regression_mis...
- tialaramex 5y ago
  It's definitely a problem that LLVM has features which are in practice not really exercised by C++ (because the ISO C++ standard says something is Undefined Behaviour) and so clang doesn't need to use those features yet LLVM developers too often act as though only C++ matters. Rust spent a while with trivial crashes if you have an infinite loop. Here's how that happened as I understand it: From Rust's point of view infinite loops are well formed. Sure, your program never exits but that's what you told it to do, so it's your problem not Rust's. However in C++ that isn't a thing. In C++ an infinite loop is Undefined Behaviour. And so LLVM just didn't have working infinite loop code. If your C++ program has an infinite loop, too bad the ISO Standard says your program is wrong, clang is behaving as intended. Sure that's crazy but C++ programmers have a sort of Stockholm Syndrome. So LLVM had a bug, but because the most important customer of LLVM is clang, and clang didn't care about this hideous bug while its users had just become numbed to the consequences, the bug lived on for ages.
  - duped 5y ago
    If we're throwing stones at LLVM features that don't work let's toss one at noalias
  - pjmlp 5y ago
    Add Swift, C and Objective-C to the "customers" list that really matter.
bla3 5y ago
It's the first release by a new release manager as far as I can tell: https://www.phoronix.com/scan.php?page=news_item&px=Tom-Stellard-LLVM-RM https://www.phoronix.com/scan.php?page=news_item&px=Tom-Stel... So hopefully just temporary transition pain while the new manager learns the ropes.
- pyler 5y ago
  Same failures https://lists.llvm.org/pipermail/llvm-dev/2021-June/150839.html https://lists.llvm.org/pipermail/llvm-dev/2021-June/150839.h... Quality really went down with that RH guy.. :/

coldtea 5y ago

>we found and reported 7 regressions from LLVM 11. However, despite having reproducible regressions reported as release blockers Well, this says that zig team reported those bugs as "release blockers". Doesn't mean the LLVM release team also considered them release blockers (or that it had too).

mkishi 5y ago
Could you elaborate on when they wouldn't treat regressions as blockers? I'm not very familiar with LLVM and couldn't gather that from the "How To Release LLVM To The Public" docs [1]. [1] https://llvm.org/docs/HowToReleaseLLVM.html https://llvm.org/docs/HowToReleaseLLVM.html
- coldtea 5y ago
  Well, e.g. they say: "Only critical bugs found during this testing phase will be fixed." - so e.g. not all regressions are necessarily release blockers. Besides, whatever the "process" says, isn't it ultimate up to the release manager?

slimsag 5y ago

Accidents happen, and LLVM is a really large complex project with tons of downstream users (like Zig and Rust) discovering issues not caught by LLVM's test suite / clang. It's definitely a real problem[0][1][2][3], I would argue LLVM needs a longer release-candidate cycle before a release is cut - to give downstream users more to pick up and report regressions, as well as LLVM contributors to fix them. A higher quality bar would be great. [0] https://twitter.com/andy_kelley/status/1365186310088974336 https://twitter.com/andy_kelley/status/1365186310088974336 [1] https://twitter.com/andy_kelley/status/1303046926430908416 https://twitter.com/andy_kelley/status/1303046926430908416 [2] https://www.reddit.com/r/rust/comments/gh3thc/llvm_10_has_performance_regressions_with_rust/ https://www.reddit.com/r/rust/comments/gh3thc/llvm_10_has_pe... [3] https://www.reddit.com/r/rust/comments/n5ne5i/regression_miscompilation_due_to_bug_in_mutable/ https://www.reddit.com/r/rust/comments/n5ne5i/regression_mis...

tialaramex 5y ago
It's definitely a problem that LLVM has features which are in practice not really exercised by C++ (because the ISO C++ standard says something is Undefined Behaviour) and so clang doesn't need to use those features yet LLVM developers too often act as though only C++ matters. Rust spent a while with trivial crashes if you have an infinite loop. Here's how that happened as I understand it: From Rust's point of view infinite loops are well formed. Sure, your program never exits but that's what you told it to do, so it's your problem not Rust's. However in C++ that isn't a thing. In C++ an infinite loop is Undefined Behaviour. And so LLVM just didn't have working infinite loop code. If your C++ program has an infinite loop, too bad the ISO Standard says your program is wrong, clang is behaving as intended. Sure that's crazy but C++ programmers have a sort of Stockholm Syndrome. So LLVM had a bug, but because the most important customer of LLVM is clang, and clang didn't care about this hideous bug while its users had just become numbed to the consequences, the bug lived on for ages.
- duped 5y ago
  If we're throwing stones at LLVM features that don't work let's toss one at noalias
- pjmlp 5y ago
  Add Swift, C and Objective-C to the "customers" list that really matter.

bla3 5y ago

It's the first release by a new release manager as far as I can tell: https://www.phoronix.com/scan.php?page=news_item&px=Tom-Stellard-LLVM-RM https://www.phoronix.com/scan.php?page=news_item&px=Tom-Stel... So hopefully just temporary transition pain while the new manager learns the ropes.

pyler 5y ago
Same failures https://lists.llvm.org/pipermail/llvm-dev/2021-June/150839.html https://lists.llvm.org/pipermail/llvm-dev/2021-June/150839.h... Quality really went down with that RH guy.. :/