6 ms·

Testing LLVM

87 points by awalGarg 10y ago

KenoFischer 10y ago
I always find committing to LLVM very nerve wrecking, because of the post-commit CI testing. LLVM has so many architectures that more often than not something I write will fail on one of them. And the only way for me to find out is to commit it, wait for the buildbot to fail (which can take a few hours, during which I really can't leave my computer lest I leave trunk broken on some buildbot, which is a big faux pas), revert it and then figure out what went wrong. I'm hoping that at some point this will be improved, such that I can run the whole buildbot army on my commit before putting it on trunk.
- wyldfire 10y ago
  I get the feeling that there's parts of the community that feel the same way. I'm hoping that the planned move to github will naturally cascade into pre-commit checks.
  - ezanmoto 10y ago
    Do you know of solutions that exist for pre-commit checks for GitHub at the moment?
    - wyldfire 10y ago
      In general? Yes, Travis CI seems to be a very popular one. Many folks use Appveyor for Windows support AFAICT.
    - dikaiosune 10y ago
      Rust uses https://github.com/rust-community/bors https://github.com/rust-community/bors which maintains a linear queue of PRs which can only land after being rebased onto the commits before the PR and subsequently passing tests.
    - Joky 10y ago
      Swift also has a lot of pre-commit CI: - here is the Jenkins bots page: https://ci.swift.org/view/Pull%20Request/ https://ci.swift.org/view/Pull%20Request/ - example of PR test reporting: https://github.com/apple/swift/pull/6802 https://github.com/apple/swift/pull/6802
    - WalterBright 10y ago
      dlang uses several. One notable one is the "autotester" written by Brad Roberts. It's built to ensure that breaking the D compiler (as tested by the test suite) does not result in commits. It's incredibly useful.
  - Joky 10y ago
    A big big problem is having enough hardware to have a CI throughput one or two order of magnitude what it is now. Unfortunately that's not an easy thing considering how "heavy" it is to build and test the full toolchain.
- shanemhansen 10y ago
  That's odd. It would be nice if branches could be tested with the build bot.
  - gus_massa 10y ago
    I agree. When I submit a commit for Racket, the branch is tested in Travis and AppVeyor. That catches a lot of error. Anyway, Racket has an additional internal CI called DrDr and some very subtle errors are only detected there, after the commit is merged.
- Locke1689 10y ago
  Most of the .NET repos are structured to use inner and outer loop testing with Jenkins. Most tests on most architectures are run in the inner loop, which are kicked off in parallel as soon as you make a pull request to one of the .NET repositories on Github. Some repositories, like CoreCLR, have outer loop testing that runs on a separate schedule (nightly, I think), but those tests are far less likely to break and are more devoted to finding rare and difficult to compute edge cases.
SloopJon 10y ago
Interesting to see screenshots of LCOV. I'm hoping to get an intern to work on test coverage this summer, and I wondered whether LCOV is still current. Looks like the latest release is from December 2016.
- Joky 10y ago
  The screenshots are from the "old" style coverage. We have much better views now: http://lab.llvm.org:8080/coverage/coverage-reports/opt/coverage/Users/buildslave/jenkins/sharedspace/clang-stage2-coverage-R@2/llvm/lib/Transforms/Scalar/CorrelatedValuePropagation.cpp.html#L354 http://lab.llvm.org:8080/coverage/coverage-reports/opt/cover... See for instance how having your cursor on top of each side of a condition tells you how many times each individual condition were evaluated!
mp3geek 10y ago
I do find it strange that such a large project isn't using a better VCS. SVN seems to be very antiquated.
- wyldfire 10y ago
  It's its size that makes it difficult to move. Some major ecosystem stuff is designed around the svn infrastructure. When the will arrived to make a change, it seemed natural to migrate not just to a different VCS but a different host. And this seemed to spawn a new debate: monorepo vs multi-repo. [still open AFAIK] At the recent 2016 US Dev Conf, there was a consensus to move to git and that the new host would be github. Really subjective IMO part: In general, there's tons of really smart folks working on really awesome stuff in LLVM+clang+etc. There's a handful of folks also focusing on the general "plumbing" software within and among those projects. The meta-plumbing job of the dev infrastructure is "kinda interesting" to several folks who want to improve the way the project is developed. But "kinda interesting" doesn't pay the bills and so it's a second (or nth responsibility) for the folks volunteering to work on it. Add to that the "no good deed goes unpunished" rule that they'll get the responsibility/blame after making a sweeping change, it means it will require extreme patience and caution.
  - Joky 10y ago
    Yes! Such work is all done on a volunteering basis, when we find time for it on top of the "real work" (bug fixes, features, ...). All the infrastructure work is not always rewarding, you just get the upset people yelling at you :) Current status: http://lists.llvm.org/pipermail/llvm-dev/2017-January/109015.html http://lists.llvm.org/pipermail/llvm-dev/2017-January/109015...
- Groxx 10y ago
  Until semi-recently, Git[1] wouldn't let you do a shallow checkout and still do useful things. For a large project, for most purposes, downloading all of history is pointless and immensely wasteful. SVN handles that just fine, and people who want git locally can use git-svn. (edit: LLVM is surprisingly small, actually - a git clone comes in at just under 900MB. for more painful examples tho, see repos that commit(ted) binaries, or the scale of Android's repos) [1]: AFAIK Mercurial still has no built-in support, though extensions exist. Which is probably the right choice for Mercurial.
bootload 10y ago
"Compilers are usually not networked, concurrent, or timing-dependent, and overall interact with the outside world only in very constrained ways." Are there any parallel compliers?
- tux3 10y ago
  I can say that for C and C++, the compilation is very often parallelized at the translation unit (file) level, by starting multiple instances of the compiler either locally or over a network with something like distcc. This is simple and effective enough that there wouldn't be much gain in parallelizing the compilers: all the cores are already busy most of the time.
- enqk 10y ago
  Microsoft's CL.exe is, through the /MP option
  - boris 10y ago
    Which only has effect if you pass multiple files to compile.
- Locke1689 10y ago
  It depends on what you mean by parallel. Certainly the Roslyn C# compiler is highly parallel. All files are parsed in parallel, then all classes are bound (semantically analyzed) in parallel, then the IL serialization phase is sequential.
  - bootload 10y ago
    "It depends on what you mean by parallel." Across different machines, not cores on ^a^ chip?
    - Locke1689 10y ago
      I wouldn't say that's what most people mean by parallel, but in that case I think you're better off building a layer on top of the compiler for that. For instance, provided deterministic compilation you could keep a networked cache of compiled libraries that would be delivered as needed. Trying to be network-parallel at any finer level is probably a waste of time -- network and (de)serialization overhead would eat away all the advantages.
newsat13 10y ago
One quirk of llvm is that they don't have a pre-commit CI.

87 points by awalGarg 10y ago

I always find committing to LLVM very nerve wrecking, because of the post-commit CI testing. LLVM has so many architectures that more often than not something I write will fail on one of them. And the only way for me to find out is to commit it, wait for the buildbot to fail (which can take a few hours, during which I really can't leave my computer lest I leave trunk broken on some buildbot, which is a big faux pas), revert it and then figure out what went wrong. I'm hoping that at some point this will be improved, such that I can run the whole buildbot army on my commit before putting it on trunk.

wyldfire 10y ago
I get the feeling that there's parts of the community that feel the same way. I'm hoping that the planned move to github will naturally cascade into pre-commit checks.
- ezanmoto 10y ago
  Do you know of solutions that exist for pre-commit checks for GitHub at the moment?
  - wyldfire 10y ago
    In general? Yes, Travis CI seems to be a very popular one. Many folks use Appveyor for Windows support AFAICT.
  - dikaiosune 10y ago
    Rust uses https://github.com/rust-community/bors https://github.com/rust-community/bors which maintains a linear queue of PRs which can only land after being rebased onto the commits before the PR and subsequently passing tests.
  - Joky 10y ago
    Swift also has a lot of pre-commit CI: - here is the Jenkins bots page: https://ci.swift.org/view/Pull%20Request/ https://ci.swift.org/view/Pull%20Request/ - example of PR test reporting: https://github.com/apple/swift/pull/6802 https://github.com/apple/swift/pull/6802
  - WalterBright 10y ago
    dlang uses several. One notable one is the "autotester" written by Brad Roberts. It's built to ensure that breaking the D compiler (as tested by the test suite) does not result in commits. It's incredibly useful.
- Joky 10y ago
  A big big problem is having enough hardware to have a CI throughput one or two order of magnitude what it is now. Unfortunately that's not an easy thing considering how "heavy" it is to build and test the full toolchain.
shanemhansen 10y ago
That's odd. It would be nice if branches could be tested with the build bot.
- gus_massa 10y ago
  I agree. When I submit a commit for Racket, the branch is tested in Travis and AppVeyor. That catches a lot of error. Anyway, Racket has an additional internal CI called DrDr and some very subtle errors are only detected there, after the commit is merged.
Locke1689 10y ago
Most of the .NET repos are structured to use inner and outer loop testing with Jenkins. Most tests on most architectures are run in the inner loop, which are kicked off in parallel as soon as you make a pull request to one of the .NET repositories on Github. Some repositories, like CoreCLR, have outer loop testing that runs on a separate schedule (nightly, I think), but those tests are far less likely to break and are more devoted to finding rare and difficult to compute edge cases.

SloopJon 10y ago

Interesting to see screenshots of LCOV. I'm hoping to get an intern to work on test coverage this summer, and I wondered whether LCOV is still current. Looks like the latest release is from December 2016.

Joky 10y ago
The screenshots are from the "old" style coverage. We have much better views now: http://lab.llvm.org:8080/coverage/coverage-reports/opt/coverage/Users/buildslave/jenkins/sharedspace/clang-stage2-coverage-R@2/llvm/lib/Transforms/Scalar/CorrelatedValuePropagation.cpp.html#L354 http://lab.llvm.org:8080/coverage/coverage-reports/opt/cover... See for instance how having your cursor on top of each side of a condition tells you how many times each individual condition were evaluated!

mp3geek 10y ago

I do find it strange that such a large project isn't using a better VCS. SVN seems to be very antiquated.

wyldfire 10y ago
It's its size that makes it difficult to move. Some major ecosystem stuff is designed around the svn infrastructure. When the will arrived to make a change, it seemed natural to migrate not just to a different VCS but a different host. And this seemed to spawn a new debate: monorepo vs multi-repo. [still open AFAIK] At the recent 2016 US Dev Conf, there was a consensus to move to git and that the new host would be github. Really subjective IMO part: In general, there's tons of really smart folks working on really awesome stuff in LLVM+clang+etc. There's a handful of folks also focusing on the general "plumbing" software within and among those projects. The meta-plumbing job of the dev infrastructure is "kinda interesting" to several folks who want to improve the way the project is developed. But "kinda interesting" doesn't pay the bills and so it's a second (or nth responsibility) for the folks volunteering to work on it. Add to that the "no good deed goes unpunished" rule that they'll get the responsibility/blame after making a sweeping change, it means it will require extreme patience and caution.
- Joky 10y ago
  Yes! Such work is all done on a volunteering basis, when we find time for it on top of the "real work" (bug fixes, features, ...). All the infrastructure work is not always rewarding, you just get the upset people yelling at you :) Current status: http://lists.llvm.org/pipermail/llvm-dev/2017-January/109015.html http://lists.llvm.org/pipermail/llvm-dev/2017-January/109015...
Groxx 10y ago
Until semi-recently, Git[1] wouldn't let you do a shallow checkout and still do useful things. For a large project, for most purposes, downloading all of history is pointless and immensely wasteful. SVN handles that just fine, and people who want git locally can use git-svn. (edit: LLVM is surprisingly small, actually - a git clone comes in at just under 900MB. for more painful examples tho, see repos that commit(ted) binaries, or the scale of Android's repos) [1]: AFAIK Mercurial still has no built-in support, though extensions exist. Which is probably the right choice for Mercurial.

bootload 10y ago

"Compilers are usually not networked, concurrent, or timing-dependent, and overall interact with the outside world only in very constrained ways." Are there any parallel compliers?

tux3 10y ago
I can say that for C and C++, the compilation is very often parallelized at the translation unit (file) level, by starting multiple instances of the compiler either locally or over a network with something like distcc. This is simple and effective enough that there wouldn't be much gain in parallelizing the compilers: all the cores are already busy most of the time.
enqk 10y ago
Microsoft's CL.exe is, through the /MP option
- boris 10y ago
  Which only has effect if you pass multiple files to compile.
Locke1689 10y ago
It depends on what you mean by parallel. Certainly the Roslyn C# compiler is highly parallel. All files are parsed in parallel, then all classes are bound (semantically analyzed) in parallel, then the IL serialization phase is sequential.
- bootload 10y ago
  "It depends on what you mean by parallel." Across different machines, not cores on ^a^ chip?
  - Locke1689 10y ago
    I wouldn't say that's what most people mean by parallel, but in that case I think you're better off building a layer on top of the compiler for that. For instance, provided deterministic compilation you could keep a networked cache of compiled libraries that would be delivered as needed. Trying to be network-parallel at any finer level is probably a waste of time -- network and (de)serialization overhead would eat away all the advantages.

newsat13 10y ago

One quirk of llvm is that they don't have a pre-commit CI.