It's not all the way to what you're proposing, but CPAN has a very nice means of testing packages for system compatibility. By default, users installing new pac

It's not all the way to what you're proposing, but CPAN has a very nice means of testing packages for system compatibility. By default, users installing new packages run the tests for those packages. Most CPAN clients can be configured to report those test results back to central locations. This isn't the same as running the tests for the consuming project, but it demonstrates the feasibility of "crowdsourcing" tests for system/language-runtime-version/other-package-version incompatibilities.