4 ms·

The most interesting part for me was > They appear to use code coverage to determine what tests > to run. "We're not going to run a test unless your

by necubi 12y ago

The most interesting part for me was > They appear to use code coverage to determine what tests > to run. "We're not going to run a test unless your diff > might actually have broken it." This seems like such an obvious optimization in hindsight, I'm surprised it's not more common. Does anybody know of other CIs that can do this?

TheLoneWolfling 12y ago
This seems like a good optimization in theory. In practice? I think it may end up selectively pruning for hard-to-find bugs, which is not a desirable feature of a test suite.
dperfect 12y ago
I'd be hesitant to rely 100% on this approach (and from the notes it sounds like FB still runs the other tests - just less frequently in some cases), especially where the code coverage tools aren't guaranteed to account for unintended interactions between seemingly unrelated pieces of code. In theory, well-written code/tests shouldn't have those kinds of problems in the first place, but larger projects often don't deserve that level of trust (in my opinion). Still, I think it's a great idea for giving quick, "preliminary" feedback to developers - as long as the full test suite is still run periodically.
- necubi 12y ago
  Yeah, absolutely. You'd want to run your full test suite (which probably includes heavier integration tests as well) before deploying, but I think in a lot of cases it's valuable to provide, say, 95% accurate feedback quickly.
  - icefox 12y ago
    Another way to put it would be I will happily only take 95% if the other option is we don't do anything and people can even commit stuff that doesn't even build let alone pass tests.
andrewjshults 12y ago
We have a home grown solution for this. We use pants (https://pantsbuild.github.io/ https://pantsbuild.github.io/) for our Python builds and phabricator for code reviews. Pants gives us a dependency graph so from which files get touched in a given diff, we can feed that back into pants to get the targets to build and the test suites to run. It's not as granular as running individual tests in a suite for small changes, but it definitely helps cut most runs down to just a small percentage of all the tests. The ability to pants to build test suites in isolation is also pretty key since it means we can run a whole bunch of suites in parallel so in many cases the total test run time can be just the time of the longest individual suite. As others have mentioned, kicking off full runs on a regular basis is important to make sure there aren't weird cases that break and aren't caught in this approach.
pfg 12y ago
@tenderlove was experimenting with this idea for ruby/rails: http://tenderlovemaking.com/2015/02/13/predicting-test-failues.html http://tenderlovemaking.com/2015/02/13/predicting-test-failu... Here's hoping someone will make a gem for this stuff.
icefox 12y ago
I have used that (only running a few tests that match the diff) in the past to great success as the first level of testing that would occur on the users computer before committing code. It reduced the amount of build failures by 99% and test failures by a similarly high amount. On the server we would run those tests first and we would continuously sort the order of tests so tests that failed in the past would be run first to reduce the build+test time and find failures faster. There are lots of tweaks that can be done, but these days having automatic build+test before a commit gets into the main branch feels like it is the norm. Even open source github projects can use something like TravisCI to prevent bad changes from getting into a project.
obstinate 12y ago
With Bazel, it's pretty easy to make this transformation: - Get a list of source files changed. - Use bazel query with rdeps to get a list of tests that depend on that source file. - Run those tests. A little less implicit than the code coverage approach, but roughly the same effect.
ignoramous 12y ago
I'm not sure how Facebook does it, but there are a lot ways to get something like this up and running. I have experience with using Facebook's pfff [0]. Its codegraph feature builds up a graph of code that you can query. Doing something like 'get me a list of all classes this node depends on' on Java code is as simple as executing 'Graph.use(node name)' where nodename can be a class, package, method... Pfff supports many more languages, and its built on OCaml. Facebook has a history of open sourcing stuff, I hope they open source their tools mentioned in the talk. [0] https://github.com/facebook/pfff https://github.com/facebook/pfff

by necubi 12y ago

TheLoneWolfling 12y ago

This seems like a good optimization in theory. In practice? I think it may end up selectively pruning for hard-to-find bugs, which is not a desirable feature of a test suite.

dperfect 12y ago

I'd be hesitant to rely 100% on this approach (and from the notes it sounds like FB still runs the other tests - just less frequently in some cases), especially where the code coverage tools aren't guaranteed to account for unintended interactions between seemingly unrelated pieces of code. In theory, well-written code/tests shouldn't have those kinds of problems in the first place, but larger projects often don't deserve that level of trust (in my opinion). Still, I think it's a great idea for giving quick, "preliminary" feedback to developers - as long as the full test suite is still run periodically.

necubi 12y ago
Yeah, absolutely. You'd want to run your full test suite (which probably includes heavier integration tests as well) before deploying, but I think in a lot of cases it's valuable to provide, say, 95% accurate feedback quickly.
- icefox 12y ago
  Another way to put it would be I will happily only take 95% if the other option is we don't do anything and people can even commit stuff that doesn't even build let alone pass tests.

andrewjshults 12y ago

We have a home grown solution for this. We use pants (https://pantsbuild.github.io/ https://pantsbuild.github.io/) for our Python builds and phabricator for code reviews. Pants gives us a dependency graph so from which files get touched in a given diff, we can feed that back into pants to get the targets to build and the test suites to run. It's not as granular as running individual tests in a suite for small changes, but it definitely helps cut most runs down to just a small percentage of all the tests. The ability to pants to build test suites in isolation is also pretty key since it means we can run a whole bunch of suites in parallel so in many cases the total test run time can be just the time of the longest individual suite. As others have mentioned, kicking off full runs on a regular basis is important to make sure there aren't weird cases that break and aren't caught in this approach.

pfg 12y ago

@tenderlove was experimenting with this idea for ruby/rails: http://tenderlovemaking.com/2015/02/13/predicting-test-failues.html http://tenderlovemaking.com/2015/02/13/predicting-test-failu... Here's hoping someone will make a gem for this stuff.

icefox 12y ago

I have used that (only running a few tests that match the diff) in the past to great success as the first level of testing that would occur on the users computer before committing code. It reduced the amount of build failures by 99% and test failures by a similarly high amount. On the server we would run those tests first and we would continuously sort the order of tests so tests that failed in the past would be run first to reduce the build+test time and find failures faster. There are lots of tweaks that can be done, but these days having automatic build+test before a commit gets into the main branch feels like it is the norm. Even open source github projects can use something like TravisCI to prevent bad changes from getting into a project.

obstinate 12y ago

With Bazel, it's pretty easy to make this transformation: - Get a list of source files changed. - Use bazel query with rdeps to get a list of tests that depend on that source file. - Run those tests. A little less implicit than the code coverage approach, but roughly the same effect.

ignoramous 12y ago

I'm not sure how Facebook does it, but there are a lot ways to get something like this up and running. I have experience with using Facebook's pfff [0]. Its codegraph feature builds up a graph of code that you can query. Doing something like 'get me a list of all classes this node depends on' on Java code is as simple as executing 'Graph.use(node name)' where nodename can be a class, package, method... Pfff supports many more languages, and its built on OCaml. Facebook has a history of open sourcing stuff, I hope they open source their tools mentioned in the talk. [0] https://github.com/facebook/pfff https://github.com/facebook/pfff