6 ms·

Would it not be possible to track which lines and branches get executed by each test (normally you'd track this for your code coverage report, yes?), then have

Would it not be possible to track which lines and branches get executed by each test (normally you'd track this for your code coverage report, yes?), then have your IDE only re-run the tests which would have been affected by any changes you've made? You'd still want to run the full-suite when you commit, but it seems like a possible sanity check. Unless I'm overlooking something :)

jdminhbg 16y ago
Yes -- in Ruby the tool for that is `autotest`. It watches your project directory and automatically runs relevant tests when you change application code or tests.
- patio11 16y ago
  Autotest has been the single biggest instigator in me adopting a more rigorous testing procedure for AR than I have previously used. The number of bugs it catches practically before I leave the method I'm tweaking is astounding. (I mostly do unit testing, but also do some testing against APIs to ensure that I'm passing them data in the format they expect. One of my external service providers had downtime yesterday, and autotest caught it about six hours before they did. I wasn't successful in waking them up, but at least I was able to push a hotfix to AR to minimize inconvenience while I waited for their API to come back.) [Edit: I might consider, if tests were taking that long, spinning up a fleet of VPSes to munch through them for me, maybe with a continuous integration server. Unit tests should be disgustingly parallel, since they're small units of work and shouldn't depend on each other. That means if one Macbook can execute the suite in 15 minutes then you should be able to have 30 Macbook-equivalents running either in the server room or the cloud munch through the same test suite in 30 seconds, give or take a little startup time.)
  - jamis 16y ago
    We do have a CI server, and as you said it works well for catching failing tests. However, it requires that you commit and push your changes in order to test them, which means you are effectively publishing untested changes to your entire team. The same for any kind of distributed testing, unless you are using a shared volume to host your sandbox. I'm running a Mac Pro with 8 cores, so there is a fair bit of parallelization I can do locally too. Unfortunately, the tests all depend on the database, and while I can certainly use tools like deep-test to spin up separate DB's for each worker, I've found that doing so adds a full 60 seconds to the test run. I fear that until we eliminate the database from (most of) our tests, super-fast runs will continue to elude us. CI and distributed tests are good things, no question, but I'm still looking for ways to make it possible to run my tests locally in TDD-fashion. I'm far from out of ideas, it's just a matter of making time to experiment.
    - patio11 16y ago
      This is totally a personal/team comfort question, but is there any reason why you can't have two remotes? "git push jamistest" might use a few bits on a spinning platter somewhere, but that is cheap, and there is no reason your team has to see it if you don't push it to the master repo, any more than they see changes you keep on your local repo.
      - jamis 16y ago
        Aside from me simply wanting to be able to quickly run my tests locally, you mean? :) Mostly it's just an issue of configuring that so it works for all the programmers. Each would need their own remote, and each would need to be hooked into CI. Definitely possible, it just hasn't been a priority.
    - xentronium 16y ago
      Most of the time you need only some of your tests (the ones you're working on currently). Then, preloading the framework becomes the real bottleneck. But it's solvable too, via spork (I believe, there is spork-testunit, but I never tried it). Most of my coding is done in minute loops (add test / watch tests fail / add couple of lines / watch tests succeed). YMMV.
      - aaronblohowiak 16y ago
        spork is friggen awesome, and if you are on MRI/Yarv you really should use it. http://spork.rubyforge.org/ http://spork.rubyforge.org/
    - gnufied 16y ago
      Hey Jamis, Your CI server should be able to accept a patchset and run it without committing it. TeamCity does this (or roll your own CI server can do this too!).
  - steveklabnik 16y ago
    There's actually a gem for your edit: https://github.com/jasonm/parallel_specs https://github.com/jasonm/parallel_specs Then there's hydra: https://github.com/ngauthier/hydra https://github.com/ngauthier/hydra And the excellent presentation on speeding up your tests with hydra that I can't find. It was one of those showoff presentations on GitHub somewhere...
    - bgentry 16y ago
      I believe you're looking for the presentation "Grease Your Suite" hosted here on Heroku: http://grease-your-suite.heroku.com/ http://grease-your-suite.heroku.com/
      - steveklabnik 16y ago
        Absolutely! Thank you!
      - anveo 16y ago
        Video: http://vimeo.com/12705404 http://vimeo.com/12705404
        JonnieCache 16y ago
        I cannot recommend this video enough. It is a step by step guide with code showing how the guy reduced his suite's running time from half an hour to 15 seconds. A lot of it is preventing so many db objects like in the OP, but it also discusses parallelizing the tests using hydra to make use of multicore. This ties into the db access thing as well though as to do parallell tests you have to make all the cases set up and tear down their own fixtures transactionally. Not doing this is a leading cause of slow tests anyway. He even has metrics for stuff like running the whole suite off a ramdisk, the effects of SSDs, etc etc. He worked on this at his company for weeks/months apparently. It's too hard to explain here, just watch that video if you do ANY ruby testing!
- tmcneal 16y ago
  Autotest sounds like a great tool. Is there a Python equivalent?
  - rcurses 16y ago
    I haven't used it, but sniffer sounds like it fits the bill. http://pypi.python.org/pypi/sniffer/0.2.2 http://pypi.python.org/pypi/sniffer/0.2.2
- gnufied 16y ago
  For a sufficiently large project, just starting autotest can be a drag. For projects with large suite, I think the only way which does not have diminishing returns is parallelizing the test suite.
- kscaldef 16y ago
  I don't know if it's gotten more sophisticated recently, but the last time I played with autotest it had a very naive concept of "relevant tests". Certainly not the sort of runtime analysis the parent describes. (Also, on our moderately large codebase it noticeably impaired the performance of my whole machine because of the way it constantly scanned the entire directory tree. I know there was some talk of adding edge triggering via kqueue or fsevents, but no one had done it at that time.)
  - xentronium 16y ago
    Autotest depends on the rules you use. Find autotest/discovery.rb for your project to see something like: Autotest.add_discovery { "Rspec2" } Autotest.add_discovery { "Rails" } # both of the frameworks have their predefined rulesets No one prevents you from adding your own discovery rules. Also you're looking for autotest-fsevent if you're using a mac. Probably something similar exists for linux.
jsankey 16y ago
It is important to remember that you are changing the code (and probably the tests) at the same time, so your coverage data is getting stale. So you can only "mostly" predict which tests to run, which is somewhat useful but can also make these cut down runs misleading.
larsberg 16y ago
My team (back at MSFT) did this on checkin. It'd look at the class you changed and what tests had code coverage that touched that class, and rerun just those tests before allowing the commit. Post-commit, the rolling builds would update the code coverage for all of the different build flavors (debug, release, etc). One nice side-effect of this support is that it also gave us code coverage deltas from the last build, so the system would send out embarrassing numbers with your commit if you'd dropped the overall coverage of the module.
deleted 16y ago
[deleted]
codex 16y ago
You can safely do this as long as your tests are fully deterministic. If they use inputs which change over time, like a system clock, or a database, or share mutable state between threads, then you can't be sure that the code coverage does not change from run to run. There's more to it than simply checking each line with a coverage tool--you also need to rerun tests when any implicitly referenced global data, strings, input files, command line parameters, etc. change, including calls to third party libraries for which the source is not available. Usually a build system and compiler can track some of these dependencies already, so you may be able to leverage those. As long as you occasionally run the full test suite, though, you may not care to cover these cases for your incremental tests.

jdminhbg 16y ago

Yes -- in Ruby the tool for that is `autotest`. It watches your project directory and automatically runs relevant tests when you change application code or tests.

patio11 16y ago
Autotest has been the single biggest instigator in me adopting a more rigorous testing procedure for AR than I have previously used. The number of bugs it catches practically before I leave the method I'm tweaking is astounding. (I mostly do unit testing, but also do some testing against APIs to ensure that I'm passing them data in the format they expect. One of my external service providers had downtime yesterday, and autotest caught it about six hours before they did. I wasn't successful in waking them up, but at least I was able to push a hotfix to AR to minimize inconvenience while I waited for their API to come back.) [Edit: I might consider, if tests were taking that long, spinning up a fleet of VPSes to munch through them for me, maybe with a continuous integration server. Unit tests should be disgustingly parallel, since they're small units of work and shouldn't depend on each other. That means if one Macbook can execute the suite in 15 minutes then you should be able to have 30 Macbook-equivalents running either in the server room or the cloud munch through the same test suite in 30 seconds, give or take a little startup time.)
- jamis 16y ago
  We do have a CI server, and as you said it works well for catching failing tests. However, it requires that you commit and push your changes in order to test them, which means you are effectively publishing untested changes to your entire team. The same for any kind of distributed testing, unless you are using a shared volume to host your sandbox. I'm running a Mac Pro with 8 cores, so there is a fair bit of parallelization I can do locally too. Unfortunately, the tests all depend on the database, and while I can certainly use tools like deep-test to spin up separate DB's for each worker, I've found that doing so adds a full 60 seconds to the test run. I fear that until we eliminate the database from (most of) our tests, super-fast runs will continue to elude us. CI and distributed tests are good things, no question, but I'm still looking for ways to make it possible to run my tests locally in TDD-fashion. I'm far from out of ideas, it's just a matter of making time to experiment.
  - patio11 16y ago
    This is totally a personal/team comfort question, but is there any reason why you can't have two remotes? "git push jamistest" might use a few bits on a spinning platter somewhere, but that is cheap, and there is no reason your team has to see it if you don't push it to the master repo, any more than they see changes you keep on your local repo.
    - jamis 16y ago
      Aside from me simply wanting to be able to quickly run my tests locally, you mean? :) Mostly it's just an issue of configuring that so it works for all the programmers. Each would need their own remote, and each would need to be hooked into CI. Definitely possible, it just hasn't been a priority.
  - xentronium 16y ago
    Most of the time you need only some of your tests (the ones you're working on currently). Then, preloading the framework becomes the real bottleneck. But it's solvable too, via spork (I believe, there is spork-testunit, but I never tried it). Most of my coding is done in minute loops (add test / watch tests fail / add couple of lines / watch tests succeed). YMMV.
    - aaronblohowiak 16y ago
      spork is friggen awesome, and if you are on MRI/Yarv you really should use it. http://spork.rubyforge.org/ http://spork.rubyforge.org/
  - gnufied 16y ago
    Hey Jamis, Your CI server should be able to accept a patchset and run it without committing it. TeamCity does this (or roll your own CI server can do this too!).
- steveklabnik 16y ago
  There's actually a gem for your edit: https://github.com/jasonm/parallel_specs https://github.com/jasonm/parallel_specs Then there's hydra: https://github.com/ngauthier/hydra https://github.com/ngauthier/hydra And the excellent presentation on speeding up your tests with hydra that I can't find. It was one of those showoff presentations on GitHub somewhere...
  - bgentry 16y ago
    I believe you're looking for the presentation "Grease Your Suite" hosted here on Heroku: http://grease-your-suite.heroku.com/ http://grease-your-suite.heroku.com/
    - steveklabnik 16y ago
      Absolutely! Thank you!
    - anveo 16y ago
      Video: http://vimeo.com/12705404 http://vimeo.com/12705404
      - JonnieCache 16y ago
        I cannot recommend this video enough. It is a step by step guide with code showing how the guy reduced his suite's running time from half an hour to 15 seconds. A lot of it is preventing so many db objects like in the OP, but it also discusses parallelizing the tests using hydra to make use of multicore. This ties into the db access thing as well though as to do parallell tests you have to make all the cases set up and tear down their own fixtures transactionally. Not doing this is a leading cause of slow tests anyway. He even has metrics for stuff like running the whole suite off a ramdisk, the effects of SSDs, etc etc. He worked on this at his company for weeks/months apparently. It's too hard to explain here, just watch that video if you do ANY ruby testing!
tmcneal 16y ago
Autotest sounds like a great tool. Is there a Python equivalent?
- rcurses 16y ago
  I haven't used it, but sniffer sounds like it fits the bill. http://pypi.python.org/pypi/sniffer/0.2.2 http://pypi.python.org/pypi/sniffer/0.2.2
gnufied 16y ago
For a sufficiently large project, just starting autotest can be a drag. For projects with large suite, I think the only way which does not have diminishing returns is parallelizing the test suite.
kscaldef 16y ago
I don't know if it's gotten more sophisticated recently, but the last time I played with autotest it had a very naive concept of "relevant tests". Certainly not the sort of runtime analysis the parent describes. (Also, on our moderately large codebase it noticeably impaired the performance of my whole machine because of the way it constantly scanned the entire directory tree. I know there was some talk of adding edge triggering via kqueue or fsevents, but no one had done it at that time.)
- xentronium 16y ago
  Autotest depends on the rules you use. Find autotest/discovery.rb for your project to see something like: Autotest.add_discovery { "Rspec2" } Autotest.add_discovery { "Rails" } # both of the frameworks have their predefined rulesets No one prevents you from adding your own discovery rules. Also you're looking for autotest-fsevent if you're using a mac. Probably something similar exists for linux.

jsankey 16y ago

It is important to remember that you are changing the code (and probably the tests) at the same time, so your coverage data is getting stale. So you can only "mostly" predict which tests to run, which is somewhat useful but can also make these cut down runs misleading.

larsberg 16y ago

My team (back at MSFT) did this on checkin. It'd look at the class you changed and what tests had code coverage that touched that class, and rerun just those tests before allowing the commit. Post-commit, the rolling builds would update the code coverage for all of the different build flavors (debug, release, etc). One nice side-effect of this support is that it also gave us code coverage deltas from the last build, so the system would send out embarrassing numbers with your commit if you'd dropped the overall coverage of the module.

deleted 16y ago

[deleted]

codex 16y ago

You can safely do this as long as your tests are fully deterministic. If they use inputs which change over time, like a system clock, or a database, or share mutable state between threads, then you can't be sure that the code coverage does not change from run to run. There's more to it than simply checking each line with a coverage tool--you also need to rerun tests when any implicitly referenced global data, strings, input files, command line parameters, etc. change, including calls to third party libraries for which the source is not available. Usually a build system and compiler can track some of these dependencies already, so you may be able to leverage those. As long as you occasionally run the full test suite, though, you may not care to cover these cases for your incremental tests.