5 ms·

Yes -- in Ruby the tool for that is `autotest`. It watches your project directory and automatically runs relevant tests when you change application code or tes

Yes -- in Ruby the tool for that is `autotest`. It watches your project directory and automatically runs relevant tests when you change application code or tests.

patio11 16y ago
Autotest has been the single biggest instigator in me adopting a more rigorous testing procedure for AR than I have previously used. The number of bugs it catches practically before I leave the method I'm tweaking is astounding. (I mostly do unit testing, but also do some testing against APIs to ensure that I'm passing them data in the format they expect. One of my external service providers had downtime yesterday, and autotest caught it about six hours before they did. I wasn't successful in waking them up, but at least I was able to push a hotfix to AR to minimize inconvenience while I waited for their API to come back.) [Edit: I might consider, if tests were taking that long, spinning up a fleet of VPSes to munch through them for me, maybe with a continuous integration server. Unit tests should be disgustingly parallel, since they're small units of work and shouldn't depend on each other. That means if one Macbook can execute the suite in 15 minutes then you should be able to have 30 Macbook-equivalents running either in the server room or the cloud munch through the same test suite in 30 seconds, give or take a little startup time.)
- jamis 16y ago
  We do have a CI server, and as you said it works well for catching failing tests. However, it requires that you commit and push your changes in order to test them, which means you are effectively publishing untested changes to your entire team. The same for any kind of distributed testing, unless you are using a shared volume to host your sandbox. I'm running a Mac Pro with 8 cores, so there is a fair bit of parallelization I can do locally too. Unfortunately, the tests all depend on the database, and while I can certainly use tools like deep-test to spin up separate DB's for each worker, I've found that doing so adds a full 60 seconds to the test run. I fear that until we eliminate the database from (most of) our tests, super-fast runs will continue to elude us. CI and distributed tests are good things, no question, but I'm still looking for ways to make it possible to run my tests locally in TDD-fashion. I'm far from out of ideas, it's just a matter of making time to experiment.
  - patio11 16y ago
    This is totally a personal/team comfort question, but is there any reason why you can't have two remotes? "git push jamistest" might use a few bits on a spinning platter somewhere, but that is cheap, and there is no reason your team has to see it if you don't push it to the master repo, any more than they see changes you keep on your local repo.
    - jamis 16y ago
      Aside from me simply wanting to be able to quickly run my tests locally, you mean? :) Mostly it's just an issue of configuring that so it works for all the programmers. Each would need their own remote, and each would need to be hooked into CI. Definitely possible, it just hasn't been a priority.
  - xentronium 16y ago
    Most of the time you need only some of your tests (the ones you're working on currently). Then, preloading the framework becomes the real bottleneck. But it's solvable too, via spork (I believe, there is spork-testunit, but I never tried it). Most of my coding is done in minute loops (add test / watch tests fail / add couple of lines / watch tests succeed). YMMV.
    - aaronblohowiak 16y ago
      spork is friggen awesome, and if you are on MRI/Yarv you really should use it. http://spork.rubyforge.org/ http://spork.rubyforge.org/
  - gnufied 16y ago
    Hey Jamis, Your CI server should be able to accept a patchset and run it without committing it. TeamCity does this (or roll your own CI server can do this too!).
- steveklabnik 16y ago
  There's actually a gem for your edit: https://github.com/jasonm/parallel_specs https://github.com/jasonm/parallel_specs Then there's hydra: https://github.com/ngauthier/hydra https://github.com/ngauthier/hydra And the excellent presentation on speeding up your tests with hydra that I can't find. It was one of those showoff presentations on GitHub somewhere...
  - bgentry 16y ago
    I believe you're looking for the presentation "Grease Your Suite" hosted here on Heroku: http://grease-your-suite.heroku.com/ http://grease-your-suite.heroku.com/
    - steveklabnik 16y ago
      Absolutely! Thank you!
    - anveo 16y ago
      Video: http://vimeo.com/12705404 http://vimeo.com/12705404
      - JonnieCache 16y ago
        I cannot recommend this video enough. It is a step by step guide with code showing how the guy reduced his suite's running time from half an hour to 15 seconds. A lot of it is preventing so many db objects like in the OP, but it also discusses parallelizing the tests using hydra to make use of multicore. This ties into the db access thing as well though as to do parallell tests you have to make all the cases set up and tear down their own fixtures transactionally. Not doing this is a leading cause of slow tests anyway. He even has metrics for stuff like running the whole suite off a ramdisk, the effects of SSDs, etc etc. He worked on this at his company for weeks/months apparently. It's too hard to explain here, just watch that video if you do ANY ruby testing!
tmcneal 16y ago
Autotest sounds like a great tool. Is there a Python equivalent?
- rcurses 16y ago
  I haven't used it, but sniffer sounds like it fits the bill. http://pypi.python.org/pypi/sniffer/0.2.2 http://pypi.python.org/pypi/sniffer/0.2.2
gnufied 16y ago
For a sufficiently large project, just starting autotest can be a drag. For projects with large suite, I think the only way which does not have diminishing returns is parallelizing the test suite.
kscaldef 16y ago
I don't know if it's gotten more sophisticated recently, but the last time I played with autotest it had a very naive concept of "relevant tests". Certainly not the sort of runtime analysis the parent describes. (Also, on our moderately large codebase it noticeably impaired the performance of my whole machine because of the way it constantly scanned the entire directory tree. I know there was some talk of adding edge triggering via kqueue or fsevents, but no one had done it at that time.)
- xentronium 16y ago
  Autotest depends on the rules you use. Find autotest/discovery.rb for your project to see something like: Autotest.add_discovery { "Rspec2" } Autotest.add_discovery { "Rails" } # both of the frameworks have their predefined rulesets No one prevents you from adding your own discovery rules. Also you're looking for autotest-fsevent if you're using a mac. Probably something similar exists for linux.

Yes -- in Ruby the tool for that is `autotest`. It watches your project directory and automatically runs relevant tests when you change application code or tests.

patio11 16y ago

Autotest has been the single biggest instigator in me adopting a more rigorous testing procedure for AR than I have previously used. The number of bugs it catches practically before I leave the method I'm tweaking is astounding. (I mostly do unit testing, but also do some testing against APIs to ensure that I'm passing them data in the format they expect. One of my external service providers had downtime yesterday, and autotest caught it about six hours before they did. I wasn't successful in waking them up, but at least I was able to push a hotfix to AR to minimize inconvenience while I waited for their API to come back.) [Edit: I might consider, if tests were taking that long, spinning up a fleet of VPSes to munch through them for me, maybe with a continuous integration server. Unit tests should be disgustingly parallel, since they're small units of work and shouldn't depend on each other. That means if one Macbook can execute the suite in 15 minutes then you should be able to have 30 Macbook-equivalents running either in the server room or the cloud munch through the same test suite in 30 seconds, give or take a little startup time.)

jamis 16y ago
We do have a CI server, and as you said it works well for catching failing tests. However, it requires that you commit and push your changes in order to test them, which means you are effectively publishing untested changes to your entire team. The same for any kind of distributed testing, unless you are using a shared volume to host your sandbox. I'm running a Mac Pro with 8 cores, so there is a fair bit of parallelization I can do locally too. Unfortunately, the tests all depend on the database, and while I can certainly use tools like deep-test to spin up separate DB's for each worker, I've found that doing so adds a full 60 seconds to the test run. I fear that until we eliminate the database from (most of) our tests, super-fast runs will continue to elude us. CI and distributed tests are good things, no question, but I'm still looking for ways to make it possible to run my tests locally in TDD-fashion. I'm far from out of ideas, it's just a matter of making time to experiment.
- patio11 16y ago
  This is totally a personal/team comfort question, but is there any reason why you can't have two remotes? "git push jamistest" might use a few bits on a spinning platter somewhere, but that is cheap, and there is no reason your team has to see it if you don't push it to the master repo, any more than they see changes you keep on your local repo.
  - jamis 16y ago
    Aside from me simply wanting to be able to quickly run my tests locally, you mean? :) Mostly it's just an issue of configuring that so it works for all the programmers. Each would need their own remote, and each would need to be hooked into CI. Definitely possible, it just hasn't been a priority.
- xentronium 16y ago
  Most of the time you need only some of your tests (the ones you're working on currently). Then, preloading the framework becomes the real bottleneck. But it's solvable too, via spork (I believe, there is spork-testunit, but I never tried it). Most of my coding is done in minute loops (add test / watch tests fail / add couple of lines / watch tests succeed). YMMV.
  - aaronblohowiak 16y ago
    spork is friggen awesome, and if you are on MRI/Yarv you really should use it. http://spork.rubyforge.org/ http://spork.rubyforge.org/
- gnufied 16y ago
  Hey Jamis, Your CI server should be able to accept a patchset and run it without committing it. TeamCity does this (or roll your own CI server can do this too!).
steveklabnik 16y ago
There's actually a gem for your edit: https://github.com/jasonm/parallel_specs https://github.com/jasonm/parallel_specs Then there's hydra: https://github.com/ngauthier/hydra https://github.com/ngauthier/hydra And the excellent presentation on speeding up your tests with hydra that I can't find. It was one of those showoff presentations on GitHub somewhere...
- bgentry 16y ago
  I believe you're looking for the presentation "Grease Your Suite" hosted here on Heroku: http://grease-your-suite.heroku.com/ http://grease-your-suite.heroku.com/
  - steveklabnik 16y ago
    Absolutely! Thank you!
  - anveo 16y ago
    Video: http://vimeo.com/12705404 http://vimeo.com/12705404
    - JonnieCache 16y ago
      I cannot recommend this video enough. It is a step by step guide with code showing how the guy reduced his suite's running time from half an hour to 15 seconds. A lot of it is preventing so many db objects like in the OP, but it also discusses parallelizing the tests using hydra to make use of multicore. This ties into the db access thing as well though as to do parallell tests you have to make all the cases set up and tear down their own fixtures transactionally. Not doing this is a leading cause of slow tests anyway. He even has metrics for stuff like running the whole suite off a ramdisk, the effects of SSDs, etc etc. He worked on this at his company for weeks/months apparently. It's too hard to explain here, just watch that video if you do ANY ruby testing!

tmcneal 16y ago

Autotest sounds like a great tool. Is there a Python equivalent?

rcurses 16y ago
I haven't used it, but sniffer sounds like it fits the bill. http://pypi.python.org/pypi/sniffer/0.2.2 http://pypi.python.org/pypi/sniffer/0.2.2

gnufied 16y ago

For a sufficiently large project, just starting autotest can be a drag. For projects with large suite, I think the only way which does not have diminishing returns is parallelizing the test suite.

kscaldef 16y ago

I don't know if it's gotten more sophisticated recently, but the last time I played with autotest it had a very naive concept of "relevant tests". Certainly not the sort of runtime analysis the parent describes. (Also, on our moderately large codebase it noticeably impaired the performance of my whole machine because of the way it constantly scanned the entire directory tree. I know there was some talk of adding edge triggering via kqueue or fsevents, but no one had done it at that time.)

xentronium 16y ago
Autotest depends on the rules you use. Find autotest/discovery.rb for your project to see something like: Autotest.add_discovery { "Rspec2" } Autotest.add_discovery { "Rails" } # both of the frameworks have their predefined rulesets No one prevents you from adding your own discovery rules. Also you're looking for autotest-fsevent if you're using a mac. Probably something similar exists for linux.