10 ms·

I answered "a few critical things" ... but, for the most part, testing is tedious, frustrating, and a time-sink for me. I recently paid someone $100+ an hour fo

I answered "a few critical things" ... but, for the most part, testing is tedious, frustrating, and a time-sink for me. I recently paid someone $100+ an hour for some remote TDD coaching. It's helping a bit but hasn't really change my attitude towards testing (yet). What bugs me: - Testing frameworks and "best practices" change way faster than language frameworks and I simply can't keep up. What rspec version do I use with what version of Rails? Now I have to use Cucumber? I learned some Cucumber ... oh, now Steak is better. [rage comic goes here] - Most bugs/edge cases I encounter in our production apps are things I'd never think to write a test for ... - I deal with custom domains, authentication, and 3rd party API calls in almost every app we have. IMO, this adds 20% or more to the (already high) testing overhead just to get these things configured right in the test suite - More code is moving to front-end Javascript stuff ... so, now I have to write Rails tests AND JS tests? Sounds delightful Feel free to try and convince me otherwise, but I don't ever see myself in the "test ALL the things" camp.

sunir 15y ago
People write and play with test frameworks because they are procrastinating from writing actual tests. Think about it. Just use Test::Unit and move on with your life. Write some tests. That's what counts.
- eric-hu 15y ago
  I'm working on a single page app that's about 30% Rails and 70% Coffeescript/Backbone.js. Test::Unit is practically useless for us since users never hit any plain HTML pages besides the login page. Imagine the current HTML5 Pandora having bugs with one particular song in Chrome only. How do you test for that using Test::Unit?
  - mattbriggs 15y ago
    there are a bajillion libraries out there for unit testing javascript (something test::unit-ish would be qunit). rails + backbone + JST style templating actually makes TDDing your js relatively painless
    - eric-hu 15y ago
      We're looking into integration-level testing for our Javascript since it'd give us full coverage of the bugs we see. I forgot to mention we also use Socket.io for real time push updates to user Backbone models. A good number of bugs don't crop up until another user modifies data and those changes get pushed out to another person.
  - sunir 15y ago
    You don't. You use QUnit and move on with your life.
  - Oompa 15y ago
    Testing with front end javascript is more difficult due to different interpreters, much more variation in environment, etc. That doesn't mean you can't have tests for your core business logic at least in something like Vows or Mocha.
jakejake 15y ago
My approach to testing is not to be obsessed with the latest, greatest framework or 100% code coverage. I try to start with just one or two tests to actually help do things that are tedious or require multiple steps. It takes some time to automate a good test but once you do it immediately starts saving time because you don't have to run the same sequence a thousand times while developing. You can think of it more like a macro that saves you time. Once you write the main test it's easy then to run it with all combinations of good and bad input. By doing that you'll often wind up hitting a pretty good percentage of your code. Then as bugs are discovered due to unexpected input you can just keeping adding more input situations.
- gbog 15y ago
  Testing is like jumping in cool water, one find so many reason to not do it beforewards, but one is so happy to have done it afterwards...
netzpirat 15y ago
At first I was doubtful about testing my JS code, but nowadays I do enjoy it much more that testing the Rails backend. I use my own gem guard-jasmine that runs the specs headless on PhantomJS and it's a real joy! My whole spec suite with over 1000 specs runs in under 3 seconds. I use SinonJS for faking AJAX calls to the backend, but that's just a small subset of all specs since most stuff isn't interacting with the backend.
diminoten 15y ago
How do you feel about regression testing? Maybe instead of "writing" tests for potential bugs, you write tests for bugs you've found already.
- 3am 15y ago
  Regression testing is when you check that new code doesn't break existing functionality. It's preventative, not reactive.
nkassis 15y ago
"- Most bugs/edge cases I encounter in our production apps are things I'd never think to write a test for ..." I feel that way often too but I write test more as a specification for how I want the code to work then as a catch all bugs thing. "- I deal with custom domains, authentication, and 3rd party API calls in almost every app we have. IMO, this adds 20% or more to the (already high) testing overhead just to get these things configured right in the test suite - More code is moving to front-end Javascript stuff ... so, now I have to write Rails tests AND JS tests? Sounds delightful" I feel your pain, I code stuff that use WebGL currently and I find it hard to test that stuff.
tikhonj 15y ago
You should check out QuickCheck for catching edge cases you did not think of. The idea behind QuickCheck is simple--you specify invariants in your code (called "properties") and the framework tests them with random inputs. This tool is very widely used in Haskell, but it's been ported to a whole bunch of other languages and could make your testing more thorough. In Haskell it's also easy to use and more fun than normal tests, but I don't know what it would be like in a different language.
ef4 15y ago
Look at it this way: you must be testing code as you write it anyway. There's really no other sane way to do it. You make a change, you load the page and see that your change worked, or you call your new function from an interactive interpreter. Smart automated testing just takes all that extra test work you're already doing and saves it as you go along. No need to try to invent extra things to test. You just test what you would have tested anyway by hand.
- Iv 15y ago
  You think like that until you hit your first serious regressions and discover it has been in the code base for several months and that the person responsible for it has left. I used to work at a company where automated testing was sending you emails about what your commits broke. It does help at improving code quality.
- harel 15y ago
  But then you'd still do the manual test after you complete your code. Nobody (I hope) codes blind hoping it would work or caught later by a test suite. Test suits don't reveal everything. Only what you tested for.
  - fragsworth 15y ago
    The process goes more like this: 1. Write some code 2. Test it 3. Debug your code 4. Test it 5. Debug your code 6. Test it ... 39. Debug your code 40. Test it; now it finally seems to work If all your "test it" steps are being done manually, you're being very inefficient. A good unit test can actually make development go by faster with the added bonus of defending your code from changes down the road that might screw things up.
    - niccl 15y ago
      That's all fine unless you're exploring a solution space. Then the overhead of writing tests which are thrown away _in entirety_ is outrageous. AFAIK TDD really only works if you're either a) prepared to waste a huge amount of time writing tests which will later be completely redundant or b) working to a very clear set of requirements with tools and frameworks you already know intimately. </Rant>
joske2 15y ago
The point of testing/TDD for me is not (just) about preventing bugs, it is more about having quick feedback. Running a test is faster than waiting until it is deployed and manually clicking around in an application. It is kind of comparable to using a REPL.
cr4zy 15y ago
"- Most bugs/edge cases I encounter in our production apps are things I'd never think to write a test for ..." This is why regression tests are my favorite type of test. The need for the test has been confirmed by real world usage and once you create the regression test to fail, fix the bug, and pass the test, you won't have to ever worry about users seeing that bug again :)
MartinCron 15y ago
but I don't ever see myself in the "test ALL the things" camp Good for you. Extremists on all sides are usually wrong. Shoot for "test MOST OF the things" or "test the MOST IMPORTANT things" or even "test just enough things so that you know if change Y totally breaks MOST IMPORTANT feature Z".
- Oompa 15y ago
  I'm pretty far into the extremist side of TDD, and I'll say that there is a thing as TOO many tests. Your test suite needs to run fast to be really useful. If you have thousands of full stack integration tests that takes an hour to run, you're not going to run them as often as you should be, if at all, ad might as well delete them.
  - MartinCron 15y ago
    If they are run by a continuous integration server on check in, it kind of doesn't matter how long they take to run.
    - andrewaylett 15y ago
      That's not strictly true: you'd still like to get reasonably quick feedback, especially if you're trying to make a release. It's also nice to be reasonably confident that your commit won't break the build, for which you should probably run a good chunk of the tests before committing. There are some workflows (Gerrit springs to mind) where you can let the CI server work on your code without breaking anything else, but even then there's a cost to the context switch when a test failure means you have to return to a piece of code you thought you'd finished.
    - Oompa 15y ago
      Then they need to be moved to a different place, at least. Tell your CI server to use them, but more them out of the way for normal developers.
    - jrockway 15y ago
      Completely false. You need to know that your change doesn't break the build, and if you wait a long time, you will have mentally switched gears when informed of the breakage. This is a big productivity sink. (What's worse is when changes come in faster than the CI system can run the tests. Then you don't know which change broke your build because many changes were tested at once.) Anything longer than 10 seconds is too long, in my opinion. As soon as you can get up to get coffee while your tests are running, you've lost a lot of productivity. (I recently finished a project where the tests took about ten minutes to run. That meant I could only change code 50 times per day. If the tests had taken 10 seconds, I would have been able to be make 300 changes per day. That's a 6x productivity increase right there. Fast running test suites are absolutely essential.
pbiggar 15y ago
I strongly feel you should try to add one test in each category. That adds a sanity check and lowers to cost to adding more tests when you really need it. It's pretty painful to think "oh, this really needs a test, but I haven't got a test suite set up and besides, I don't know to write a test of this kind". Writing tests for edge cases we see in production is the most valuable thing we do. We use Airbrake to find the bugs, and then we add a test for it, if possible (it's not always possible). That gives us good confidence that other changes aren't fucking things up. It's also a pretty sane strategy for growing a test suite when you inevitably have some portion of your code which has no tests at all.
pbiggar 15y ago
> - Testing frameworks and "best practices" change way faster than language frameworks and I simply can't keep up. What rspec version do I use with what version of Rails? Now I have to use Cucumber? I learned some Cucumber ... oh, now Steak is better. [rage comic goes here] I think this is only in the Rails community, where all new things is quick to be pronounced "the new right way to do things", not just in testing. > I deal with custom domains, authentication, and 3rd party API calls in almost every app we have. IMO, this adds 20% or more to the (already high) testing overhead just to get these things configured right in the test suite We do the same (tests for interaction with EC2, Github, and a few other providers). It is more expensive, but we find it more worthwhile too. Normally, 3rd party APIs are insufficiently specified, especially for error conditions. So when we have a failure in production, we can easily add tests to make sure we handle that edge case in future.
harel 15y ago
I don't want to convince you - just strengthen your point.
smsm42 15y ago
My experience shows that tests is not very useful in protecting from "hard" mistaken (like unusual combination of inputs, missing condition branch coverage, etc) because even with 100% code coverage you don't actually cover 100% of input/state combinations. And things you didn't think in development are usually things you didn't think in tests too. Tests are, however, always been amazingly helpful for me in: 1. Protecting me from stupid mistakes like using wrong variable in parameters, etc. (yes, it is embarrassing to have something like this, but I better be embarrassed by test and fix it before anybody seen it than be embarrassed by somebody else hitting it when using my code). 2. Ensuring refactoring and adding new things didn't break anything. 3. After "hard" bug has been found, ensuring it never reoccurs again. As for dealing with authentication, etc. - that's what unit tests are for, testing stuff that is under these layers directly. And I don't see it matters what you are using for tests - almost any framework would do fine, it's having tests that matters, not how you run them. I think you can unit-test javascript too, though I never had to deal with it myself so I don't know how.
- Produce 15y ago
  I think that the main advantage of unit testing is that you have to write testable, modular code. It ensures a sound design, which is the cheapest phase to catch bugs. The regression proofing is not a particularly big advantage of unit testing since functional and integration tests catch more bugs anyway.
lucian1900 15y ago
I've found tests very useful for refactoring. I can pretty much go wild, as long as the tests pass at the end. About bugs in production, after you find a bug write a test that exercises that bug. Then make the test pass. That way, you're unlikely to ever have a regression on that bug. For browser-side UI tests, selenium is very useful.

sunir 15y ago

People write and play with test frameworks because they are procrastinating from writing actual tests. Think about it. Just use Test::Unit and move on with your life. Write some tests. That's what counts.

eric-hu 15y ago
I'm working on a single page app that's about 30% Rails and 70% Coffeescript/Backbone.js. Test::Unit is practically useless for us since users never hit any plain HTML pages besides the login page. Imagine the current HTML5 Pandora having bugs with one particular song in Chrome only. How do you test for that using Test::Unit?
- mattbriggs 15y ago
  there are a bajillion libraries out there for unit testing javascript (something test::unit-ish would be qunit). rails + backbone + JST style templating actually makes TDDing your js relatively painless
  - eric-hu 15y ago
    We're looking into integration-level testing for our Javascript since it'd give us full coverage of the bugs we see. I forgot to mention we also use Socket.io for real time push updates to user Backbone models. A good number of bugs don't crop up until another user modifies data and those changes get pushed out to another person.
- sunir 15y ago
  You don't. You use QUnit and move on with your life.
- Oompa 15y ago
  Testing with front end javascript is more difficult due to different interpreters, much more variation in environment, etc. That doesn't mean you can't have tests for your core business logic at least in something like Vows or Mocha.

jakejake 15y ago

My approach to testing is not to be obsessed with the latest, greatest framework or 100% code coverage. I try to start with just one or two tests to actually help do things that are tedious or require multiple steps. It takes some time to automate a good test but once you do it immediately starts saving time because you don't have to run the same sequence a thousand times while developing. You can think of it more like a macro that saves you time. Once you write the main test it's easy then to run it with all combinations of good and bad input. By doing that you'll often wind up hitting a pretty good percentage of your code. Then as bugs are discovered due to unexpected input you can just keeping adding more input situations.

gbog 15y ago
Testing is like jumping in cool water, one find so many reason to not do it beforewards, but one is so happy to have done it afterwards...

netzpirat 15y ago

At first I was doubtful about testing my JS code, but nowadays I do enjoy it much more that testing the Rails backend. I use my own gem guard-jasmine that runs the specs headless on PhantomJS and it's a real joy! My whole spec suite with over 1000 specs runs in under 3 seconds. I use SinonJS for faking AJAX calls to the backend, but that's just a small subset of all specs since most stuff isn't interacting with the backend.

diminoten 15y ago

How do you feel about regression testing? Maybe instead of "writing" tests for potential bugs, you write tests for bugs you've found already.

3am 15y ago
Regression testing is when you check that new code doesn't break existing functionality. It's preventative, not reactive.

nkassis 15y ago

"- Most bugs/edge cases I encounter in our production apps are things I'd never think to write a test for ..." I feel that way often too but I write test more as a specification for how I want the code to work then as a catch all bugs thing. "- I deal with custom domains, authentication, and 3rd party API calls in almost every app we have. IMO, this adds 20% or more to the (already high) testing overhead just to get these things configured right in the test suite - More code is moving to front-end Javascript stuff ... so, now I have to write Rails tests AND JS tests? Sounds delightful" I feel your pain, I code stuff that use WebGL currently and I find it hard to test that stuff.

tikhonj 15y ago

You should check out QuickCheck for catching edge cases you did not think of. The idea behind QuickCheck is simple--you specify invariants in your code (called "properties") and the framework tests them with random inputs. This tool is very widely used in Haskell, but it's been ported to a whole bunch of other languages and could make your testing more thorough. In Haskell it's also easy to use and more fun than normal tests, but I don't know what it would be like in a different language.

ef4 15y ago

Look at it this way: you must be testing code as you write it anyway. There's really no other sane way to do it. You make a change, you load the page and see that your change worked, or you call your new function from an interactive interpreter. Smart automated testing just takes all that extra test work you're already doing and saves it as you go along. No need to try to invent extra things to test. You just test what you would have tested anyway by hand.

Iv 15y ago
You think like that until you hit your first serious regressions and discover it has been in the code base for several months and that the person responsible for it has left. I used to work at a company where automated testing was sending you emails about what your commits broke. It does help at improving code quality.
harel 15y ago
But then you'd still do the manual test after you complete your code. Nobody (I hope) codes blind hoping it would work or caught later by a test suite. Test suits don't reveal everything. Only what you tested for.
- fragsworth 15y ago
  The process goes more like this: 1. Write some code 2. Test it 3. Debug your code 4. Test it 5. Debug your code 6. Test it ... 39. Debug your code 40. Test it; now it finally seems to work If all your "test it" steps are being done manually, you're being very inefficient. A good unit test can actually make development go by faster with the added bonus of defending your code from changes down the road that might screw things up.
  - niccl 15y ago
    That's all fine unless you're exploring a solution space. Then the overhead of writing tests which are thrown away _in entirety_ is outrageous. AFAIK TDD really only works if you're either a) prepared to waste a huge amount of time writing tests which will later be completely redundant or b) working to a very clear set of requirements with tools and frameworks you already know intimately. </Rant>

joske2 15y ago

The point of testing/TDD for me is not (just) about preventing bugs, it is more about having quick feedback. Running a test is faster than waiting until it is deployed and manually clicking around in an application. It is kind of comparable to using a REPL.

cr4zy 15y ago

"- Most bugs/edge cases I encounter in our production apps are things I'd never think to write a test for ..." This is why regression tests are my favorite type of test. The need for the test has been confirmed by real world usage and once you create the regression test to fail, fix the bug, and pass the test, you won't have to ever worry about users seeing that bug again :)

MartinCron 15y ago

but I don't ever see myself in the "test ALL the things" camp Good for you. Extremists on all sides are usually wrong. Shoot for "test MOST OF the things" or "test the MOST IMPORTANT things" or even "test just enough things so that you know if change Y totally breaks MOST IMPORTANT feature Z".

Oompa 15y ago
I'm pretty far into the extremist side of TDD, and I'll say that there is a thing as TOO many tests. Your test suite needs to run fast to be really useful. If you have thousands of full stack integration tests that takes an hour to run, you're not going to run them as often as you should be, if at all, ad might as well delete them.
- MartinCron 15y ago
  If they are run by a continuous integration server on check in, it kind of doesn't matter how long they take to run.
  - andrewaylett 15y ago
    That's not strictly true: you'd still like to get reasonably quick feedback, especially if you're trying to make a release. It's also nice to be reasonably confident that your commit won't break the build, for which you should probably run a good chunk of the tests before committing. There are some workflows (Gerrit springs to mind) where you can let the CI server work on your code without breaking anything else, but even then there's a cost to the context switch when a test failure means you have to return to a piece of code you thought you'd finished.
  - Oompa 15y ago
    Then they need to be moved to a different place, at least. Tell your CI server to use them, but more them out of the way for normal developers.
  - jrockway 15y ago
    Completely false. You need to know that your change doesn't break the build, and if you wait a long time, you will have mentally switched gears when informed of the breakage. This is a big productivity sink. (What's worse is when changes come in faster than the CI system can run the tests. Then you don't know which change broke your build because many changes were tested at once.) Anything longer than 10 seconds is too long, in my opinion. As soon as you can get up to get coffee while your tests are running, you've lost a lot of productivity. (I recently finished a project where the tests took about ten minutes to run. That meant I could only change code 50 times per day. If the tests had taken 10 seconds, I would have been able to be make 300 changes per day. That's a 6x productivity increase right there. Fast running test suites are absolutely essential.

pbiggar 15y ago

I strongly feel you should try to add one test in each category. That adds a sanity check and lowers to cost to adding more tests when you really need it. It's pretty painful to think "oh, this really needs a test, but I haven't got a test suite set up and besides, I don't know to write a test of this kind". Writing tests for edge cases we see in production is the most valuable thing we do. We use Airbrake to find the bugs, and then we add a test for it, if possible (it's not always possible). That gives us good confidence that other changes aren't fucking things up. It's also a pretty sane strategy for growing a test suite when you inevitably have some portion of your code which has no tests at all.

pbiggar 15y ago

> - Testing frameworks and "best practices" change way faster than language frameworks and I simply can't keep up. What rspec version do I use with what version of Rails? Now I have to use Cucumber? I learned some Cucumber ... oh, now Steak is better. [rage comic goes here] I think this is only in the Rails community, where all new things is quick to be pronounced "the new right way to do things", not just in testing. > I deal with custom domains, authentication, and 3rd party API calls in almost every app we have. IMO, this adds 20% or more to the (already high) testing overhead just to get these things configured right in the test suite We do the same (tests for interaction with EC2, Github, and a few other providers). It is more expensive, but we find it more worthwhile too. Normally, 3rd party APIs are insufficiently specified, especially for error conditions. So when we have a failure in production, we can easily add tests to make sure we handle that edge case in future.

harel 15y ago

I don't want to convince you - just strengthen your point.

smsm42 15y ago

My experience shows that tests is not very useful in protecting from "hard" mistaken (like unusual combination of inputs, missing condition branch coverage, etc) because even with 100% code coverage you don't actually cover 100% of input/state combinations. And things you didn't think in development are usually things you didn't think in tests too. Tests are, however, always been amazingly helpful for me in: 1. Protecting me from stupid mistakes like using wrong variable in parameters, etc. (yes, it is embarrassing to have something like this, but I better be embarrassed by test and fix it before anybody seen it than be embarrassed by somebody else hitting it when using my code). 2. Ensuring refactoring and adding new things didn't break anything. 3. After "hard" bug has been found, ensuring it never reoccurs again. As for dealing with authentication, etc. - that's what unit tests are for, testing stuff that is under these layers directly. And I don't see it matters what you are using for tests - almost any framework would do fine, it's having tests that matters, not how you run them. I think you can unit-test javascript too, though I never had to deal with it myself so I don't know how.

Produce 15y ago
I think that the main advantage of unit testing is that you have to write testable, modular code. It ensures a sound design, which is the cheapest phase to catch bugs. The regression proofing is not a particularly big advantage of unit testing since functional and integration tests catch more bugs anyway.

lucian1900 15y ago

I've found tests very useful for refactoring. I can pretty much go wild, as long as the tests pass at the end. About bugs in production, after you find a bug write a test that exercises that bug. Then make the test pass. That way, you're unlikely to ever have a regression on that bug. For browser-side UI tests, selenium is very useful.