6 ms·

In my experience unit tests don't catch many bugs. On other hands end-to-end tests always surprise me with the bugs they catch. Based on that, I'm not sure advi

by msoad 11y ago

In my experience unit tests don't catch many bugs. On other hands end-to-end tests always surprise me with the bugs they catch. Based on that, I'm not sure advising for more stubbing is a good advise. Maybe I'm wrong but that was my experience building UI apps.

DannoHung 11y ago
In my experience building backend apps, unit tests are way more effective at catching logic bugs than end to end tests. End to end tests are great for catching connectivity bugs though.
- reverius42 11y ago
  UIs tend to have a ton of connectivity and little logic/computation.
- michaelmcmillan 11y ago
  An added benefit of using unit tests over integration tests is that unit tests shows you what went wrong very clearly. If an integration test fails I have literally no idea where to start hunting for the bug. It could be located at any level in the stack. I highly recommend this brilliant talk by J.B Rainsberger on how integration tests are a scam: https://vimeo.com/80533536 https://vimeo.com/80533536
  - specialk 11y ago
    If you don't have that much time to write tests I'd recommend integration tests. Then at least you have some tests. Treat integration tests as smoke tests. If there is smoke then you can start looking for the fire. Integration tests are always better then no tests at all. At least they tell you there is a fire even if it is vague in telling you where it is.
    - michaelmcmillan 11y ago
      As someone who always writes unit tests before any production code, I don't know how I would have the time to not write tests. It drives the design and reduces time debugging by a ridiculous amount. I work with information systems though, might be different for other fields.
      - lmm 11y ago
        Since I've switched to a language with a richer type system (Scala) I've found a lot of what I used to do with unit tests can go into the type level instead (see e.g. http://spin.atomicobject.com/2014/12/09/typed-language-tdd-part1/ http://spin.atomicobject.com/2014/12/09/typed-language-tdd-p... - but in Scala the techniques are a lot less verbose to apply).
        michaelmcmillan 11y ago
        I am most familiar with Javascript & Python, but I think you're onto something having fiddled with Java for 6 months or so. Some logic obviously can't be replaced with only a rich type system, but a lot can.
noselasd 11y ago
Though you also have to ask how many additional bugs you'd have if there were no unit tests.
juliangregorian 11y ago
Doesn't sound like you're unit testing well then. I'd take good unit tests over integration tests any day. Integration tests are inherently flaky.
- kansface 11y ago
  Your sentiment is strange to me - I've only ever seen unit tests as absolutely worthless. The reason integration tests are flaky is exactly the same reason they are valuable.
  - hinkley 11y ago
    Someone who checks in broken code will also take flaky tests as an excuse not to look at the build results, or to convince themselves that the failure is caused by flakiness and not their changes. If I had a dollar for every time I asked someone why they weren't fixing their shit, and they answered "Oh, I thought the build was failing randomly again." I'd have a lot of dollars.
    - phpnode 11y ago
      The fact that your tests are flaky at all means that the integration tests have revealed a bug that you need to fix. The tests have done their job. The fact that people aren't looking at the results speaks to a larger cultural issue.
      - hinkley 11y ago
        Not if it's a timing error in the test harness. That's usually what people mean when they say flaky tests. Selenium, especially three or four years ago, would simply fail occasionally no matter how good your tests are. Yes, there are bugs that look like bad luck, but are really flaws in your logic. I have fixed a lot of bugs that look Random, but eventually you're left with, for instance, selenium just refusing to click a button.
- sheepmullet 11y ago
tunesmith 11y ago
For me, the value of writing unit tests is in the writing of them more than the running of them.
- amyjess 11y ago
  Alternately, the reading of them as well. Reading the unit tests can help you understand an unfamiliar project by acting as documentation, telling you what is expected behavior.
  - curun1r 11y ago
    Yeah...for me, unit tests are the gift that you give to the developer who will be maintaining and improving the code 6 months from the time you write it. Beyond documentation, it gives him/her the ability to refactor the code without fear of it breaking horribly and tells that developer which behavior of the application is intentional and which was accidental. And before you say, "Screw 'em...I've got a code freeze to meet," consider that the person maintaining the code 6 months from now is probably you.
    - madeofpalk 11y ago
      IMO, this all relies on having good unit tests. If you have bad application code that's hard to read, what's to say that the tests will be any better?
      - tunesmith 11y ago
        How would you have good unit tests if you have bad application code? It's writing the tests that forces you to improve the application code, because otherwise you won't find it straightforward to write the tests.
- deleted 11y ago
  [deleted]
EugeneOZ 11y ago
Try to add fuzz-testing into your unit tests.
bliti 11y ago
That's why you don't stop with unit tests. You make sure to automate testing of ever layer of the application. It does require a bit more effort, but it's it's a savior when the UI breaks, and it's not something obvious yet breaks the user experience.
- jononor 11y ago
  I prefer to write end-to-end tests first, then fill in integration and unit-tests on a as-needed basis. If I am finding it complicated to write, or if experience from production says it is a problematic area. After all, the important thing is whether your application/service/library does what it promises to its users/consumers. Whether some function/class deep inside works or not is mostly irrelevant (can help in fixing the issue faster though). Unit-testing-first has a tendency to make the tests very fine-grained, often need changes when you refactor internals (how do you know test did not break?), and needlessly keeping rigid internal details that one should have the freedom to change. Have seen plenty of code being kept around because it had lots of tests - when it was not actually used for anything at all! Note: Above approach is coupled with a pretty strict adherence to keeping projects small and independent, preferably <10kLOC.
Confusion 11y ago
The distribution of when bugs are caught, and what bugs are caught, by different kinds of tests is very different. Units tests catch bugs while they are being written and when the units they test are being refactored. The bugs they catch are trivial, silly, 'would have been caught anyway'. Outside of those activities, they rarely catch bugs. On the other hand, integration tests more consistently catch bugs and catch bugs you didn't foresee, have to investigate before understanding. You don't remember the last time a unit test caught a bug, because on average it is much longer ago and if it was a test for a unit you wrote this morning, you usually blame it on 'work in progress' and think you would have caught that trivial bug anyway. I think the nature of units tests make it easy to underappreciate them.
- Glide 11y ago
  This is too true. A team I was on started seeing way more value in integration tests and ended up writing way more integration tests than unit tests. What ended up happening is that the code quality ended up worse because they stopped running tests very often because there were too many integration tests and they took too long to run. Sad part is, a lot of those tests would have made a lot more sense as unit tests.
LoSboccacc 11y ago
my experience is like yours that unit test developed in development are rarely useful. however, having a framework in place helps a lot when working at user submitted issues: those are the issue one should strive to replicate trough unit test. In a six month of production, you get a fairly robust suites of test on all corner cases undiscovered in development.
ed_blackburn 11y ago
Admittedly not in node, but I've found that writing end-to-end tests but substituting different 'units' for in-memory or not depending on the test marries the best of both worlds. In-memory tests tend to focus on behaviour. In the rush to shoe-horn tests into a taxonomy, or 'integration' vs 'acceptances' vs 'unit' we often loose sight of the actual value proposition. Assisting design and building confidence to make changes.
taco_emoji 11y ago
You can use both things...
danieltillett 11y ago
It depends a bit on the problem you are trying to solve and the structure of your code. I have projects where only end-to-end tests find bugs and others where unit tests are helpful. What is never helpful is imposing some sort of semi-religious "thou shalt do x" in every case.
josteink 11y ago
> In my experience unit tests don't catch many bugs. On other hands end-to-end tests always surprise me with the bugs they catch. Obviously if you've walked the walk with unit-tests and done proper TDD and all that, you've caught the bugs before submitting your code. Ofcourse the remaining bugs (which there will be plenty of!) will be found in your end-to-end testing where all the bits are tied together. This is only natural and does is no way suggest unit-testing was a wasted effort.
- sheepmullet 11y ago
  "Obviously if you've walked the walk with unit-tests and done proper TDD and all that, you've caught the bugs before submitting your code." Unit testing and TDD are quite separate things. Perhaps you should try writing unit tests after you write your code and then you can come back and add value to the conversation.
  - josteink 11y ago
    > Perhaps you should try writing unit tests after you write your code and then you can come back and add value to the conversation. Nice snark there. Feel free to keep it to yourself. To add actual value to the conversation (as opposed to your contribution), I can very much recommend the book "Working Effectively with Legacy Code"[1] for how to handle unit-testing in the scenario of existing "legacy" code-bases. It's full of useful tips and methods to get testing in place "anywhere" and has a pragmatical (as opposed religious) approach to getting it done. To spark some interest: The book defines "legacy code" as any code not covered by unit-tests. It may be seem dated (from 2004 and all), but it's been the most useful book I've read on unit-testing by far. [1] http://www.amazon.com/gp/product/0131177052/ref=as_li_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=0131177052&linkCode=as2&tag=jostekjoni-20&linkId=XZ6RUGMKOUPE4HTQ http://www.amazon.com/gp/product/0131177052/ref=as_li_tl?ie=...
    - sheepmullet 11y ago
      You came along and dismissed the parents views without any real thought. If you write the tests afterwards you can better gauge how well they catch bugs. It sounds like the parent has tried this and has found them pretty useless. I've personally found that unit tests are not worthwhile for many components, and yet are critical for others.

by msoad 11y ago

DannoHung 11y ago

In my experience building backend apps, unit tests are way more effective at catching logic bugs than end to end tests. End to end tests are great for catching connectivity bugs though.

reverius42 11y ago
UIs tend to have a ton of connectivity and little logic/computation.
michaelmcmillan 11y ago
An added benefit of using unit tests over integration tests is that unit tests shows you what went wrong very clearly. If an integration test fails I have literally no idea where to start hunting for the bug. It could be located at any level in the stack. I highly recommend this brilliant talk by J.B Rainsberger on how integration tests are a scam: https://vimeo.com/80533536 https://vimeo.com/80533536
- specialk 11y ago
  If you don't have that much time to write tests I'd recommend integration tests. Then at least you have some tests. Treat integration tests as smoke tests. If there is smoke then you can start looking for the fire. Integration tests are always better then no tests at all. At least they tell you there is a fire even if it is vague in telling you where it is.
  - michaelmcmillan 11y ago
    As someone who always writes unit tests before any production code, I don't know how I would have the time to not write tests. It drives the design and reduces time debugging by a ridiculous amount. I work with information systems though, might be different for other fields.
    - lmm 11y ago
      Since I've switched to a language with a richer type system (Scala) I've found a lot of what I used to do with unit tests can go into the type level instead (see e.g. http://spin.atomicobject.com/2014/12/09/typed-language-tdd-part1/ http://spin.atomicobject.com/2014/12/09/typed-language-tdd-p... - but in Scala the techniques are a lot less verbose to apply).
      - michaelmcmillan 11y ago
        I am most familiar with Javascript & Python, but I think you're onto something having fiddled with Java for 6 months or so. Some logic obviously can't be replaced with only a rich type system, but a lot can.

noselasd 11y ago

Though you also have to ask how many additional bugs you'd have if there were no unit tests.

juliangregorian 11y ago

Doesn't sound like you're unit testing well then. I'd take good unit tests over integration tests any day. Integration tests are inherently flaky.

kansface 11y ago
Your sentiment is strange to me - I've only ever seen unit tests as absolutely worthless. The reason integration tests are flaky is exactly the same reason they are valuable.
- hinkley 11y ago
  Someone who checks in broken code will also take flaky tests as an excuse not to look at the build results, or to convince themselves that the failure is caused by flakiness and not their changes. If I had a dollar for every time I asked someone why they weren't fixing their shit, and they answered "Oh, I thought the build was failing randomly again." I'd have a lot of dollars.
  - phpnode 11y ago
    The fact that your tests are flaky at all means that the integration tests have revealed a bug that you need to fix. The tests have done their job. The fact that people aren't looking at the results speaks to a larger cultural issue.
    - hinkley 11y ago
      Not if it's a timing error in the test harness. That's usually what people mean when they say flaky tests. Selenium, especially three or four years ago, would simply fail occasionally no matter how good your tests are. Yes, there are bugs that look like bad luck, but are really flaws in your logic. I have fixed a lot of bugs that look Random, but eventually you're left with, for instance, selenium just refusing to click a button.
sheepmullet 11y ago

tunesmith 11y ago

For me, the value of writing unit tests is in the writing of them more than the running of them.

amyjess 11y ago
Alternately, the reading of them as well. Reading the unit tests can help you understand an unfamiliar project by acting as documentation, telling you what is expected behavior.
- curun1r 11y ago
  Yeah...for me, unit tests are the gift that you give to the developer who will be maintaining and improving the code 6 months from the time you write it. Beyond documentation, it gives him/her the ability to refactor the code without fear of it breaking horribly and tells that developer which behavior of the application is intentional and which was accidental. And before you say, "Screw 'em...I've got a code freeze to meet," consider that the person maintaining the code 6 months from now is probably you.
  - madeofpalk 11y ago
    IMO, this all relies on having good unit tests. If you have bad application code that's hard to read, what's to say that the tests will be any better?
    - tunesmith 11y ago
      How would you have good unit tests if you have bad application code? It's writing the tests that forces you to improve the application code, because otherwise you won't find it straightforward to write the tests.
deleted 11y ago
[deleted]

EugeneOZ 11y ago

Try to add fuzz-testing into your unit tests.

bliti 11y ago

That's why you don't stop with unit tests. You make sure to automate testing of ever layer of the application. It does require a bit more effort, but it's it's a savior when the UI breaks, and it's not something obvious yet breaks the user experience.

jononor 11y ago
I prefer to write end-to-end tests first, then fill in integration and unit-tests on a as-needed basis. If I am finding it complicated to write, or if experience from production says it is a problematic area. After all, the important thing is whether your application/service/library does what it promises to its users/consumers. Whether some function/class deep inside works or not is mostly irrelevant (can help in fixing the issue faster though). Unit-testing-first has a tendency to make the tests very fine-grained, often need changes when you refactor internals (how do you know test did not break?), and needlessly keeping rigid internal details that one should have the freedom to change. Have seen plenty of code being kept around because it had lots of tests - when it was not actually used for anything at all! Note: Above approach is coupled with a pretty strict adherence to keeping projects small and independent, preferably <10kLOC.

Confusion 11y ago

The distribution of when bugs are caught, and what bugs are caught, by different kinds of tests is very different. Units tests catch bugs while they are being written and when the units they test are being refactored. The bugs they catch are trivial, silly, 'would have been caught anyway'. Outside of those activities, they rarely catch bugs. On the other hand, integration tests more consistently catch bugs and catch bugs you didn't foresee, have to investigate before understanding. You don't remember the last time a unit test caught a bug, because on average it is much longer ago and if it was a test for a unit you wrote this morning, you usually blame it on 'work in progress' and think you would have caught that trivial bug anyway. I think the nature of units tests make it easy to underappreciate them.

Glide 11y ago
This is too true. A team I was on started seeing way more value in integration tests and ended up writing way more integration tests than unit tests. What ended up happening is that the code quality ended up worse because they stopped running tests very often because there were too many integration tests and they took too long to run. Sad part is, a lot of those tests would have made a lot more sense as unit tests.

LoSboccacc 11y ago

my experience is like yours that unit test developed in development are rarely useful. however, having a framework in place helps a lot when working at user submitted issues: those are the issue one should strive to replicate trough unit test. In a six month of production, you get a fairly robust suites of test on all corner cases undiscovered in development.

ed_blackburn 11y ago

Admittedly not in node, but I've found that writing end-to-end tests but substituting different 'units' for in-memory or not depending on the test marries the best of both worlds. In-memory tests tend to focus on behaviour. In the rush to shoe-horn tests into a taxonomy, or 'integration' vs 'acceptances' vs 'unit' we often loose sight of the actual value proposition. Assisting design and building confidence to make changes.

taco_emoji 11y ago

You can use both things...

danieltillett 11y ago

It depends a bit on the problem you are trying to solve and the structure of your code. I have projects where only end-to-end tests find bugs and others where unit tests are helpful. What is never helpful is imposing some sort of semi-religious "thou shalt do x" in every case.

josteink 11y ago

> In my experience unit tests don't catch many bugs. On other hands end-to-end tests always surprise me with the bugs they catch. Obviously if you've walked the walk with unit-tests and done proper TDD and all that, you've caught the bugs before submitting your code. Ofcourse the remaining bugs (which there will be plenty of!) will be found in your end-to-end testing where all the bits are tied together. This is only natural and does is no way suggest unit-testing was a wasted effort.

sheepmullet 11y ago
"Obviously if you've walked the walk with unit-tests and done proper TDD and all that, you've caught the bugs before submitting your code." Unit testing and TDD are quite separate things. Perhaps you should try writing unit tests after you write your code and then you can come back and add value to the conversation.
- josteink 11y ago
  > Perhaps you should try writing unit tests after you write your code and then you can come back and add value to the conversation. Nice snark there. Feel free to keep it to yourself. To add actual value to the conversation (as opposed to your contribution), I can very much recommend the book "Working Effectively with Legacy Code"[1] for how to handle unit-testing in the scenario of existing "legacy" code-bases. It's full of useful tips and methods to get testing in place "anywhere" and has a pragmatical (as opposed religious) approach to getting it done. To spark some interest: The book defines "legacy code" as any code not covered by unit-tests. It may be seem dated (from 2004 and all), but it's been the most useful book I've read on unit-testing by far. [1] http://www.amazon.com/gp/product/0131177052/ref=as_li_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=0131177052&linkCode=as2&tag=jostekjoni-20&linkId=XZ6RUGMKOUPE4HTQ http://www.amazon.com/gp/product/0131177052/ref=as_li_tl?ie=...
  - sheepmullet 11y ago
    You came along and dismissed the parents views without any real thought. If you write the tests afterwards you can better gauge how well they catch bugs. It sounds like the parent has tried this and has found them pretty useless. I've personally found that unit tests are not worthwhile for many components, and yet are critical for others.