5 ms·

Favourite paragraph: "If you're a Java programmer and want to have a rude awakening, go download Jester. Jester is an automated mutation testing tool - it goes

Favourite paragraph: "If you're a Java programmer and want to have a rude awakening, go download Jester. Jester is an automated mutation testing tool - it goes in and replaces "<=" with "<", "&&" with "||", "!" with whitespace. And then it re-runs the tests. And then you get to watch in horror as your tests still all pass, regardless of what the product code actually does."

fleitz 16y ago
Everyone knows that the best way to program java is via a DSL called XML. The .java files are just an illusion.
njharman 16y ago
I don't understand at all. Why would the tests still pass?
- WorkerBee 16y ago
  Because the author wrote poor tests and thinks that you did too.
  - Quarrelsome 16y ago
    I think its more likely that the author _inherited_ some lame tests and this has coloured his opinion somewhat.
  - cabalamat 16y ago
    Indeed. The whole essay boils down to "bad unit tests are useless, therefore unit testing isn't worth bothering with".
    - dagw 16y ago
      More like "bad unit tests are useless, good and useful unit tests are hard and take a significant chunk of time, therefore weigh the costs vs the benefits before deciding whether to bother or not"
      - WorkerBee 16y ago
        Hm. Before deciding that good and useful unit tests are too expensive, spend a week or three doing live support of poorly reported and hard to reproduce issues where the numbers were slightly out or the result came back ten seconds later than it should, one time. that will change your perspective on what's hard and what's not. And possibly damage your will to live.
        cabalamat 16y ago
        Indeed. If you think good unit tests are time-consuming, try debugging code with lots of subtle bugs in it.
        dagw 16y ago
        If the bugs are that subtle what makes you think you're unit tests are comprehensive enough to catch them? The hardest bugs to debug are the types that show only on high latency networks and only when dealing with files bigger than 2GB and only after the app has been running for at least 50 hours. Those are also the type of bugs which are almost impossible to write unit tests for. I'm not arguing against unit tests, they definitely have a time and a place. I'm just not convinced that they are an easy and instant cure for all your hardest bugs.
        WorkerBee 16y ago
        Unit tests may leave you with remaining subtle bugs. This is better than having both subtle and obvious bugs. Obvious under test, e.g. "the class returns null when it should return object y", which can have subtle and easy-to-miss symptoms in a production system
- z92 16y ago
  That's the horror.
- metamemetics 16y ago
  Because they are arbitrary and not a one-to-one correspondence with what they purport to be testing. One false positive and everything could still be broken.
- scorpioxy 16y ago
  Never tried it. But its probably because unit tests can only tests things you explicitly test for and if you don't have your corner cases covered, then these changes wouldn't blow up your tests. Someone correct me if I'm wrong. Not a fan of unit testing my self. I feel the approach lacking.
- baddox 16y ago
  It shows that green bars on all your tests don't necessarily mean anything. All too often, developers don't test for the cases that may actually fail (because, obviously, their actual code is only correct for the cases they're cognizant of).
- jacquesm 16y ago
  Normally they really shouldn't, but if you have a problem writing test cases (which probably means you have a problem programming to begin with) then you may not be testing with enough variation in your input to detect corner cases, tests that should come up false etc. An example: int fun(int a, int b) { if (a >= b) { return 1; } return 0; } now you could test this by saying: fun(4,3) should give '1' as the output and fun(3,4) should give '0' as the output But you'd miss to test for equality, with fun(3,3) which should *also* give 1 as output So you'd need an extra test for the '=' part of the condition. Replacing '>=' with '>' will still pass the first two tests, so if you forgot about the third you will not even realize something is wrong until your code starts breaking in mysterious ways. Unit tests have to be just as good as the code you wrote.
  - njharman 16y ago
    Thanks. I just can't imagine enough people being so incompetent to not test the obvious three cases that this is an issue.
    - jacquesm 16y ago
      I'm not sure whether you're joking or not but I've seen code that was in production for many years with mistakes like that.
bh42 16y ago
So basically what he's saying is that super naive unit tests which barely even test the happy path are hardly better then nothing. I fully agree, but I am getting tired of all of this slam down style, flame-bait headlined, writing.
aarongough 16y ago
If you're a Ruby developer you should check out Heckle: http://glu.ttono.us/articles/2006/12/19/tormenting-your-tests-with-heckle http://glu.ttono.us/articles/2006/12/19/tormenting-your-test... Heckle is a mutation testing framework for Ruby. I don't use it all the time, but my tests are definitely better now that I've tried it and seen what some of the edge cases are...

fleitz 16y ago

Everyone knows that the best way to program java is via a DSL called XML. The .java files are just an illusion.

njharman 16y ago

I don't understand at all. Why would the tests still pass?

WorkerBee 16y ago
Because the author wrote poor tests and thinks that you did too.
- Quarrelsome 16y ago
  I think its more likely that the author _inherited_ some lame tests and this has coloured his opinion somewhat.
- cabalamat 16y ago
  Indeed. The whole essay boils down to "bad unit tests are useless, therefore unit testing isn't worth bothering with".
  - dagw 16y ago
    More like "bad unit tests are useless, good and useful unit tests are hard and take a significant chunk of time, therefore weigh the costs vs the benefits before deciding whether to bother or not"
    - WorkerBee 16y ago
      Hm. Before deciding that good and useful unit tests are too expensive, spend a week or three doing live support of poorly reported and hard to reproduce issues where the numbers were slightly out or the result came back ten seconds later than it should, one time. that will change your perspective on what's hard and what's not. And possibly damage your will to live.
      - cabalamat 16y ago
        Indeed. If you think good unit tests are time-consuming, try debugging code with lots of subtle bugs in it.
        dagw 16y ago
        If the bugs are that subtle what makes you think you're unit tests are comprehensive enough to catch them? The hardest bugs to debug are the types that show only on high latency networks and only when dealing with files bigger than 2GB and only after the app has been running for at least 50 hours. Those are also the type of bugs which are almost impossible to write unit tests for. I'm not arguing against unit tests, they definitely have a time and a place. I'm just not convinced that they are an easy and instant cure for all your hardest bugs.
        WorkerBee 16y ago
        Unit tests may leave you with remaining subtle bugs. This is better than having both subtle and obvious bugs. Obvious under test, e.g. "the class returns null when it should return object y", which can have subtle and easy-to-miss symptoms in a production system
z92 16y ago
That's the horror.
metamemetics 16y ago
Because they are arbitrary and not a one-to-one correspondence with what they purport to be testing. One false positive and everything could still be broken.
scorpioxy 16y ago
Never tried it. But its probably because unit tests can only tests things you explicitly test for and if you don't have your corner cases covered, then these changes wouldn't blow up your tests. Someone correct me if I'm wrong. Not a fan of unit testing my self. I feel the approach lacking.
baddox 16y ago
It shows that green bars on all your tests don't necessarily mean anything. All too often, developers don't test for the cases that may actually fail (because, obviously, their actual code is only correct for the cases they're cognizant of).
jacquesm 16y ago
Normally they really shouldn't, but if you have a problem writing test cases (which probably means you have a problem programming to begin with) then you may not be testing with enough variation in your input to detect corner cases, tests that should come up false etc. An example: int fun(int a, int b) { if (a >= b) { return 1; } return 0; } now you could test this by saying: fun(4,3) should give '1' as the output and fun(3,4) should give '0' as the output But you'd miss to test for equality, with fun(3,3) which should *also* give 1 as output So you'd need an extra test for the '=' part of the condition. Replacing '>=' with '>' will still pass the first two tests, so if you forgot about the third you will not even realize something is wrong until your code starts breaking in mysterious ways. Unit tests have to be just as good as the code you wrote.
- njharman 16y ago
  Thanks. I just can't imagine enough people being so incompetent to not test the obvious three cases that this is an issue.
  - jacquesm 16y ago
    I'm not sure whether you're joking or not but I've seen code that was in production for many years with mistakes like that.

bh42 16y ago

So basically what he's saying is that super naive unit tests which barely even test the happy path are hardly better then nothing. I fully agree, but I am getting tired of all of this slam down style, flame-bait headlined, writing.

aarongough 16y ago

If you're a Ruby developer you should check out Heckle: http://glu.ttono.us/articles/2006/12/19/tormenting-your-tests-with-heckle http://glu.ttono.us/articles/2006/12/19/tormenting-your-test... Heckle is a mutation testing framework for Ruby. I don't use it all the time, but my tests are definitely better now that I've tried it and seen what some of the edge cases are...