7 ms·
It's OK Not to Write Unit Tests
- johnswamps 16y agoOriginal discussion: http://news.ycombinator.com/item?id=922715 http://news.ycombinator.com/item?id=922715
- nopassrecover 16y agoFavourite paragraph: "If you're a Java programmer and want to have a rude awakening, go download Jester. Jester is an automated mutation testing tool - it goes in and replaces "<=" with "<", "&&" with "||", "!" with whitespace. And then it re-runs the tests. And then you get to watch in horror as your tests still all pass, regardless of what the product code actually does."
- fleitz 16y agoEveryone knows that the best way to program java is via a DSL called XML. The .java files are just an illusion.
- njharman 16y agoI don't understand at all. Why would the tests still pass?
- WorkerBee 16y agoBecause the author wrote poor tests and thinks that you did too.
- Quarrelsome 16y agoI think its more likely that the author _inherited_ some lame tests and this has coloured his opinion somewhat.
- cabalamat 16y agoIndeed. The whole essay boils down to "bad unit tests are useless, therefore unit testing isn't worth bothering with".
- dagw 16y agoMore like "bad unit tests are useless, good and useful unit tests are hard and take a significant chunk of time, therefore weigh the costs vs the benefits before deciding whether to bother or not"
- WorkerBee 16y agoHm. Before deciding that good and useful unit tests are too expensive, spend a week or three doing live support of poorly reported and hard to reproduce issues where the numbers were slightly out or the result came back ten seconds later than it should, one time. that will change your perspective on what's hard and what's not. And possibly damage your will to live.
- cabalamat 16y agoIndeed. If you think good unit tests are time-consuming, try debugging code with lots of subtle bugs in it.
- dagw 16y agoIf the bugs are that subtle what makes you think you're unit tests are comprehensive enough to catch them? The hardest bugs to debug are the types that show only on high latency networks and only when dealing with files bigger than 2GB and only after the app has been running for at least 50 hours. Those are also the type of bugs which are almost impossible to write unit tests for. I'm not arguing against unit tests, they definitely have a time and a place. I'm just not convinced that they are an easy and instant cure for all your hardest bugs.
- WorkerBee 16y ago
- z92 16y agoThat's the horror.
- metamemetics 16y agoBecause they are arbitrary and not a one-to-one correspondence with what they purport to be testing. One false positive and everything could still be broken.
- scorpioxy 16y agoNever tried it. But its probably because unit tests can only tests things you explicitly test for and if you don't have your corner cases covered, then these changes wouldn't blow up your tests. Someone correct me if I'm wrong. Not a fan of unit testing my self. I feel the approach lacking.
- baddox 16y agoIt shows that green bars on all your tests don't necessarily mean anything. All too often, developers don't test for the cases that may actually fail (because, obviously, their actual code is only correct for the cases they're cognizant of).
- jacquesm 16y agoNormally they really shouldn't, but if you have a problem writing test cases (which probably means you have a problem programming to begin with) then you may not be testing with enough variation in your input to detect corner cases, tests that should come up false etc. An example: int fun(int a, int b) { if (a >= b) { return 1; } return 0; } now you could test this by saying: fun(4,3) should give '1' as the output and fun(3,4) should give '0' as the output But you'd miss to test for equality, with fun(3,3) which should *also* give 1 as output So you'd need an extra test for the '=' part of the condition. Replacing '>=' with '>' will still pass the first two tests, so if you forgot about the third you will not even realize something is wrong until your code starts breaking in mysterious ways. Unit tests have to be just as good as the code you wrote.
- bh42 16y agoSo basically what he's saying is that super naive unit tests which barely even test the happy path are hardly better then nothing. I fully agree, but I am getting tired of all of this slam down style, flame-bait headlined, writing.
- aarongough 16y agoIf you're a Ruby developer you should check out Heckle: http://glu.ttono.us/articles/2006/12/19/tormenting-your-tests-with-heckle http://glu.ttono.us/articles/2006/12/19/tormenting-your-test... Heckle is a mutation testing framework for Ruby. I don't use it all the time, but my tests are definitely better now that I've tried it and seen what some of the edge cases are...
- gruseom 16y agoI don't see why a fanatical anti-rant is any better than a fanatical rant. There's this theory out there that Agile projects can refactor fearlessly because there's this immaculate suite of tests that can sound the alarm the second the smallest regression gets introduced. But anyone who's actually tried it knows that it's mostly just a fantasy. Poppycock. My project has an extensive test suite that we rely on to make major design changes with alacrity, just exactly the way that unit-test advocates advocate. We couldn't dream of doing what we're doing without it. It's true that not every module of every system responds equally well to this style of testing. It's true that putzing around revising test infrastructure for the nth time really sucks. But these are points that belong in a serious discussion, the kind that recognizes tradeoffs.
- douglasputnam 16y agocashto's post was a "serious" effort to disperse the cloud of dogma surrounding unit testing and TDD. I hardly think that lashing back with "poppycock" is driving the conversation in the direction of "serious discussion". cashto emphasized that unit testing is useful for teams with non-expert programmers (viz. his comment about the Dreyfus model of skill acquisition). Many of us are non-experts who benefit from working with the TDD training wheels on; it's O.K. to be less a Jedi All-Star hacker. But the phenomenal programmers, the men and women who were 10 times better than me, could not be bothered with TDD, even through the endorsed it for mere mortals. To paraphrase the TDD credo of “no test is worse than having no tests”, we can assert that blind faith is not better than no faith at all.
- pmccool 16y agoThe training wheels analogy was the weakest part of the article to me. TDD isn't a teaching tool, and the fact that they useless to people in a very specialized niche doesn't really say anything about their general value. I also notice reference to writing tests after the fact. I understand TDD to involve writing tests first. That's how I practice it anyway.
- ZeroGravitas 16y agoThat's just poisoning the well. Yes, X is fine if you're an idiot is exactly what you would say if you didn't want to have a conversation about something and would rather shut down discussion.
- deleted 16y ago[deleted]
- tetha 16y agoAh, when I first began to read this, I expected one of those rants against unit tests which don't leave any kinda good thing about tests and I mentally got ready to comment about that. However, I am quire pleased that this is a very sensible and mature position about unit testing (I think a short version would be: well, testing can help, it won't carry you and you have to stand on your own feet, but it can help), which is pretty close to my own position. Good Article.
- jwr 16y agoAs with all things, truth is somewhere in the middle. I've found that unit tests do help me write better code and find corner cases which I would otherwise miss. But I don't treat them as a religion, and I don't write them for every function. Just don't overdo either way and you'll be fine.
- arethuza 16y agoBut people really don't want that kind of sensible mature argument and would rather have set of "best practices" that they can blindly follow.
- abp 16y agoYes, but only if one got zero or not much experience in the field. So actually you are saying something that is true for everything.
- arethuza 16y agoYou'd be amazed at the number of experienced developers who tell me "it is best practice to do XYZ" who when challenged as to what is the underlying justification for this recommendation can't actually provide a sensible answer.
- crististm 16y agoI've tried TDD for one project and it gave me a nice feeling of having a safety net. It was a good feeling. The problems started to appear when the project increased in complexity and needed a redesign in some of it's parts. I had to do _again_ twice the work to have the tests match the code behavior. The article mentions this problem too. The benefit of the tests disappear when your code changes beyond refactoring.
- allend 16y agoThen why didn't you drop the tests if you think there was no benefit?
- philjackson 16y agoThis whole article could be boiled down to: "bad tests are useless".
- cawhitworth 16y agoSQLite would like to disagree with this rant, I think.
- alextgordon 16y agoThe article doesn't seem to be claiming that unit testing is bad, rather that religious promotion of it is bad, and the benefits and drawbacks should be weighted up like any other decision. Obviously SQLite has got a tremendous amount of value out of unit testing. But SQLite is quite abnormal in many respects. It's used in aviation, so the standards of testing are ridiculously high. It's also a library, and libraries tend to be very easy to test. It's also a long-term project with no deadline to speak of.
- humblepatience 16y agoThe way I read the article it sounded like he was downplaying the usefulness of even healthy amount of testing i.e. writing tests along with the code your writing. Certainly, all his arguments downplay TDD in general
- regularfry 16y ago"Many of them will not even survive the process at all." That's the point, and I think everything else in this article stems from it. Unit tests are supposed to be transient; you're supposed to be able to just ditch them and replace them when you ditch and replace the code they're testing. Functional tests, on the other hand, should be longer-lived.
- chmike 16y agoWith my students and used the analogy of unit tests (or asserts) with a fisher net to catch bugs. As a fisher net, it is made of a mesh that won't be able to catch small fishes. It may also have big holes in it here and there. So there is always a limit to its efficiency. It requires a significant effort to make unit tests more efficient. This effort must then be balanced with the potential costs of the bugs and across the application, because some part of the program are more critical than others. A good programmer should thus balance his effort in unit tests writing. I also suggest to write unit tests as needed to be more efficient. If the software is a data processing pipeline kind, simply write a very tight data validity checks at the end of the processing pipe and inject test data (including bogus one). If no problem ever show up with the test data, there is no point in writing unit tests for all intermediate steps of the data processing pipeline. If a problem is later discovered, then write the test, but only at the middle of the processing pipeline. You'll know if the problem is before or after. Then continue as needed. With this approach, you'll write unit tests only where it is needed and won't waste you time. Suggesting as the author that we don't need to write unit tests is like saying we don't need to put the car safety belt because people still die in car accident with it, and most of the time we don't have any accident even if we don't wear it. Is this a smart advise ? I don't think so. BTW, I use asserts wherever I can by reflex in my code because it saved me valuable time on many occasions.
- 10ren 16y agoAside on testing: I was surprised in re-reading Fred Brooks how cloose he sounds to unit-testing, in Mythical Man Month, 2ed, pp.147-148. He suggests testing components separately, before integrating, even though this requires lots of "scaffolding". Scaffolding includes dummy components, miniature file (minimal eg of a file format), "generators of test data, special analysis printouts, cross-reference table analyzers" (jigs and fixtures). "It is not unreasonable for there to be half as much code in scaffolding as there is in product". I end up writing a lot of the above kind of code - not as automated unit tests, but just as a way to make sure my code is doing what I think it's doing - just plain ol' testing. It can feel wasteful, so it was reassuring to hear Fred's take.
- deleted 16y ago[deleted]
- JoeAltmaier 16y agoHow about "buckshot testing"? I wrote a file system test that made every single API call with null arguments, good arguments, illegal arguments and then logged the error code. We ran this against every release of the OS and simply dif'd the result against the previous release run. It not only aided in bugfinding, it identified necessary release notes about API differences. The bigger point is, you have a computer at your disposal. It can make LOTS of calls in a short time. Try calling that function with every conceivable argument. Once you have a baseline, a diff tool can reveal everything.
- projectileboy 16y agoI'm not sure how valuable these articles - or threads - really are for anyone. Don't we have to balance cost vs. value for _anything_ we do? Find and replace "unit testing" with "documentation" in this article and we could be having the same conversation.
- j_baker 16y agoPersonally, if anyone asks me if they should unit test, I'll play the fanatic and take a hard line. Mostly because the way to learn the usefulness of unit tests is to write them. You quickly learn what they're good at and what they are bad at. Once you've written enough of them, you kind of seem to develop an instinct as to what unit tests are useful for.
- humblepatience 16y agoI wrote a response to this on my blog http://www.scottschulthess.com/coding/?p=143 http://www.scottschulthess.com/coding/?p=143 Basically, there are other benefits to TDD than just preventing regression. Documentation and ease of debugging while your writing code being the biggest ones. In a non-compiled language it can serve to verify that your at least runs without gross failure a little like the compiler does.