5 ms·

Great story, also looks like the guy really knows what hes doing. That said i read a post by him a while back where he talked about how useless unit tests are a

by kayoone 16y ago

Great story, also looks like the guy really knows what hes doing. That said i read a post by him a while back where he talked about how useless unit tests are and how they dont help finding bugs: http://abstractstuff.livejournal.com/60388.html http://abstractstuff.livejournal.com/60388.html I found that quite irritating from somebody who seems so experienced.

anamax 16y ago
> I found that quite irritating from somebody who seems so experienced. Okay, but is he wrong? Let's be specific. He says "[Unit testing] just doesn't cover a large enough class of potential problems to be generally useful." He said how he evaluates bug-finding tools and discussed his experience. What is yours?
- nostrademons 16y ago
  The dumb part about this debate is that it's being framed as all-or-nothing. The important question isn't "Are unit tests useful?" (the answer to that is "It depends..."), it's "Under which circumstances are unit tests useful?" If a particular unit test costs you more to write than it saves you in bugs, don't write it. And if it starts costing more in maintenance than it saves in confidence, get rid of it. There's a fair bit of cargo culting going on about how much to test - in my experience, it's not worth testing getters/setters, it's not worth testing simple "if this then set these variables" logic, and it's not worth unit testing database or external calls (you should have some integration tests for them though). It is very much worth testing tricky cases like a parser, or a bit-level decoder, or complicated algorithms. I just recently got up to 100% code coverage on a UTF-8 decoder I'd written. I found about a dozen bugs in the process. They were silly things, like leaving a digit off a hex constant, but they rippled throughout the decoder so that the results were completely wrong. If I'd just included that in another library without testing, it'd make the dependent library subtly wrong too, and I'd never hear about it until users started complaining, and they'd waste hours trying to find bugs in their code that are really bugs in mine. I'll also mention one other thing: the point of unit testing isn't to find bugs, it's to give you enough confidence in your code that you can freely change it. The real cost of bugs isn't that one erroneous event that customers will probably never notice, it's that the fear of that one unpredictable event makes management (which is sometimes you, if it's your startup) say "It works. Don't touch it, because you'll break it" and never improve the software. That's how you get shitty software that you can't fix.
  - petenixey 16y ago
    This is a very good point: are they useful? I think it's fair to say that integration tests (make sure that the functionality works) are useful; period. Not knowing whether your app is going to work when you push it is not fun. From a maintenance perspective and assuming that a problem will show up in integration testing or via a bug report, further testing is only really useful to the extent that it lets you debug a failure faster. However that acceleration is a tradeoff against the time taken to write the tests in the first place - ergo: is it really worth it? I personally don't think the real win from tests is in saving debugging time. I have found it's in making it quicker to write code. I know that sounds crazy as most people view tests as overhead but I view them as development tools. The key to them being useful though is writing them before you write the code. If you write them afterwards you will have already verified the code in an ad-hoc fashion and the tests are pure overhead. If you write them before they do the verification for you. I'm coding in Rails and I've found the outside-in approach to development: write the six line Cucumber integration test and then write the RSpec expectations for the object; act as a verified to-do list and make my development far more efficient and robust. I'm sure that other developers with better focus than I might not get the same benefit but I find it tremendously useful to immediately list down all of the edge cases and general behaviour that a piece of code should exhibit and then keep coding till growl glows green. I trade the time that I'd otherwise spend faffing around in a browser or a debugger for writing tests instead. It's a bonus that all of those tests then ensure ongoing integrity, the real win is that tests stop me from wandering around the trail and keep me knocking things off in the order that they need to be done. [Caveat: it has taken me months and months to come up to speed on how to write tests well] (great article btw Bill)
- deleted 16y ago
  [deleted]
  - justin 16y ago
    FWIW: With Socialcam, Eric implemented unit tests for the API (which powers the client apps), and initially for the website but later we decided against them for the site. Having them on the API definitely has been worth the initial cost as the backend has needed to be very stable for client development, and they are easily testable. Tim, perhaps you should have made your point more vigorously when you were here :). That said, I think that many / most of our projects fall into the bus count 1, cowboy coder hacking (at least, initially), so we've been able to move fast by not having them.
gridspy 16y ago
Speaking from personal experience, there are a lot of fads in software development. From the most recent language to the latest and greatest library. From design patterns through to work practices like Agile and TDD. You must be critical in your thinking for each one of these. Just because it works for someone else doesn't mean it will work for you. Personally, I think that most design patterns are used as an excuse to make a simple system complex. I think that unit tests are more painful to maintain than they are worth (in many cases). I think that new technology should be approached carefully and only proven tech with minimum lockin should be used. Don't go round saying that "experienced developers always do X" because a real experienced developer knows that X is great for some things and a huge waste of time for others.
nahname 16y ago
He really does not come off well in those comments. Repeatedly saying I am successful, therefore what I am doing is right. In his defense, it sounds like he has mostly worked by himself. Some practices don't have a very good ROI when you are just hacking away by yourself. I would argue that unit testing is one of them, if you are not very experienced with it anyways.
- gridspy 16y ago
  It sounds to me like he got beaten up in his own comments section. The people commenting don't seem to realize that he is talking about a different type of programming than the commenters are doing.
  - nahname 16y ago
    What do you mean by "type of programming"?
    - gridspy 16y ago
      I probably should have said "Problem Domain" I.e Are you an author of a library, a browser, a startup, experimental code, rapidly evolving code, space shuttle code ? Each one has different measurements of "success" and these can effect your choices re Unit Tests.
- deleted 16y ago
  [deleted]
kenjackson 16y ago
Are there any "hacker gods" that are really big on unit testing or use TDD? Most of the recognizable names that I see talk about it are agile book writers. Not actual devs. It was Paul Bucheit who says (from that link), "I'll make a somewhat heretical claim even: Unit test are 20% useful engineering, and 80% fad. They are great for tricky code with well defined behavior (like a parser), but wasteful for most code." Does Sweeney, Carmack, Cutler, Joy, Thompson, Knuth, Page, Steele, Zuckerburg, Anders, etc... use unit tests in a big way? I'm actually a fan of unit tests, although I generally dislike TDD.
- abstractbill 16y ago
  Nobody in Peter Seibel's "Coders At Work" seemed to have very much time for unit testing as far as I remember, and they pretty much all qualify for that title in my opinion. I'd stand by pretty much everything I wrote in my original post, for what it's worth, but I guess I don't really understand why it seemed to ruffle so many feathers to be honest. Unit testing doesn't help me enough for the work I have to put in (compared to things like good logging and monitoring, for example, unless I'm writing something like a parser, with well-specified behavior). It doesn't seem to have helped justin.tv too much either, when we've tried it. But if it helps someone else, then good for them!
  - hello_moto 16y ago
    I agree with you: if it works for someone else, good for them. Definitely. I think the main reason why people want to burn your article to ground was because there are way more companies out there that would benefit from their people writing unit-tests. Now when some lazy developers who think that they're smart starting to quote "the VP of Software Development from JTV sez unit-test is useless", then we have a problem there. These passionate software developers don't want to go back to the time of Wild Wild West where coders slinging code with no reasons and start throwing incomplete shitty crap to QA and call it "DONE", get the gold medals from their team lead for being able to solve things very fast. You know that movie... Spiderman? "With Power comes great responsibility"... since you're advertising how cool your company is and you're their "VP", then there's that responsibility attached with it. The thing is, it requires a smart person with high level of discipline, strong principal, and the ability to say "No" to crazy requirements in order to have the luxury of not writing unit-tests. Especially when there's no safety nets (I mean QA) around that person. I hold nothing against you though.

by kayoone 16y ago

anamax 16y ago

> I found that quite irritating from somebody who seems so experienced. Okay, but is he wrong? Let's be specific. He says "[Unit testing] just doesn't cover a large enough class of potential problems to be generally useful." He said how he evaluates bug-finding tools and discussed his experience. What is yours?

nostrademons 16y ago
The dumb part about this debate is that it's being framed as all-or-nothing. The important question isn't "Are unit tests useful?" (the answer to that is "It depends..."), it's "Under which circumstances are unit tests useful?" If a particular unit test costs you more to write than it saves you in bugs, don't write it. And if it starts costing more in maintenance than it saves in confidence, get rid of it. There's a fair bit of cargo culting going on about how much to test - in my experience, it's not worth testing getters/setters, it's not worth testing simple "if this then set these variables" logic, and it's not worth unit testing database or external calls (you should have some integration tests for them though). It is very much worth testing tricky cases like a parser, or a bit-level decoder, or complicated algorithms. I just recently got up to 100% code coverage on a UTF-8 decoder I'd written. I found about a dozen bugs in the process. They were silly things, like leaving a digit off a hex constant, but they rippled throughout the decoder so that the results were completely wrong. If I'd just included that in another library without testing, it'd make the dependent library subtly wrong too, and I'd never hear about it until users started complaining, and they'd waste hours trying to find bugs in their code that are really bugs in mine. I'll also mention one other thing: the point of unit testing isn't to find bugs, it's to give you enough confidence in your code that you can freely change it. The real cost of bugs isn't that one erroneous event that customers will probably never notice, it's that the fear of that one unpredictable event makes management (which is sometimes you, if it's your startup) say "It works. Don't touch it, because you'll break it" and never improve the software. That's how you get shitty software that you can't fix.
- petenixey 16y ago
  This is a very good point: are they useful? I think it's fair to say that integration tests (make sure that the functionality works) are useful; period. Not knowing whether your app is going to work when you push it is not fun. From a maintenance perspective and assuming that a problem will show up in integration testing or via a bug report, further testing is only really useful to the extent that it lets you debug a failure faster. However that acceleration is a tradeoff against the time taken to write the tests in the first place - ergo: is it really worth it? I personally don't think the real win from tests is in saving debugging time. I have found it's in making it quicker to write code. I know that sounds crazy as most people view tests as overhead but I view them as development tools. The key to them being useful though is writing them before you write the code. If you write them afterwards you will have already verified the code in an ad-hoc fashion and the tests are pure overhead. If you write them before they do the verification for you. I'm coding in Rails and I've found the outside-in approach to development: write the six line Cucumber integration test and then write the RSpec expectations for the object; act as a verified to-do list and make my development far more efficient and robust. I'm sure that other developers with better focus than I might not get the same benefit but I find it tremendously useful to immediately list down all of the edge cases and general behaviour that a piece of code should exhibit and then keep coding till growl glows green. I trade the time that I'd otherwise spend faffing around in a browser or a debugger for writing tests instead. It's a bonus that all of those tests then ensure ongoing integrity, the real win is that tests stop me from wandering around the trail and keep me knocking things off in the order that they need to be done. [Caveat: it has taken me months and months to come up to speed on how to write tests well] (great article btw Bill)
deleted 16y ago
[deleted]
- justin 16y ago
  FWIW: With Socialcam, Eric implemented unit tests for the API (which powers the client apps), and initially for the website but later we decided against them for the site. Having them on the API definitely has been worth the initial cost as the backend has needed to be very stable for client development, and they are easily testable. Tim, perhaps you should have made your point more vigorously when you were here :). That said, I think that many / most of our projects fall into the bus count 1, cowboy coder hacking (at least, initially), so we've been able to move fast by not having them.

gridspy 16y ago

Speaking from personal experience, there are a lot of fads in software development. From the most recent language to the latest and greatest library. From design patterns through to work practices like Agile and TDD. You must be critical in your thinking for each one of these. Just because it works for someone else doesn't mean it will work for you. Personally, I think that most design patterns are used as an excuse to make a simple system complex. I think that unit tests are more painful to maintain than they are worth (in many cases). I think that new technology should be approached carefully and only proven tech with minimum lockin should be used. Don't go round saying that "experienced developers always do X" because a real experienced developer knows that X is great for some things and a huge waste of time for others.

nahname 16y ago

He really does not come off well in those comments. Repeatedly saying I am successful, therefore what I am doing is right. In his defense, it sounds like he has mostly worked by himself. Some practices don't have a very good ROI when you are just hacking away by yourself. I would argue that unit testing is one of them, if you are not very experienced with it anyways.

gridspy 16y ago
It sounds to me like he got beaten up in his own comments section. The people commenting don't seem to realize that he is talking about a different type of programming than the commenters are doing.
- nahname 16y ago
  What do you mean by "type of programming"?
  - gridspy 16y ago
    I probably should have said "Problem Domain" I.e Are you an author of a library, a browser, a startup, experimental code, rapidly evolving code, space shuttle code ? Each one has different measurements of "success" and these can effect your choices re Unit Tests.
deleted 16y ago
[deleted]

kenjackson 16y ago

Are there any "hacker gods" that are really big on unit testing or use TDD? Most of the recognizable names that I see talk about it are agile book writers. Not actual devs. It was Paul Bucheit who says (from that link), "I'll make a somewhat heretical claim even: Unit test are 20% useful engineering, and 80% fad. They are great for tricky code with well defined behavior (like a parser), but wasteful for most code." Does Sweeney, Carmack, Cutler, Joy, Thompson, Knuth, Page, Steele, Zuckerburg, Anders, etc... use unit tests in a big way? I'm actually a fan of unit tests, although I generally dislike TDD.

abstractbill 16y ago
Nobody in Peter Seibel's "Coders At Work" seemed to have very much time for unit testing as far as I remember, and they pretty much all qualify for that title in my opinion. I'd stand by pretty much everything I wrote in my original post, for what it's worth, but I guess I don't really understand why it seemed to ruffle so many feathers to be honest. Unit testing doesn't help me enough for the work I have to put in (compared to things like good logging and monitoring, for example, unless I'm writing something like a parser, with well-specified behavior). It doesn't seem to have helped justin.tv too much either, when we've tried it. But if it helps someone else, then good for them!
- hello_moto 16y ago
  I agree with you: if it works for someone else, good for them. Definitely. I think the main reason why people want to burn your article to ground was because there are way more companies out there that would benefit from their people writing unit-tests. Now when some lazy developers who think that they're smart starting to quote "the VP of Software Development from JTV sez unit-test is useless", then we have a problem there. These passionate software developers don't want to go back to the time of Wild Wild West where coders slinging code with no reasons and start throwing incomplete shitty crap to QA and call it "DONE", get the gold medals from their team lead for being able to solve things very fast. You know that movie... Spiderman? "With Power comes great responsibility"... since you're advertising how cool your company is and you're their "VP", then there's that responsibility attached with it. The thing is, it requires a smart person with high level of discipline, strong principal, and the ability to say "No" to crazy requirements in order to have the luxury of not writing unit-tests. Especially when there's no safety nets (I mean QA) around that person. I hold nothing against you though.