21 ms·

Exploding Software-Engineering Myths (2009)

145 points by luu 12y ago

amelius 12y ago
"higher code coverage was not the best measure of post-release failures in the field" Perhaps they should measure the coverage by taking the number of tested paths through the software divided by the total number of paths.
2muchcoffeeman 12y ago
"Code coverage measures how comprehensively a piece of code has been tested; if a program contains 100 lines of code and the quality-assurance process tests 95 lines, the effective code coverage is 95 percent." I've never heard of this before. Where does this come from and why did they suspect this would be indicative of anything?
- aout 12y ago
  It is a fairly common practice in companies that do not have technical managers (http://en.wikipedia.org/wiki/Code_coverage http://en.wikipedia.org/wiki/Code_coverage). Basically it is useful to measure the coverage of existing conditions (and insure stability) but clearly lacks the ability to test the conditions you forgot to write. Kind of defeats half of the purpose.
  - robfblair 12y ago
    Companies also track and report is when required by external entities such as the FAA for safety critical systems. I think it goes deeper than "the conditions you forgot to write" although that's part of it. A developer's test suite tends to make assumptions and these are the same assumptions that exist in the production code. If your assumptions about usage, or input, or edge cases are poor then the tests you write will be poor too and in that case 100% coverage isn't worth much. Big reason I'm a fan of things like QuickCheck in Haskell.
    - zelos 12y ago
      A quick Google suggests the FAA uses some version of decision coverage (i.e. for each branch in your code, what percentage were taken during testing), not simple line coverage.
- cgh 12y ago
  The idea is that shipping untested code is worse than shipping (unit) tested code. Code coverage reporting is quite common and I'm surprised you've never even heard of it if you've ever written software professionally.
  - tbrownaw 12y ago
    It's popular among the hobbyist / activist crowd. People who like to talk about the "proper" way to develop software tend to like it. People who run open source projects tend to either like it or bow to peer pressure. But people working on in-house and enterprisey stuff? Yeah, not so much there.
    - KMag 12y ago
      > But people working on in-house and enterprisey stuff? Yeah, not so much there. This depends a lot on your corporate development culture, even for in-house enterprisey software.
    - jacquesm 12y ago
      > But people working on in-house and enterprisey stuff? Yeah, not so much there. I suspect you are working from a limited sample size there.
    - cgh 12y ago
      "But people working on in-house and enterprisey stuff? Yeah, not so much there." That is absolutely incorrect, in my experience.
- dankohn1 12y ago
  Take a look at tools like Istanbul for Node.js and Simplecov for Ruby on Rails. https://coveralls.io/ https://coveralls.io/ has a nice list of code coverage tools for many languages. If you don't have tests, it is an extremely manual process to change anything about your codebase (including dependencies) and have any confidence that you haven't broken things. Once you write tests, code coverage lets you figure out what code is not being tested.
- wyldfire 12y ago
  > Where does this come from It's defining the term "code coverage" to be the ratio of lines covered by tests to the total lines. > why did they suspect this would be indicative of anything? If anything, it's a useful feedback mechanism. e.g. "Gee, we wrote sixteen unit tests for this module but we're still only at 34% coverage. Ohh, I see it now -- framistan.cpp doesn't get used by these tests at all. We should write a test that enables framistan mode!"
- jacquesm 12y ago
  Checking your coverage for tests is a pretty old concept. Early 1960's or so!
aout 12y ago
I'm not sure I fully understood the moral of the article but it seems to me that all of these findings are basically what "people knew for a long time". Okay it's good to put data on "beliefs" but is it really useful when quantifying "promoted usage" and "obligation" has failed? The article does prove that some people tried to impose (via management or structure in large organizations) metrics on coding as a way to rationalize the development quality and miserably failed but clearly lack the counterpart: what happens when you don't impose these metrics? Experience is important but that's not new is it? In the end it just feels like: small programs won't have much problem as do small organizations - Big organization should function like small companies. Not much of an answer. Also, I'm not sure that a manager saying something like "according to data, we need one more month to make the program 35% more stable" is a good argument (or even a new one). What would be interesting is to try something like "ship the program one month earlier and dedicate the two remaining month to fix what's broken according to feedback", then compare with the first approach.
deleted 12y ago
[deleted]
asQuirreL 12y ago
After reading the paper [1] regarding the empirical study on TDD, I'm pretty dubious about how well backed the conclusions are. If you look at Table 2, in the 4 case studies, 2 of the projects took longer with TDD (2.00x, and 2.64x ratio in man months to 3sf) but then the other 2 did better: (0.319x and 0.476x ratio in man months, 3sf). But all of this seems to be ignored in proving the original statement that TDD take 15-35% longer, which appears to be backed by what is described as "Management Estimates" [Table 3]. In fact, taking the geometric mean of the man month values (which, I think, but am not sure is the right metric to use when comparing ratios), we find the average ratio to be 0.947x. Make of this what you will, given the size of the sample. [1]: http://research.microsoft.com/en-us/groups/ese/nagappan_tdd.pdf http://research.microsoft.com/en-us/groups/ese/nagappan_tdd....
- Silhouette 12y ago
  The Nagappan paper gets cited a fair bit, mostly because there is so little hard evidence of any sort that actually backs TDD as presented by the big name evangelists. However, on a closer reading, even that paper isn't really assessing TDD. It basically defines any test-first coding practice to be TDD, including those that still spend considerable time and resources on up-front design. What we really need is some properly structured study that compares professional practitioners (not just students or recent grads) over extended periods (not just a brief project that doesn't show the long-term benefits or costs in terms of quality and maintainability) who are using no unit tests but otherwise common development practices (control group), the same but writing unit tests after the main code, the same but with unit tests written before the main code, and full test-driven development where other aspects of the development process, particularly design, are effectively replaced by the test-writing activities as advocated in well-known TDD books and other training materials. Of course it's all but impossible to really achieve that sort of like-for-like comparison with usefully controlled conditions, which is why these issues are so hard to debate objectively. But try finding any study with pro-TDD conclusions that doesn't obviously fail the objectivity test in at least one of the three ways above (not a sample of professional programmers, too short term to draw general conclusions, or conflating TDD with some other unit-test-related development process).
cubano 12y ago
there needs to be a culture of using assertions in order to produce the desired results. IMO, proper "culture" is a significant part of almost all aspects of project management and no matter what, without it, the teams will never reach their optimum potentials, unless some highly motivated dev takes upon their shoulders to implement these base tenets for everyone else. I'm not saying it's always easy to implement or it guarantees success, but that the lack of it always rears it's ugly head at some point in the development process.
aetherson 12y ago
You know, I understand and agree that code coverage isn't the be-all and end-all of testing. But... we do all secretly agree that if it isn't pretty high, then the testing isn't doing much, right? Like, does anyone feel really good about saying there's good test coverage of a class with 50% code coverage? 60%? After I hit 80-90% coverage, I start wanting to know about other attributes of the tests. Are they really really really testing that all the outputs are exactly what we expect? Are they controlling dependencies in a useful, realistic way? Are they exploring a wide range of inputs? Before that, though, all of the above seem a bit premature. I don't deeply care if you've done a really thorough job of testing 20% of the code if the other 80% isn't even run.
- derefr 12y ago
  I would agree generally (because most code has exposure to the consumer even when it shouldn't), but, in the specific instance that 1. the code is effectively formed into a library/component/service that you only interact with through a well-defined public API, and 2. the tests test every exposed function of the API, then you don't need to test anything other than that API—100% code coverage (covering code-paths the API can't reach) is meaningless. Any code that the tests don't test in such a scenario is actually code that the library doesn't use; it's dead weight, and should be culled (and actually, given a whole-program optimization pass, it would be.)
- sbov 12y ago
  Like most things, the answer is... it depends. We have several with 0% and I'm just fine with that. They're exceedingly simple. And/or they're 15 years old and only change in very slight ways every few years. There's other classes that are 100% that I'm not fine with. Because while the code is short, the implications are complex. I feel like blindly talking about these metrics is like saying "hey, I bought an earring for $100, did I get ripped off?"
  - aetherson 12y ago
    I think that you're broadening the question. Sure, sometimes you feel okay with code that's not at all tested. But surely you don't feel that code that is not at all tested is well tested. I'm talking about to what extent code coverage can and can not be used as a proxy for "well tested," not to what extent "well tested" can and can not be used as a proxy for "I feel good about my code."
serve_yay 12y ago
Data helps. But you still have to know how to apply it.
tokenrove 12y ago
A more recent paper that's related to this, "Coverage Is Not Strongly Correlated with Test Suite Effectiveness": http://www.linozemtseva.com/research/2014/icse/coverage/coverage_paper.pdf http://www.linozemtseva.com/research/2014/icse/coverage/cove... I wouldn't want people to conclude that coverage was a useless metric based on this, but it does seem to support the idea that chasing code coverage dogmatically is not necessarily the best use of your time.
- wyldfire 12y ago
  I agree -- IMO being dogmatic about 100% or any other number can be a big waste of time. But a coverage ratio and/or browsing annotated source is a good feedback mechanism for how many tests you've written.
pacaro 12y ago
One way of looking at the coverage issue is that high coverage isn't a very strong positive signal, but low or absent coverage is a strong negative signal. It's also interesting to combine metrics like cyclomatic complexity, or relative churn rate, with coverage. Files/modules with high complexity/churn and low coverage are much scarier than those with low complexity and low coverage
- pckspcks 12y ago
  Empirically, I have not seen this to be the case. Test coverage has had little correlation to code quality on the projects I've worked on. I would agree that well-written tests of the most complex parts of code help contribute towards a quality system. Most of the other tests -- sometimes more so, and sometimes less so, depending on other aspects of the project.
  - pacaro 12y ago
    I was trying hard not to make a strong assertion, so I'm not sure what you are disagreeing with entirely... TL;DR - it's a matter of context But here's an anecdote where two complex pieces of code interacted... A few years ago I was working on a fairly big project (~50 developers) and was responsible for much of the architecture, and on an implementation level, implementing some of the lines that connected boxes in said architecture. One of those was a line that crossed what can loosely be thought of as a system/application boundary, this involved both implementing the application hosting layer and writing the marshalling code that moved data in either direction, there was a bunch of complexity because of not quite convergent type systems, some threading issues etc... So all this is done, not too many tests have been written, but hey, it works, if it didn't nothing in the system would work... a year or so into the project, the most senior technical person on the team has a flaky test that once in a while just completely fails to make progress, he investigates, and comes to the conclusion that once in a blue moon, the transport layer (my code) is dropping a message. Needless to say I'm totally flummoxed, there is no code path that can drop messages without also creating a huge stink in logs etc. I spend two months investigating this, eventually I have a consistent repo, a fairly big hole in the other guys state machine that causes it to just stop making forward progress. So despite the fact that a test exposed the bug, neither piece of complex code really had proper tests, but I would assert that the "line" didn't need the same level of tests, it was being exercised constantly by dozens of "boxes", but the unicorn "box" with the logic error could sure as hell have used some unit tests to validate its state transitions.
wampler 12y ago
The Mythical Man Month is damn correct.
douche 12y ago
One thing I have never understood is how to apply TDD when the code that you are writing is nearly 100% dependent on external services and "black-box" library code. For instance, I rely on several more or less undocumented Microsoft APIs in my day job. How do I even know what correct outputs to code interacting with those components would be? It could depend on server policies, registry settings, firewall settings, SQL server settings, Active directory settings, and a hundred other things. My application code is bog-simple next to all of the code handling possible FUBAR states of all of this environment.
- BurningFrog 12y ago
  What you're saying is that it's not clear to you what your code needs to do. That's a fundamental problem that neither TDD or any other methodology can really solve.
  - douche 12y ago
    It's clear to me what my code needs to do, however there are a myriad of environmental factors that can fuck up what my simple code needs to do. For instance, a common thing that I need to do is to look up one Active Directory property based on another. Depending on the environment, I may have access to that property using the logged-in user's credentials, or I may need to use a special service account set of credentials in my application, defined in a database or an app/web.config. At some point, an over-zealous administrator can disable the rights for the account that my application is using inside a client organization. Then I end up on the phone for an hour or more, figuring out why an account that used to work no longer does, and getting my clients to raise a change request with the admins that broke our application in the first place to put things back.
    - coldtea 12y ago
      >It's clear to me what my code needs to do, however there are a myriad of environmental factors that can fuck up what my simple code needs to do. For instance, a common thing that I need to do is to look up one Active Directory property based on another. Depending on the environment, I may have access to that property using the logged-in user's credentials, or I may need to use a special service account set of credentials in my application, defined in a database or an app/web.config. Some stuff you test in unit tests, other you do functional testing for. What you describe is probably higher level than unit tests. >At some point, an over-zealous administrator can disable the rights for the account that my application is using inside a client organization. Then I end up on the phone for an hour or more, figuring out why an account that used to work no longer does, and getting my clients to raise a change request with the admins that broke our application in the first place to put things back. So? That's orthogonal to your app. You test that your app works if the rights are working OK, and you test what it shows to the user if the rights are not OK. Other than that, obviously you cannot ensure it works without things it needs to have, nor do you need to test each and every externally dependent failure mode (e.g. user wiped out his hard drive).
BurningFrog 12y ago
"Over a development cycle of 12 months, 35 percent is another four months, which is huge,” Nagappan says. “However, the tradeoff is that you reduce post-release maintenance costs significantly, since code quality is so much better." Another way of describing this is that the non TDD teams weren't actually done when they shipped.
- simula67 12y ago
  Unless the TDD teams shipped code without any bugs, they were not done when they shipped either.
- coldtea 12y ago
  >Another way of describing this is that the non TDD teams weren't actually done when they shipped. Was ANY code, TDD or not, ever "done" when it shipped?
  - sjwright 12y ago
    Absolutely. Off the top of my head, the first thing that comes to mind are video games that were shipped on ROM cartridges. Bugs or not, when shipped, it's done.
    - yitchelle 12y ago
      Those days are long gone. Even in an deeply embedded world like automotive or AV equipment, code is (or can be) frequently updated.
      - sjwright 12y ago
        In the very deeply embedded world, there are often no update mechanisms, period.
  - actsasbuffoon 12y ago
    Possibly pacemaker firmware? I have no real knowledge of the subject, but I'm guessing it'd be kind of risky to equip a pacemaker with Bluetooth to accept over-the-air firmware updates.
ploxiln 12y ago
Lots of discussion here about TDD and test coverage... but the most significant factor of "code quality" found was "organizational metrics": > Organizational metrics, which are not related to the code, can predict software failure-proneness with a precision and recall of 85 percent. This is a significantly higher precision than traditional metrics such as churn, complexity, or coverage that have been used until now to predict failure-proneness. This was probably the most surprising outcome of all the studies. Which I take to confirm my belief that the freedom and responsibility given to developers, and how good they are, is more significant than the specific testing policy enforced :)
- UK-AL 12y ago
  More freedom, means more freedom to test correctly. As opposed to be constantly being pushed to release as fast as possible.
  - mreiland 12y ago
    It's a hell of a lot more complex than that. A developer with a short deadline is going to design things differently than one with a longer deadline. That superior design can often result in better long term maintenance. There's also the idea of requirements gathering. Some people take it for granted, but if your developers aren't in a position of power in your company, they can't insist on good requirements for software. I've experienced this firsthand. I once worked for a company in which the field engineers brought in a large chunk of the money for the company. They were the "big dick swingers" in the company, if you will, while the software development team was an attempt at the company to automate a lot of the work done by engineers and then sell services with the gathered data. This presented a conflict of interest in that the software would have actively made the company need less engineers as they could do their job faster, but because of the two groups relative status within the company, there was no way for the software team to insist on accurate specifications. The day I walked out of that company I had been asked to rewrite a module of the software for the 3rd time and requested written documentation. I was on the phone with both my manager and the engineer we were supposed to be coordinating with. The engineer flat out refused and told me I didn't need it. I asked him how many years of software dev experience he had. That's one anecdote, but the point is, politics in a company severely affect the performance of developers and not simply because of testing. I honestly think that's an awfully naive view of the world.
jraedisch 12y ago
The mentioned technical paper about assertions states: "The assertions are primarily of two types of specification: specification of function interfaces (e.g. consistency between arguments, dependency of return value on arguments, effect of global state etc.) and specification of function bodies (e.g. consistency of default branch in switch statement, consistency between related data etc.)." from http://research.microsoft.com/pubs/70290/tr-2006-54.pdf http://research.microsoft.com/pubs/70290/tr-2006-54.pdf Is there somebody here, who could elaborate on this, ideally with some small examples?
- qznc 12y ago
  You want to have examples for assertions? Consistency between arguments: memcpy "The memory areas must not overlap." Dependency of return value on arguments: max returns one of its arguments. r = max(x,y); assert (r == x || r == y); Effect on global state: printf might change the global errno variable.
  - jraedisch 12y ago
    So this means embedding small tests into the production code itself while accepting minor performance loss? And you somehow get a feeling for when this will help through experience? I wonder, whether this impacts all programming languages the same.
    - qznc 12y ago
      It is not necessarily about runtime tests. Often assertions are only used for debug builds. Java requires an extra flag to enable assert statements. Alternatively, assertions could be invariants or pre/post conditions for your formal proofs of correctness. I'm too lazy to check the actual paper, if they clarify their understanding. ;)

145 points by luu 12y ago

"higher code coverage was not the best measure of post-release failures in the field" Perhaps they should measure the coverage by taking the number of tested paths through the software divided by the total number of paths.

2muchcoffeeman 12y ago

"Code coverage measures how comprehensively a piece of code has been tested; if a program contains 100 lines of code and the quality-assurance process tests 95 lines, the effective code coverage is 95 percent." I've never heard of this before. Where does this come from and why did they suspect this would be indicative of anything?

aout 12y ago
It is a fairly common practice in companies that do not have technical managers (http://en.wikipedia.org/wiki/Code_coverage http://en.wikipedia.org/wiki/Code_coverage). Basically it is useful to measure the coverage of existing conditions (and insure stability) but clearly lacks the ability to test the conditions you forgot to write. Kind of defeats half of the purpose.
- robfblair 12y ago
  Companies also track and report is when required by external entities such as the FAA for safety critical systems. I think it goes deeper than "the conditions you forgot to write" although that's part of it. A developer's test suite tends to make assumptions and these are the same assumptions that exist in the production code. If your assumptions about usage, or input, or edge cases are poor then the tests you write will be poor too and in that case 100% coverage isn't worth much. Big reason I'm a fan of things like QuickCheck in Haskell.
  - zelos 12y ago
    A quick Google suggests the FAA uses some version of decision coverage (i.e. for each branch in your code, what percentage were taken during testing), not simple line coverage.
cgh 12y ago
The idea is that shipping untested code is worse than shipping (unit) tested code. Code coverage reporting is quite common and I'm surprised you've never even heard of it if you've ever written software professionally.
- tbrownaw 12y ago
  It's popular among the hobbyist / activist crowd. People who like to talk about the "proper" way to develop software tend to like it. People who run open source projects tend to either like it or bow to peer pressure. But people working on in-house and enterprisey stuff? Yeah, not so much there.
  - KMag 12y ago
    > But people working on in-house and enterprisey stuff? Yeah, not so much there. This depends a lot on your corporate development culture, even for in-house enterprisey software.
  - jacquesm 12y ago
    > But people working on in-house and enterprisey stuff? Yeah, not so much there. I suspect you are working from a limited sample size there.
  - cgh 12y ago
    "But people working on in-house and enterprisey stuff? Yeah, not so much there." That is absolutely incorrect, in my experience.
dankohn1 12y ago
Take a look at tools like Istanbul for Node.js and Simplecov for Ruby on Rails. https://coveralls.io/ https://coveralls.io/ has a nice list of code coverage tools for many languages. If you don't have tests, it is an extremely manual process to change anything about your codebase (including dependencies) and have any confidence that you haven't broken things. Once you write tests, code coverage lets you figure out what code is not being tested.
wyldfire 12y ago
> Where does this come from It's defining the term "code coverage" to be the ratio of lines covered by tests to the total lines. > why did they suspect this would be indicative of anything? If anything, it's a useful feedback mechanism. e.g. "Gee, we wrote sixteen unit tests for this module but we're still only at 34% coverage. Ohh, I see it now -- framistan.cpp doesn't get used by these tests at all. We should write a test that enables framistan mode!"
jacquesm 12y ago
Checking your coverage for tests is a pretty old concept. Early 1960's or so!

aout 12y ago

I'm not sure I fully understood the moral of the article but it seems to me that all of these findings are basically what "people knew for a long time". Okay it's good to put data on "beliefs" but is it really useful when quantifying "promoted usage" and "obligation" has failed? The article does prove that some people tried to impose (via management or structure in large organizations) metrics on coding as a way to rationalize the development quality and miserably failed but clearly lack the counterpart: what happens when you don't impose these metrics? Experience is important but that's not new is it? In the end it just feels like: small programs won't have much problem as do small organizations - Big organization should function like small companies. Not much of an answer. Also, I'm not sure that a manager saying something like "according to data, we need one more month to make the program 35% more stable" is a good argument (or even a new one). What would be interesting is to try something like "ship the program one month earlier and dedicate the two remaining month to fix what's broken according to feedback", then compare with the first approach.

deleted 12y ago

[deleted]

asQuirreL 12y ago

After reading the paper [1] regarding the empirical study on TDD, I'm pretty dubious about how well backed the conclusions are. If you look at Table 2, in the 4 case studies, 2 of the projects took longer with TDD (2.00x, and 2.64x ratio in man months to 3sf) but then the other 2 did better: (0.319x and 0.476x ratio in man months, 3sf). But all of this seems to be ignored in proving the original statement that TDD take 15-35% longer, which appears to be backed by what is described as "Management Estimates" [Table 3]. In fact, taking the geometric mean of the man month values (which, I think, but am not sure is the right metric to use when comparing ratios), we find the average ratio to be 0.947x. Make of this what you will, given the size of the sample. [1]: http://research.microsoft.com/en-us/groups/ese/nagappan_tdd.pdf http://research.microsoft.com/en-us/groups/ese/nagappan_tdd....

Silhouette 12y ago
The Nagappan paper gets cited a fair bit, mostly because there is so little hard evidence of any sort that actually backs TDD as presented by the big name evangelists. However, on a closer reading, even that paper isn't really assessing TDD. It basically defines any test-first coding practice to be TDD, including those that still spend considerable time and resources on up-front design. What we really need is some properly structured study that compares professional practitioners (not just students or recent grads) over extended periods (not just a brief project that doesn't show the long-term benefits or costs in terms of quality and maintainability) who are using no unit tests but otherwise common development practices (control group), the same but writing unit tests after the main code, the same but with unit tests written before the main code, and full test-driven development where other aspects of the development process, particularly design, are effectively replaced by the test-writing activities as advocated in well-known TDD books and other training materials. Of course it's all but impossible to really achieve that sort of like-for-like comparison with usefully controlled conditions, which is why these issues are so hard to debate objectively. But try finding any study with pro-TDD conclusions that doesn't obviously fail the objectivity test in at least one of the three ways above (not a sample of professional programmers, too short term to draw general conclusions, or conflating TDD with some other unit-test-related development process).

cubano 12y ago

there needs to be a culture of using assertions in order to produce the desired results. IMO, proper "culture" is a significant part of almost all aspects of project management and no matter what, without it, the teams will never reach their optimum potentials, unless some highly motivated dev takes upon their shoulders to implement these base tenets for everyone else. I'm not saying it's always easy to implement or it guarantees success, but that the lack of it always rears it's ugly head at some point in the development process.

aetherson 12y ago

You know, I understand and agree that code coverage isn't the be-all and end-all of testing. But... we do all secretly agree that if it isn't pretty high, then the testing isn't doing much, right? Like, does anyone feel really good about saying there's good test coverage of a class with 50% code coverage? 60%? After I hit 80-90% coverage, I start wanting to know about other attributes of the tests. Are they really really really testing that all the outputs are exactly what we expect? Are they controlling dependencies in a useful, realistic way? Are they exploring a wide range of inputs? Before that, though, all of the above seem a bit premature. I don't deeply care if you've done a really thorough job of testing 20% of the code if the other 80% isn't even run.

derefr 12y ago
I would agree generally (because most code has exposure to the consumer even when it shouldn't), but, in the specific instance that 1. the code is effectively formed into a library/component/service that you only interact with through a well-defined public API, and 2. the tests test every exposed function of the API, then you don't need to test anything other than that API—100% code coverage (covering code-paths the API can't reach) is meaningless. Any code that the tests don't test in such a scenario is actually code that the library doesn't use; it's dead weight, and should be culled (and actually, given a whole-program optimization pass, it would be.)
sbov 12y ago
Like most things, the answer is... it depends. We have several with 0% and I'm just fine with that. They're exceedingly simple. And/or they're 15 years old and only change in very slight ways every few years. There's other classes that are 100% that I'm not fine with. Because while the code is short, the implications are complex. I feel like blindly talking about these metrics is like saying "hey, I bought an earring for $100, did I get ripped off?"
- aetherson 12y ago
  I think that you're broadening the question. Sure, sometimes you feel okay with code that's not at all tested. But surely you don't feel that code that is not at all tested is well tested. I'm talking about to what extent code coverage can and can not be used as a proxy for "well tested," not to what extent "well tested" can and can not be used as a proxy for "I feel good about my code."

serve_yay 12y ago

Data helps. But you still have to know how to apply it.

tokenrove 12y ago

A more recent paper that's related to this, "Coverage Is Not Strongly Correlated with Test Suite Effectiveness": http://www.linozemtseva.com/research/2014/icse/coverage/coverage_paper.pdf http://www.linozemtseva.com/research/2014/icse/coverage/cove... I wouldn't want people to conclude that coverage was a useless metric based on this, but it does seem to support the idea that chasing code coverage dogmatically is not necessarily the best use of your time.

wyldfire 12y ago
I agree -- IMO being dogmatic about 100% or any other number can be a big waste of time. But a coverage ratio and/or browsing annotated source is a good feedback mechanism for how many tests you've written.

pacaro 12y ago

One way of looking at the coverage issue is that high coverage isn't a very strong positive signal, but low or absent coverage is a strong negative signal. It's also interesting to combine metrics like cyclomatic complexity, or relative churn rate, with coverage. Files/modules with high complexity/churn and low coverage are much scarier than those with low complexity and low coverage

pckspcks 12y ago
Empirically, I have not seen this to be the case. Test coverage has had little correlation to code quality on the projects I've worked on. I would agree that well-written tests of the most complex parts of code help contribute towards a quality system. Most of the other tests -- sometimes more so, and sometimes less so, depending on other aspects of the project.
- pacaro 12y ago
  I was trying hard not to make a strong assertion, so I'm not sure what you are disagreeing with entirely... TL;DR - it's a matter of context But here's an anecdote where two complex pieces of code interacted... A few years ago I was working on a fairly big project (~50 developers) and was responsible for much of the architecture, and on an implementation level, implementing some of the lines that connected boxes in said architecture. One of those was a line that crossed what can loosely be thought of as a system/application boundary, this involved both implementing the application hosting layer and writing the marshalling code that moved data in either direction, there was a bunch of complexity because of not quite convergent type systems, some threading issues etc... So all this is done, not too many tests have been written, but hey, it works, if it didn't nothing in the system would work... a year or so into the project, the most senior technical person on the team has a flaky test that once in a while just completely fails to make progress, he investigates, and comes to the conclusion that once in a blue moon, the transport layer (my code) is dropping a message. Needless to say I'm totally flummoxed, there is no code path that can drop messages without also creating a huge stink in logs etc. I spend two months investigating this, eventually I have a consistent repo, a fairly big hole in the other guys state machine that causes it to just stop making forward progress. So despite the fact that a test exposed the bug, neither piece of complex code really had proper tests, but I would assert that the "line" didn't need the same level of tests, it was being exercised constantly by dozens of "boxes", but the unicorn "box" with the logic error could sure as hell have used some unit tests to validate its state transitions.

wampler 12y ago

The Mythical Man Month is damn correct.

douche 12y ago

One thing I have never understood is how to apply TDD when the code that you are writing is nearly 100% dependent on external services and "black-box" library code. For instance, I rely on several more or less undocumented Microsoft APIs in my day job. How do I even know what correct outputs to code interacting with those components would be? It could depend on server policies, registry settings, firewall settings, SQL server settings, Active directory settings, and a hundred other things. My application code is bog-simple next to all of the code handling possible FUBAR states of all of this environment.

BurningFrog 12y ago
What you're saying is that it's not clear to you what your code needs to do. That's a fundamental problem that neither TDD or any other methodology can really solve.
- douche 12y ago
  It's clear to me what my code needs to do, however there are a myriad of environmental factors that can fuck up what my simple code needs to do. For instance, a common thing that I need to do is to look up one Active Directory property based on another. Depending on the environment, I may have access to that property using the logged-in user's credentials, or I may need to use a special service account set of credentials in my application, defined in a database or an app/web.config. At some point, an over-zealous administrator can disable the rights for the account that my application is using inside a client organization. Then I end up on the phone for an hour or more, figuring out why an account that used to work no longer does, and getting my clients to raise a change request with the admins that broke our application in the first place to put things back.
  - coldtea 12y ago
    >It's clear to me what my code needs to do, however there are a myriad of environmental factors that can fuck up what my simple code needs to do. For instance, a common thing that I need to do is to look up one Active Directory property based on another. Depending on the environment, I may have access to that property using the logged-in user's credentials, or I may need to use a special service account set of credentials in my application, defined in a database or an app/web.config. Some stuff you test in unit tests, other you do functional testing for. What you describe is probably higher level than unit tests. >At some point, an over-zealous administrator can disable the rights for the account that my application is using inside a client organization. Then I end up on the phone for an hour or more, figuring out why an account that used to work no longer does, and getting my clients to raise a change request with the admins that broke our application in the first place to put things back. So? That's orthogonal to your app. You test that your app works if the rights are working OK, and you test what it shows to the user if the rights are not OK. Other than that, obviously you cannot ensure it works without things it needs to have, nor do you need to test each and every externally dependent failure mode (e.g. user wiped out his hard drive).

BurningFrog 12y ago

"Over a development cycle of 12 months, 35 percent is another four months, which is huge,” Nagappan says. “However, the tradeoff is that you reduce post-release maintenance costs significantly, since code quality is so much better." Another way of describing this is that the non TDD teams weren't actually done when they shipped.

simula67 12y ago
Unless the TDD teams shipped code without any bugs, they were not done when they shipped either.
coldtea 12y ago
>Another way of describing this is that the non TDD teams weren't actually done when they shipped. Was ANY code, TDD or not, ever "done" when it shipped?
- sjwright 12y ago
  Absolutely. Off the top of my head, the first thing that comes to mind are video games that were shipped on ROM cartridges. Bugs or not, when shipped, it's done.
  - yitchelle 12y ago
    Those days are long gone. Even in an deeply embedded world like automotive or AV equipment, code is (or can be) frequently updated.
    - sjwright 12y ago
      In the very deeply embedded world, there are often no update mechanisms, period.
- actsasbuffoon 12y ago
  Possibly pacemaker firmware? I have no real knowledge of the subject, but I'm guessing it'd be kind of risky to equip a pacemaker with Bluetooth to accept over-the-air firmware updates.

ploxiln 12y ago

Lots of discussion here about TDD and test coverage... but the most significant factor of "code quality" found was "organizational metrics": > Organizational metrics, which are not related to the code, can predict software failure-proneness with a precision and recall of 85 percent. This is a significantly higher precision than traditional metrics such as churn, complexity, or coverage that have been used until now to predict failure-proneness. This was probably the most surprising outcome of all the studies. Which I take to confirm my belief that the freedom and responsibility given to developers, and how good they are, is more significant than the specific testing policy enforced :)

UK-AL 12y ago
More freedom, means more freedom to test correctly. As opposed to be constantly being pushed to release as fast as possible.
- mreiland 12y ago
  It's a hell of a lot more complex than that. A developer with a short deadline is going to design things differently than one with a longer deadline. That superior design can often result in better long term maintenance. There's also the idea of requirements gathering. Some people take it for granted, but if your developers aren't in a position of power in your company, they can't insist on good requirements for software. I've experienced this firsthand. I once worked for a company in which the field engineers brought in a large chunk of the money for the company. They were the "big dick swingers" in the company, if you will, while the software development team was an attempt at the company to automate a lot of the work done by engineers and then sell services with the gathered data. This presented a conflict of interest in that the software would have actively made the company need less engineers as they could do their job faster, but because of the two groups relative status within the company, there was no way for the software team to insist on accurate specifications. The day I walked out of that company I had been asked to rewrite a module of the software for the 3rd time and requested written documentation. I was on the phone with both my manager and the engineer we were supposed to be coordinating with. The engineer flat out refused and told me I didn't need it. I asked him how many years of software dev experience he had. That's one anecdote, but the point is, politics in a company severely affect the performance of developers and not simply because of testing. I honestly think that's an awfully naive view of the world.

jraedisch 12y ago

The mentioned technical paper about assertions states: "The assertions are primarily of two types of specification: specification of function interfaces (e.g. consistency between arguments, dependency of return value on arguments, effect of global state etc.) and specification of function bodies (e.g. consistency of default branch in switch statement, consistency between related data etc.)." from http://research.microsoft.com/pubs/70290/tr-2006-54.pdf http://research.microsoft.com/pubs/70290/tr-2006-54.pdf Is there somebody here, who could elaborate on this, ideally with some small examples?

qznc 12y ago
You want to have examples for assertions? Consistency between arguments: memcpy "The memory areas must not overlap." Dependency of return value on arguments: max returns one of its arguments. r = max(x,y); assert (r == x || r == y); Effect on global state: printf might change the global errno variable.
- jraedisch 12y ago
  So this means embedding small tests into the production code itself while accepting minor performance loss? And you somehow get a feeling for when this will help through experience? I wonder, whether this impacts all programming languages the same.
  - qznc 12y ago
    It is not necessarily about runtime tests. Often assertions are only used for debug builds. Java requires an extra flag to enable assert statements. Alternatively, assertions could be invariants or pre/post conditions for your formal proofs of correctness. I'm too lazy to check the actual paper, if they clarify their understanding. ;)