8 ms·

The sample size of this study is statistically too insignificant to draw the conclusions given. The counterfactuals to the claims of dynamic type advocates were

The sample size of this study is statistically too insignificant to draw the conclusions given. The counterfactuals to the claims of dynamic type advocates were already true and provable, so even if the sample size of the codebases studied were statistically significant (not to mention vetted for quality of unit test as well as coverage) the conclusions are nevertheless trivial. In addition, the hidden assumption is that all static and dynamic typing are created equal, i.e., since Haskell is statically typed and Haskell appears to have caught Python bugs that unit tests did not, therefore Java will catch bugs in a Ruby codebase, C++ will catch bugs in a JavaScript codebase, etc. Of course this assumption is gratuitous. Haskell in particular has a specific sort of type checking that is far different from Java's or C++'s, for instance. Further, not all dynamic systems are created equal. Ruby, for instance, I think can be shown to require fewer lines of code to achieve similar functionality to, for instance, Java. Fewer lines of code, should in principle mean fewer opportunities for defects. Dynamic languages with metaprogramming features like Ruby's or Smalltalk's should in principle be able to eliminate more code duplication than an environment like C++. This aspect of dynamic languages should be taken into account, again with a statistically significant sample size, and weighed against bugs caught by static typing. The study is interesting as a preliminary investigation, but the conclusions should have been much more modest, proportionate to both the sample size, in terms of % of production codebases and the extremely important idiosyncratic nature of Haskell vs. other static typed environments. Something like: "The study has shown Haskell's type system will catch some bugs not caught in an otherwise well-covered Python codebase. These bugs could in theory have been caught by unit tests, therefore it is recommended that when using a dynamic language, more care must be taken to cover these types of bugs." That would have been a more appropriate and modest conclusion, consistent with the data, than the sweeping generalization "You need Static Typing."

papsosouid 14y ago
>In addition, the hidden assumption is that all static and dynamic typing are created equal, i.e., since Haskell is statically typed and Haskell appears to have caught Python bugs that unit tests did not, therefore Java will catch bugs in a Ruby codebase, C++ will catch bugs in a JavaScript codebase, etc. That assumption isn't hidden, it is made up. By you. The question was "can static typing catch bugs that made it past a decent (and common) test suite". The answer to that can drive interest in static typing, and thus more language with useful static type systems. Just because java has a crappy type system, doesn't mean we should be content with that.
- davesims 14y ago
  > That assumption isn't hidden, it is made up. By you. Not at all. The assumption is clearly implied by the conclusion of the study, which makes an unwarranted equivalence of all languages that have 'static type checking': "The translation of these four software projects from Python to Haskell proved to be an effective way of measuring the effects of applying static type checking to unit tested software." > Just because java has a crappy type system, doesn't mean we should be content with that. I don't know what this means. If the study was meant to comprehend such a broad category as 'static type systems,' and from the explicit language of the study, it clearly was, then absolutely Java must necessarily be included. Otherwise, the study, as I noted, should have restricted its conclusions to a scope of Haskell vs. Python, with at most modest and well-qualified statements regarding the broader implications of static vs. dynamic in general.
  - papsosouid 14y ago
    >which makes an unwarranted equivalence of all languages that have 'static type checking': No it doesn't. Read what you quoted, it says nothing even remotely resembling "this benefit applies to all languages with static typing". It is testing static typing, not a specific language. It uses the best static typing system to do so. You are entirely inventing the notion that this must then apply to java. > If the study was meant to comprehend such a broad category as 'static type systems,' and from the explicit language of the study, it clearly was, then absolutely Java must necessarily be included No it mustn't. Comparing the best of dynamic vs the best of static is a useful test. Just as nobody is complaining they didn't use a worse language than python, it makes no sense to complain they didn't use a worse language than haskell. You don't draw conclusions about the potential of X by examining the worst example of X possible.
    - cantankerous 14y ago
      This. Not all statically typed languages are created equal. Java's type system is old and is not state of the art. I wish people would stop using it as a straw man when anybody brings up static typing. Java was state of the art 20 years ago, but it's definitely not the case any more.
deleted 14y ago
[deleted]
davesims 14y ago
I think I should clarify what I meant by "the conclusions are nevertheless trivial." Let's look at the key statement in the conclusion of the study: "Based on these results, the conclusion can be reached that while unit testing can detect some type errors, in practice it is an inadequate replacement for static type checking." As I've already pointed out, this seems to me an ambitious and over-reaching conclusion, given the scope of the study. But, equally important, it is simply an example of something that was already provable. It should be axiomatic that in principle automatically-generated validation like that provided by static typing should in theory be able to catch type errors not caught manually in a dynamic context, either for reasons of human oversight or human error. In other words, it seems to me that all that has been done here, is to provide a few concrete examples of what was already true and uncontroversial: auto-generated coverage of specific types of validations can be more comprehensive than some human beings will be in some environments and contexts. It has not shown that the perceived benefits of dynamic typing with good unit tests are outweighed by this fact, nor that, statistically-speaking, errors of this type are common enough to warrant a preference of static typing over dynamic typing with unit tests in all contexts.
wissler 14y ago
You make a good point -- I don't think any statistical study will ever be able to show that static typing is better. I do think that a rational argument can show it, but my argument is too long to fit into the margin.
- scott_s 14y ago
  I think that a study over a broad set of applications of considerable complexity could provide enough statistical evidence that most people would be comfortable coming to a conclusion. That study, though, would take a very large effort. Large enough that it may never be done.
ScottBurson 14y ago
I see both sides of this argument. The OP -- at least in this blog post; I haven't read the paper -- spends most of his time talking about how he's demonstrated the insufficiency of unit testing. For the purpose of that argument, it really doesn't matter that he used Haskell as opposed to some other type checker. It's only in the last two sentences of his "Conclusion" section that he turns the argument around, and here is where he oversteps: While unit testing does catch many errors it is difficult to construct unit tests that will detect the kinds of defects that would be programatically detected by static typing. The application of static type checking to many programs written in dynamically typed programming languages would catch many defects that were not detected with unit testing[...] Clearly, this is overbroad. For starters, he should have used "could" in place of "would". And it wouldn't have been a bad time to remind the reader that Haskell's type system differs from those of other statically typed languages with which the reader may be more familiar. I don't quite agree, though, that the conclusion is "trivial". Maybe I'm just out of touch, but I wasn't aware of a good test of how true the dynamic argument was in practice, as opposed to theory -- particularly claim #2.
efarrer 14y ago
I (the author) appreciate the feedback. I believe that many of your criticisms are addressed in the actual paper. First of all I completely agree that my sample size is too small for a conclusive proof. I mention in the paper that I hope that others will try and replicate this experiment on other pieces of software. I do think it's appropriate when conducting an experiment to publish a conclusion, not that the experiment will constitute proof (or an established scientific theory), but as a conclusion to the study that others can try to confirm or refute. I also mention in the paper that it would be beneficial to conduct this experiment using different type systems for the reasons that you stated above. The argument against static typing that I was testing didn't mention any particular type system nor any particular dynamically typed language, it was a general argument that stated that unit testing obviated static typing. Because the argument was so general and absolute I felt that any static type system that could be shown to expose bugs that were not caught by unit testing would be enough to refute the argument. I was not trying to prove that any type system would catch bugs not found by any unit tested software. The paper also points out that I'm trying to see whether unit testing obviates static typing in practice, in theory you could implement a poor mans type checker as unit tests, but my experiment was focused on whether in practice unit testing obviates static typing. Finally I believe that my conclusion in the paper was at least a bit more modest than that of the blog post. The lack of apparent modesty in the blog post was caused more by a lack of ability on my part to accurately summarize than an inflated sense of accomplishment and self importance.
- davesims 14y ago
  Thanks for the response! I appreciate the effort you went to here, this was no small task you set yourself to. I appreciate the clarification. I think now I see better where your emphasis was: the purpose of the paper was to refute an argument, and of course the level of burden of proof is different and far less in that case. I think this misunderstanding on my part is what caused me to call the conclusions 'trivial' -- too strong and dismissive language on my part anyway. The irony is, you were attempting to do to the unit-testing-is-sufficient argument what I was attempting to do to what I assumed yours was: provide one counter-example to falsify a broad and generalized thesis. That said, I think I would have liked to have seen your original unit-testing-is-sufficient argument punched up and qualified into something a little more reasonable and real-world. As you stated the argument, it seems like a straw man to me. It seems one could reduce your version of the argument to something like: "Dynamic languages with unit test coverage will always catch errors that statically-typed environments will." And of course this is far too broad and unqualified a statement, and that is precisely why all you needed was one counter-factual to refute it. You didn't even need a handful of Python programs, or 9 or 20 or 100 errors to prove your point. You only needed one, as you stated above. This is why the burden of proof for your thesis was so small, but also why, in my opinion, even with that reduced scope and more modest conclusion, we haven't really learned much. As someone who has spent most of my career in statically-typed environments and the last 6 years or so mostly in dynamic environments, and also as someone who has made something like the argument you were attempting to refute, I have to say I would definitely never have made such a brittle and unqualified statement as the one you refuted in your paper. To put it more directly, I think I'm probably a poster-child for the kind of developer you were aiming your thesis at, and I don't feel that my perspective was adequately or reasonably represented. More importantly, having looked at the examples given in your paper, I may have learned a bit about the kinds of errors that Haskell can catch automatically that some coders might miss in a dynamic environment, but not much useful to me in my everyday work context. I think a more reasonable version of the argument, but more qualified and therefore requiring a far larger sampling of code to prove or refute, would be something like: "Programs written in a dynamic language with adequate or near-100 percent unit test coverage, are no more prone to defects than programs written in a statically-typed language with a comparable level of unit test coverage." I agree this is a very important conversation to have, and again kudos to the work you put in here. Obviously people have strong opinions both directions, and the discussion, however heated at various moments, is an important one, so thanks for this!

papsosouid 14y ago

>In addition, the hidden assumption is that all static and dynamic typing are created equal, i.e., since Haskell is statically typed and Haskell appears to have caught Python bugs that unit tests did not, therefore Java will catch bugs in a Ruby codebase, C++ will catch bugs in a JavaScript codebase, etc. That assumption isn't hidden, it is made up. By you. The question was "can static typing catch bugs that made it past a decent (and common) test suite". The answer to that can drive interest in static typing, and thus more language with useful static type systems. Just because java has a crappy type system, doesn't mean we should be content with that.

davesims 14y ago
> That assumption isn't hidden, it is made up. By you. Not at all. The assumption is clearly implied by the conclusion of the study, which makes an unwarranted equivalence of all languages that have 'static type checking': "The translation of these four software projects from Python to Haskell proved to be an effective way of measuring the effects of applying static type checking to unit tested software." > Just because java has a crappy type system, doesn't mean we should be content with that. I don't know what this means. If the study was meant to comprehend such a broad category as 'static type systems,' and from the explicit language of the study, it clearly was, then absolutely Java must necessarily be included. Otherwise, the study, as I noted, should have restricted its conclusions to a scope of Haskell vs. Python, with at most modest and well-qualified statements regarding the broader implications of static vs. dynamic in general.
- papsosouid 14y ago
  >which makes an unwarranted equivalence of all languages that have 'static type checking': No it doesn't. Read what you quoted, it says nothing even remotely resembling "this benefit applies to all languages with static typing". It is testing static typing, not a specific language. It uses the best static typing system to do so. You are entirely inventing the notion that this must then apply to java. > If the study was meant to comprehend such a broad category as 'static type systems,' and from the explicit language of the study, it clearly was, then absolutely Java must necessarily be included No it mustn't. Comparing the best of dynamic vs the best of static is a useful test. Just as nobody is complaining they didn't use a worse language than python, it makes no sense to complain they didn't use a worse language than haskell. You don't draw conclusions about the potential of X by examining the worst example of X possible.
  - cantankerous 14y ago
    This. Not all statically typed languages are created equal. Java's type system is old and is not state of the art. I wish people would stop using it as a straw man when anybody brings up static typing. Java was state of the art 20 years ago, but it's definitely not the case any more.

deleted 14y ago

[deleted]

davesims 14y ago

I think I should clarify what I meant by "the conclusions are nevertheless trivial." Let's look at the key statement in the conclusion of the study: "Based on these results, the conclusion can be reached that while unit testing can detect some type errors, in practice it is an inadequate replacement for static type checking." As I've already pointed out, this seems to me an ambitious and over-reaching conclusion, given the scope of the study. But, equally important, it is simply an example of something that was already provable. It should be axiomatic that in principle automatically-generated validation like that provided by static typing should in theory be able to catch type errors not caught manually in a dynamic context, either for reasons of human oversight or human error. In other words, it seems to me that all that has been done here, is to provide a few concrete examples of what was already true and uncontroversial: auto-generated coverage of specific types of validations can be more comprehensive than some human beings will be in some environments and contexts. It has not shown that the perceived benefits of dynamic typing with good unit tests are outweighed by this fact, nor that, statistically-speaking, errors of this type are common enough to warrant a preference of static typing over dynamic typing with unit tests in all contexts.

wissler 14y ago

You make a good point -- I don't think any statistical study will ever be able to show that static typing is better. I do think that a rational argument can show it, but my argument is too long to fit into the margin.

scott_s 14y ago
I think that a study over a broad set of applications of considerable complexity could provide enough statistical evidence that most people would be comfortable coming to a conclusion. That study, though, would take a very large effort. Large enough that it may never be done.

ScottBurson 14y ago

I see both sides of this argument. The OP -- at least in this blog post; I haven't read the paper -- spends most of his time talking about how he's demonstrated the insufficiency of unit testing. For the purpose of that argument, it really doesn't matter that he used Haskell as opposed to some other type checker. It's only in the last two sentences of his "Conclusion" section that he turns the argument around, and here is where he oversteps: While unit testing does catch many errors it is difficult to construct unit tests that will detect the kinds of defects that would be programatically detected by static typing. The application of static type checking to many programs written in dynamically typed programming languages would catch many defects that were not detected with unit testing[...] Clearly, this is overbroad. For starters, he should have used "could" in place of "would". And it wouldn't have been a bad time to remind the reader that Haskell's type system differs from those of other statically typed languages with which the reader may be more familiar. I don't quite agree, though, that the conclusion is "trivial". Maybe I'm just out of touch, but I wasn't aware of a good test of how true the dynamic argument was in practice, as opposed to theory -- particularly claim #2.

efarrer 14y ago

I (the author) appreciate the feedback. I believe that many of your criticisms are addressed in the actual paper. First of all I completely agree that my sample size is too small for a conclusive proof. I mention in the paper that I hope that others will try and replicate this experiment on other pieces of software. I do think it's appropriate when conducting an experiment to publish a conclusion, not that the experiment will constitute proof (or an established scientific theory), but as a conclusion to the study that others can try to confirm or refute. I also mention in the paper that it would be beneficial to conduct this experiment using different type systems for the reasons that you stated above. The argument against static typing that I was testing didn't mention any particular type system nor any particular dynamically typed language, it was a general argument that stated that unit testing obviated static typing. Because the argument was so general and absolute I felt that any static type system that could be shown to expose bugs that were not caught by unit testing would be enough to refute the argument. I was not trying to prove that any type system would catch bugs not found by any unit tested software. The paper also points out that I'm trying to see whether unit testing obviates static typing in practice, in theory you could implement a poor mans type checker as unit tests, but my experiment was focused on whether in practice unit testing obviates static typing. Finally I believe that my conclusion in the paper was at least a bit more modest than that of the blog post. The lack of apparent modesty in the blog post was caused more by a lack of ability on my part to accurately summarize than an inflated sense of accomplishment and self importance.

davesims 14y ago
Thanks for the response! I appreciate the effort you went to here, this was no small task you set yourself to. I appreciate the clarification. I think now I see better where your emphasis was: the purpose of the paper was to refute an argument, and of course the level of burden of proof is different and far less in that case. I think this misunderstanding on my part is what caused me to call the conclusions 'trivial' -- too strong and dismissive language on my part anyway. The irony is, you were attempting to do to the unit-testing-is-sufficient argument what I was attempting to do to what I assumed yours was: provide one counter-example to falsify a broad and generalized thesis. That said, I think I would have liked to have seen your original unit-testing-is-sufficient argument punched up and qualified into something a little more reasonable and real-world. As you stated the argument, it seems like a straw man to me. It seems one could reduce your version of the argument to something like: "Dynamic languages with unit test coverage will always catch errors that statically-typed environments will." And of course this is far too broad and unqualified a statement, and that is precisely why all you needed was one counter-factual to refute it. You didn't even need a handful of Python programs, or 9 or 20 or 100 errors to prove your point. You only needed one, as you stated above. This is why the burden of proof for your thesis was so small, but also why, in my opinion, even with that reduced scope and more modest conclusion, we haven't really learned much. As someone who has spent most of my career in statically-typed environments and the last 6 years or so mostly in dynamic environments, and also as someone who has made something like the argument you were attempting to refute, I have to say I would definitely never have made such a brittle and unqualified statement as the one you refuted in your paper. To put it more directly, I think I'm probably a poster-child for the kind of developer you were aiming your thesis at, and I don't feel that my perspective was adequately or reasonably represented. More importantly, having looked at the examples given in your paper, I may have learned a bit about the kinds of errors that Haskell can catch automatically that some coders might miss in a dynamic environment, but not much useful to me in my everyday work context. I think a more reasonable version of the argument, but more qualified and therefore requiring a far larger sampling of code to prove or refute, would be something like: "Programs written in a dynamic language with adequate or near-100 percent unit test coverage, are no more prone to defects than programs written in a statically-typed language with a comparable level of unit test coverage." I agree this is a very important conversation to have, and again kudos to the work you put in here. Obviously people have strong opinions both directions, and the discussion, however heated at various moments, is an important one, so thanks for this!