19 ms·
Experiment: Unit testing isn't enough; You need static types, too
- timruffles 14y agoI wish more of these open questions in coding (and everything) had people making such a good effort to research them! There never is a simple answer, or they'd not be an open question. Really inspiring, painstaking piece of work.
- efarrer 14y agoThanks for the compliment, one of my goals is to put the science back in computer science.
- timclark 14y agoInteresting, but worth remembering, as Rich Hickey says, every bug has got past both your unit tests and your type checking.
- barrkel 14y agoThat statement, though, is almost content-free. It's a truism. If you don't have type checking, then every bug has gotten past your unit tests; but some proportion may have been prevented with type checking. And vice versa. The statement doesn't say anything about the value or non-value of unit tests or type checking.
- mike_ivanov 14y agoNo, it's not content-free, at least when put back into context. It's like coffee. If your coffee is shitty (complect), then no amount of sugar (unit tests) or cream (type checking) will make it any better.
- barrkel 14y agoYour coffee analogy also says nothing about whether sugar or cream are better than one another, or if either is necessary. Though I can guarantee you that either sugar or cream with only 0.1% coffee content will still be usable as sugar or cream for other purposes, so I can't even agree with your analogy taken to its limit: that no amount will make the coffee better, because enough will make it usable for other things. In fact, unit testing and static type checking are largely orthogonal to the complecting issue that Rich was getting act. Unit testing and static typing will both increase complexity; they are both ways of making assertions about the behaviour of the code, and for working code, they should both actually be redundant. But that doesn't mean, as a practical issue, that we should do away with either or both. We're fallible. Saying the same thing twice or three times increases the probability of finding something inconsistent if our statements are not representative of the Platonic ideal we're trying to express.
- pbiggar 14y agoSometimes you have to drink the coffee anyway, and cream and sugar do in practice make it much better.
- brohee 14y agoExcept unlike sugar and cream, unit tests and type checking prevent the crappy coffee being served (well, unless you're from the PHP Coffee House where serving an unit test failing brew is perfectly acceptable (https://bugs.php.net/bug.php?id=55439 https://bugs.php.net/bug.php?id=55439)
- Flow 14y agoSpeaking of Clojure, I've recently started to learn it since I like Richs reasoning about state, time, value and so on. His speeches are nothing short of fantastic in my opinion. Anyway, it occurred to me that with Clojure it could easily happen that I trade bugs caused by state problems to bugs cause by type problems.
- drharris 14y agoIt is definitely refreshing to see some actual evidence in something where arguments tend to be based on speculation, experience, or opinion. Now, we just need someone to research Emacs v. Vim, Tabs v. Spaces, etc.
- crazygringo 14y agoDon't forget about braces versus indents, and above all, semicolons or not! :) On second thought, let's forget about them after all...
- Deestan 14y agoAny development environment where there exists a rule on Braces versus Indents is a place I'd stay well away from. Put braces where they make the code readable. Use indents instead where they make the code more readable. And if you do need a rule, make sure it's based on actual need. I.e. start the discussion with "the past two months we've had 4 non-trivial bugs which could have been avoided if we enforced braces.", not have a bunch of people bicker about their Personal Preferences (usually goes by the name "Best Practices").
- simias 14y agoI disagree. I prefer a development environment that settles on a coding style, even if I don't like it, rather than chaos. I find reading code using multiple coding style painful. It also ends the bikeshedding, "that's how we do it there, deal with it".
- Deestan 14y agoSuch binary arguments bother me - Things won't degenerate to complete and utter chaos unless you go and add strict rules for everything. Good developers tend to follow the style of code they're in, and over time converge on a consistent style within projects or at least modules. If they don't respect existing style, you don't have a "lack of rules" problem. You have a lack of education problem. Do code reviews. Point out to Bob that he's making the code less by letting the style alternate. But you're sort of touching on what I meant when you say you don't want "chaos". If you can point at a code file and say "this is chaotic and hard to read - we could improve that by converging on style X or Y.", then we could come to some agreement. Personally, I'd rather deal with slight aberrations in style than having a flaming row and angering half my team.
- glenjamin 14y agoI applaud the effort to try and dissect the problem scientifically. If a whole program has 1 bug due to being implemented with dynamic types over static types, and that bug has gone unnoticed, then it can't be particularly important. The other common proposition is that dynamically typed languages are faster to write in than statically typed languages. If this is true then we need to compare the saving in development time with the cost of the bugs which go undetected. My gut feeling says that this line of analysis is never going to prove that static typing in inherently "better".
- cageface 14y agoWhat? He specifically says that many of the bugs he discovered were exploitable. Just because you missed them doesn't mean some hacker won't.
- Deestan 14y ago> The other common proposition is that dynamically typed languages are faster to write in than statically typed languages. This part is also very tricky, as most of people's hard-earned experience with this is (almost by definition) old. In recent years, type inference has reduced the type-caused slowdown immensely. E.g. in Haskell you can (+) write large statically typed programs without specifying any types at all - they are inferred by the compiler. (+) But please don't.
- johnkchow 14y agoFrom my personal experience, the saving in development time is very much real. I used to work on .NET, and you'd have to program against crazy, non-intuitive patterns in order to have your code "clean" (I hated IoC containers as well as writing all that boilerplate code for Attributes. And for Java, remember that hilarious post about Factory-Factory-Factory patterns http://discuss.joelonsoftware.com/default.asp?joel.3.219431? http://discuss.joelonsoftware.com/default.asp?joel.3.219431?) I'm not saying statically typed languages are bad. Type errors always bite me the ass in Ruby, but it's a small price to pay (IMO) for better maintainability. EDIT: By the way, it sounds like I'm an ignorant dynamically-typed lover. I'm not, I still yearn for those type safety net, but I'm just speaking from a pragmatic perspective.
- pimentel 14y agoHow can you translate from python to a static language, when the code is written for the interfaces, and not types? How willl you translate a function receiving a (possibly custom) iterable, when the function doesn't care about the type, but just whether it implements a next() method?
- sbmassey 14y agoHaskell type classes are sufficiently flexible to represent things like 'does this type have a next method' without having to instrument the actual type.
- pimentel 14y agoBut could that kind of type system be considered similar to the usual Java-like little-flexible static typing? I think I would prefer a static vs dynamic implemented as Java vs Python (since those are usually the subjects on every one of these discussions)
- hythloday 14y agoIt's identical, conceptually. A type class can be thought of as a degenerate kind of Adapter pattern.
- ufo 14y agoALthough the name is confusing type classes are very much like interfaces in Java: you define a set of operations that you want and then instantiate concrete types to that interface. The major differences between type classes and interfaces are: * In Java interfaces the interfaced type is restricted to the first argument (the this). In Haskell interfaces the interfaced type can also appear on return values and arguments. * In Java you need to decide what interfaces to implement when you create your class. Haskell allows interfaces to be implemented for previously existing types. * Java allows for subtyping. You can turn a monomorphic program into a polymorphic one just by creating subclasses while in Haskell you would need to rewrite your code to be explicitly polymorphic.
- 14y ago
- cageface 14y agoI'm convinced that dynamically typed languages are a transitional technology that will be superseded once we develop type systems that are both usefully strict but also flexible. After over ten years working in dynamic languages I'm very happy to have a compiler on my side again.
- gnuvince 14y agoWhat dynamic language were you using and which language are you using now?
- cageface 14y agoI pretty much lived in Rails from 2004-2011 but did a lot of Python and Perl before that. These days it's mostly C++/Obj-C but I'm keeping an eye on Haskell for iOS.
- eta_carinae 14y ago> I'm keeping an eye on Haskell for iOS. Really? Is that even possible? Can't imagine Apple being okay with that.
- gsnedders 14y agoApple's fine with such things… provided you don't let arbitrary code run in your system. If all your VM/runtime runs is what is in the application, it's fine.
- cageface 14y agoThey've significantly relaxed their restrictions on iOS languages. It's not any weirder than running C# binaries via Mono.
- gaius 14y agoSure, GHC is on ARM now.
- 14y ago
- afrozenator 14y agoThe author really needs to be complemented on rewriting swathes of code from Python to Haskell. In Google, in my project, we've had runtime errors in Python code, due to wrongly spelled variables(although that is a different problem), and type error, something a compiler would have caught. Strong type checking is something that I truly like about Haskell and OCaml, I'm reasonably convinced that once my program has passed the typechecker, it is logically correct. Though debugging in Haskell is truly a different ballgame altogether (I'm a Haskell noob). I'll stop here lest this turns into a flame war.
- 16s 14y agopylint can help with misspelled variables and type errors. I started using it recently and love it. I still love my C++ compiler though and would not trade it for anything else.
- riffraff 14y agopyflakes also detects typos easily, and it's quite useful. In languages that don't catch anything you can usually still use tools for static verification. As another example, Java may let you get NullPointerException, but FindBugs detects a lot of those.
- afrozenator 14y agoWow thanks, I didn't know pyflakes. I knew pylint, but will make it more of a point to run it. I only occasionally need to code in Python, but when I have to, it is legacy code that I modify.
- riffraff 14y agoglad to be helpful. Depending on your editor of choice, you can get integrated pyflakes/pyling/flake8, e.g. I use vim and the syntastic plugin which is great. This way, you get a bit of on the fly static checking without needing to remember command line tools/using vcs hooks, which is much more productive.
- kingkilr 14y agoIf you can convert a program from one language to another (which is non-trivially different), in the time it takes to complete your masters, I'm pretty sure it wasn't a very interesting program. Further, the quality of the developers is going to play a large role in how effective any tool (and make no mistake, static typing is a tool) is. This is not intended as a disparaging remark to the authors, but in the 30 seconds I spent reviewing each of these code bases, I was totally unimpressed: none of them seemed to follow PEP8, and several of their test files weren't even unit tests, they were just a random scrip that appeared to exercise a tiny part of the codebase. I therefore conclude that the methodology used in operating this experiment was flawed and, consequently the conclusion cannot be taken as scientifically valid.
- scott_s 14y agoI have not looked at the programs, but programs don't have to be long to be interesting. There are many "interesting" programs under 100 lines of code - and they can be important if they form the kernel of a larger program. Also, the author responds to a similar point in his comments: http://evanfarrer.blogspot.com/2012/06/unit-testing-isnt-enough-you-need.html?showComment=1340076330696#c8509818811145591834 http://evanfarrer.blogspot.com/2012/06/unit-testing-isnt-eno...
- kingkilr 14y agoThe author doesn't really respond to the point, he simply says they were non-trivial in complexity, which may be true, but that doesn't mean they're non-trivial in their dynamacism (is that a word?). Moreover it in no way responds to the claim that these codebases aren't very good.
- lambda 14y agoDoes it matter if "these codebases aren't very good"? Have you ever actually seen a good code base? Most code I've seen has some poorly written corners; something where someone was in a hurry, or didn't know what they were doing, or someone inexperienced with the project started working on it, or the like. The point is, these were real-world codebases with substantial unit tests, and they had type errors that weren't caught by these unit tests. Honestly, in several groups I've worked with, I've had trouble getting people to add unit tests even to code in dynamic languages. A codebase which already has substantial unit tests is likely to be better than average, on that basis alone. This is about saying, in the messy real world, do unit tests actually make type safety obsolete. And the answer is, no, they don't, even code with fairly good test coverage can be improved by adding typechecking. Now, there is the question of whether writing in the dynamic language allowed people to write code faster (it's generally a lot easier to translate correct code once it's written than to write it in the first place); or whether some of the more highly dynamic features of dynamic languages benefit writing or deploying code.
- dmethvin 14y agoThe lack of static types really comes into play not inside a library, but in the interface between the external code and a library. Note that at least a few of the bugs that were found involved invalid API inputs. From the library writer's view it's not a bug because those values are not in the domain of defined behavior. From the caller's view it's a PITA that the library doesn't yell at them when they pass it garbage. Of course, they could find the problem with unit tests but the further up the food chain you get the more scarce unit tests tend to be.
- jknupp 14y agoThe author's interpretation of the argument in favor of dynamic languages seems purposefully naive. I don't think that any proponent of dynamic languages or unit testing claimed that the mere presence of unit tests guaranteed bug free code or that it was impossible to have type related errors at run time if you have unit tests. It's a more nuanced argument asserting that the benefit to programmer productivity when using dynamic languages outweighs the cost of potential type related errors not possible in a statically typed language. Whether it has merit or not is the question I hoped this paper would answer.
- soc88 14y agoI wonder how long it takes until the first one picks Java to argue against typed languages.
- AngryParsley 14y agoI think most people would agree that these two circles overlap on a Venn diagram: ( Bugs found by unit tests ( ) Bugs found by type-checking ) The disagreement is how much. Also, type-checking is free[1], while unit tests have to be manually written. I'm glad someone spent a lot of time trying to answer this question, but I don't think it will affect my choice of language in any new project. I like to write code in the languages I like, and bugs be damned. 1. A common argument is that static-typed languages slow development. I'm not touching that land-mine.
- swannodette 14y agoVia Philip Wadler, http://wadler.blogspot.com/2011/09/experiment-about-static-and-dynamic.html http://wadler.blogspot.com/2011/09/experiment-about-static-a...
- akkartik 14y agoOn HN: http://news.ycombinator.com/item?id=4082775 http://news.ycombinator.com/item?id=4082775
- Goladus 14y ago> 1. A common argument is that static-typed languages slow development. I'm not touching that land-mine. If you claim type-checking is free, the counter-argument is not that it "slows development." The counter is that type-checking is not free because it incurs measurable costs. You may sacrifice dynamic features, you may have to add declarations and type casts-- these are all costs whether they "slow development" or not. (Simple solution: don't waste time trying to claim type-checking is free and just focus on the benefits.)
- AngryParsley 14y agoI would feel bad editing my comment now that you've replied, but my original intent was to say that type-checking is "free." At some point the quotes were lost. To get as close to the land-mine as I am willing: I like C. I like Python. I like Ruby. But most of all, I like using the right tool for the job.
- davesims 14y agoThe sample size of this study is statistically too insignificant to draw the conclusions given. The counterfactuals to the claims of dynamic type advocates were already true and provable, so even if the sample size of the codebases studied were statistically significant (not to mention vetted for quality of unit test as well as coverage) the conclusions are nevertheless trivial. In addition, the hidden assumption is that all static and dynamic typing are created equal, i.e., since Haskell is statically typed and Haskell appears to have caught Python bugs that unit tests did not, therefore Java will catch bugs in a Ruby codebase, C++ will catch bugs in a JavaScript codebase, etc. Of course this assumption is gratuitous. Haskell in particular has a specific sort of type checking that is far different from Java's or C++'s, for instance. Further, not all dynamic systems are created equal. Ruby, for instance, I think can be shown to require fewer lines of code to achieve similar functionality to, for instance, Java. Fewer lines of code, should in principle mean fewer opportunities for defects. Dynamic languages with metaprogramming features like Ruby's or Smalltalk's should in principle be able to eliminate more code duplication than an environment like C++. This aspect of dynamic languages should be taken into account, again with a statistically significant sample size, and weighed against bugs caught by static typing. The study is interesting as a preliminary investigation, but the conclusions should have been much more modest, proportionate to both the sample size, in terms of % of production codebases and the extremely important idiosyncratic nature of Haskell vs. other static typed environments. Something like: "The study has shown Haskell's type system will catch some bugs not caught in an otherwise well-covered Python codebase. These bugs could in theory have been caught by unit tests, therefore it is recommended that when using a dynamic language, more care must be taken to cover these types of bugs." That would have been a more appropriate and modest conclusion, consistent with the data, than the sweeping generalization "You need Static Typing."
- papsosouid 14y ago>In addition, the hidden assumption is that all static and dynamic typing are created equal, i.e., since Haskell is statically typed and Haskell appears to have caught Python bugs that unit tests did not, therefore Java will catch bugs in a Ruby codebase, C++ will catch bugs in a JavaScript codebase, etc. That assumption isn't hidden, it is made up. By you. The question was "can static typing catch bugs that made it past a decent (and common) test suite". The answer to that can drive interest in static typing, and thus more language with useful static type systems. Just because java has a crappy type system, doesn't mean we should be content with that.
- swannodette 14y agoStatic types or static analysis? * KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs http://llvm.org/pubs/2008-12-OSDI-KLEE.html * Erlang Dialyzer http://www.erlang.org/doc/man/dialyzer.html * Datalog based systems http://www.cse.msu.edu/~cse914/Overheads/mmcgill-java-static-race-detector.pdf If you want static analysis hard coded into your language - what feature set do you want to support? The following support different styles of programming thus have very different static type systems. * Standard ML / OCaml * Haskell * Scala * Typed Racket * Qi And there's still the question that some kinds of extremely useful programs are very difficult to write in popular languages with strong static typing. miniKanren, a flexible embedding of Prolog and Constraint Logic Programming into Lisp, comes to mind here. I've seen versions of miniKanren written in Haskell and it abandons the most powerful feature of miniKanren - it can be trivially applied back on the language it is written in!
- deleted 14y ago[deleted]
- IsTom 14y agoDialyzer is not very good. We've been using it, but it rarely catches anything non-trivial. On the other hand Haskell has a better support for unit tests than many of dynamic languages.
- gtani 14y agorelated: http://www.artima.com/forums/flat.jsp?forum=106&thread=335110 http://www.artima.com/forums/flat.jsp?forum=106&thread=3... http://tratt.net/laurie/tech_articles/articles/a_modest_attempt_to_help_prevent_unnecessary_static_dynamic_typing_debates http://tratt.net/laurie/tech_articles/articles/a_modest_atte... http://www.tbray.org/ongoing/When/201x/2011/12/27/Type-Systems http://www.tbray.org/ongoing/When/201x/2011/12/27/Type-Syste... http://blog.jcoglan.com/2011/12/29/black-box-criteria/ http://blog.jcoglan.com/2011/12/29/black-box-criteria/ http://www.stanford.edu/class/cs242/slides/Types.ppt http://www.stanford.edu/class/cs242/slides/Types.ppt
- eta_carinae 14y agoThe advantage of statically typed languages has a lot less to do with tests and everything to do with tools. IDE's can do very little when no type information is available, and most automatic refactorings require human supervision when performed on dynamically typed languages (read this for details: http://beust.com/weblog/2006/10/01/dynamic-language-refactoring-ide-pick-one/ http://beust.com/weblog/2006/10/01/dynamic-language-refactor... ).
- thebluesky 14y agoTooling and performance are two largest advantages. Disadvantage is more verbosity, but it's a trade-off. With dynamically typed languages I find that you still need to worry about types, but you have to trace through the code to figure out what type a particular variable is (esp. if you're not the only one working on a project). In a statically typed language that information is readily available.
- Peaker 14y agoThe claim static typing adds verbosity is a common fallacy. Haskell code is more concise than most dynamically typed languages.
- papsosouid 14y ago>Disadvantage is more verbosity How so? Java is verbose, but that isn't due to being statically typed, it is just a verbose language. Statically typed languages with type inference are no more verbose than dynamically typed languages.
- gnuvince 14y ago> Disadvantage is more verbosity, but it's a trade-off. When I was learning Haskell, I wrote a program to count the value of a cribbage hand. The code was half the length of an equivalent JavaScript program. The expressiveness of a language has more to do with the length of a resulting program than static or dynamic typing.
- eta_carinae 14y ago> Disadvantage is more verbosity Type inference cuts down the verbosity tremendously.
- debacle 14y agoThe better programmer you are, the less you need static types. But a group of ten programmers create a codebase that is only as good as the worst programmer's code.
- simias 14y agoNo true scotsman much?
- debacle 14y agoI don't know how you could contrive what I said to be cognitive dissonance. If you have a team of N good programmers, you can probably write without static typing. If you have a team of N-1 good programmers and 1 bad programmer, you should really use a language static typing.
- simias 14y agoI just read the first part of your comment as "no true programmer would need no filthy static type checking". Sorry if I misunderstood. I find it hard to correlate use of static typing with development experience, that's all.
- debacle 14y agoIf you are a good programmer, you don't need typing, you don't need interfaces, you don't need many things that make bad programmers better programmers. You might still use them, and there's nothing entirely wrong with that, but you probably don't need them. This is why so many people have problems writing JavaScript.
- dllthomas 14y agoEven if you're a good programmer, typing and interfaces make refactoring quicker and typos easier to find. This is why so many people have problems writing in JavaScript.
- njharman 14y ago> "frequently cited claim by proponents of dynamically typed programming languages that static typing was not needed for detecting bugs in programs" Who says that? There's trade offs, multitudes, in choosing paradigm / language. It's never so black and white (except for academics, and twits who like to argue more than code) I did not read paper, don't have time for 60 pages of pointlessness.
- zopa 14y ago> "Who says that?" http://news.ycombinator.com/item?id=4137283 http://news.ycombinator.com/item?id=4137283 comes pretty close to making that claim, and that's just on this page. > "There's trade offs, multitudes, in choosing paradigm / language. It's never so black and white" Sometimes, it is. Most people don't use COBOL anymore, with reason; better languages came along. Programming is still a new field in the scheme of things. It would be strange if our languages were perfectly optimized, with no room for improvement without offsetting costs. > "twits who like to argue more than code" Some of us like to do both :) But go program; we won't stop you...
- efarrer 14y agoI agree there are generally trade offs, that's essentially the results of my "60 pages of pointlessness" paper. If you did have time to read the paper you would notice a reference to this book http://my.safaribooksonline.com/book/software-engineering-and-development/9781590595008/bruce-eckel-strong-typing-vsdot-strong-testing/bruce_eckel_colon_strong_typing_vs http://my.safaribooksonline.com/book/software-engineering-an... that is an agument for unit testing instead of static typing. I think we need to use the scientific method in computer science and not just base our ideas of intuition, belief or absolutes like "It's never so black and white".
- jhrobert 14y agoThere is a case for "optional typing". It's speed. I guess everyone agree on this one. Better than "strict" typing, sometimes "hints" about the type would be enough. ie: "ii often integer" versus "ii" or "ii integer" From then on, those who favor strict typing would be happy, those who favor dynamic typing would be happy and those who favor "efficiency" (whatever this means) would be happy. Let's close this silly debate.
- 6ren 14y agoRegarding bugs, I think it's clear that static typing does detect more errors - the issue is whether it's worth the trouble. I think a more significant difference is the inertia of the codebase, how difficult it is to change. Static types make it harder to evolve interfaces; but unit tests make it even harder. But both the above are just fiddling: the real advantage of static typing is runtime speed; the advantage of dynamic types is development productivity. In the history of programming, the latter always wins.
- Peaker 14y ago> But both the above are just fiddling: the real advantage of static typing is runtime speed; the advantage of dynamic types is development productivity. In the history of programming, the latter always wins Have you used Haskell? Speed is just a (nice) bonus. Type safety is not about speed at all. And I am much more productive in Haskell than I could be in a dynamically typed language. I can turn the code base inside out, and trust the type system to guide me to everywhere that needs to be fixed. When I compile it, it almost always works. I barely have to test anything.
- dspeyer 14y agoFor the cases I was easily able to count, this is one bug per thousand lines of code. I realize that's a handwavy metric, but it's enough to say we're not talking about a ton of bugs. On the other hand, successful conversion of these codebases is a very interesting result. Apparently static, at least static as sophisticated as Haskell, actually can express most of the idioms in standard python. This surprises me, as I've never seen a C++ program of significant complexity that didn't resort to void*s somewhere.
- gouranga 14y agoSlightly different view here. I think the nirvana of typing is hybrid static/dynamic and it hit us with visual basic 6. I don't think anyone really noticed it though. It supports traditional type checking by the compiler, runtime type inference and dynamic typing without boxing. Each case can be chosen at will. It supports all theoretical programs, supports unit testing and runtime assertions. The least buggy software I've seen over the years was written in vb6 (by professionals, not the crap that haunts the web). Now I'm not saying we should all switch to vb6 but some of the ideas may be worth investigating.
- deleted 14y ago[deleted]
- petercooper 14y agoOn a related topic, another interesting paper: http://www.cs.washington.edu/education/courses/cse590n/10au/hanenberg-oopsla2010.pdf http://www.cs.washington.edu/education/courses/cse590n/10au/...
- anamax 14y agoIt's almost always the case that applying a new set of tests finds more bugs so this experiment doesn't actually prove its conclusion.