6 ms·
Analysis of whether unit testing obviates static type checking (2012)
- koito17 3y agoTL;DR: The blogger took 4 Python projects, rewrote them in Haskell, and 3 of them had type errors. Therefore unit tests are not replacements for static typing. --- I don't get how this post says anything about unit testing or static typing at all. Doesn't it just imply that bugs are easier to detect with one method than another method? Most statically typed languages out there have a type system too weak to make the expression 1/0 a compile-time error. But you work around this with linting, unit tests, using a language with a stronger type system, etc.
- l0b0 3y agoYeah, as a big unit testing proponent I'd still definitely say they are not a silver bullet. They simply have a good cost-to-confidence-in-code ratio. So do static types, or type annotated code with strict validation.
- AnimalMuppet 3y agoBut did they have type errors in Python? Or was the type error introduced by translation?
- efarrer 3y agoHi, I'm the author of the blog post and the paper. Every bug that was caught by the Haskel transition was verified to exist in the original Python. If I couldn't reproduce it in the Python, I fixed my translation. More information on my methodology can be found in the paper. https://drive.google.com/file/d/0B5C1aVVb3qRONVhiNDBiNUw0am8/view?resourcekey=0-K61wvljWNvWuiqYkwSUSlQ https://drive.google.com/file/d/0B5C1aVVb3qRONVhiNDBiNUw0am8...
- tome 3y ago> Doesn't it just imply that bugs are easier to detect with one method than another method? Yes, but then how can you say > I don't get how this post says anything about unit testing or static typing at all. It does say something about unit testing and static typing!
- wzdd 3y agoThe blog sets up to address the "claim by proponents of dynamically typed programming languages" that "once you have unit testing static type checking is redundant". This is a strawman and I'm fairly sure it was a strawman in 2012 as well. The more interesting claim, which this blog post doesn't address, is that dynamic typing plus unit testing lets you produce working programs faster, without having to wrestle a type system for things you don't care about (e.g. malformed input leading to crashes when parsing machine-generated, standardised inputs, as is the case for three of the test programs -- keeping in mind that, in Python, a crash due to a type error produces an orderly, if surprising, program exit rather than, for example, memory corruption).
- SkyMarshal 3y ago> The more interesting claim, which this blog post doesn't address, is that dynamic typing plus unit testing lets you produce working programs faster, That's an interesting question for any program or app in the move-fast-break-things domain. But perhaps a suboptimal, and potentially inaccurate, one for high-assurance systems that can't break without losing substantial financial value or lives, and/or for which system-wide refactoring may be necessary as the system evolves. Know your domain, choose your tools and methodologies appropriately.
- Hemospectrum 3y ago> The blog sets up to address the "claim by proponents of dynamically typed programming languages" that "once you have unit testing static type checking is redundant". This is a strawman and I'm fairly sure it was a strawman in 2012 as well. In the past year I've seen at least half a dozen posts here on HN making this exact claim. Now I kind of wish I'd saved a few.
- a1369209993 3y ago> "once you have unit testing static type checking is redundant" I have seen literally that exact claim, word-for-word, from people who it seems reasonable to characterize as (rather zealous) "proponents of dynamically typed programming languages". It might be a weak man[0][1], but it's definitely not a straw man. 0: https://slatestarcodex.com/2014/11/03/all-in-all-another-brick-in-the-motte/ https://slatestarcodex.com/2014/11/03/all-in-all-another-bri... 1: https://slatestarcodex.com/2014/05/12/weak-men-are-superweapons/ https://slatestarcodex.com/2014/05/12/weak-men-are-superweap...
- Pannoniae 3y agoThe better the type system is, the more bugs it prevents, so one has to spend less time unit testing. Sadly, it also means the language is harder to learn, so the rest of us are stuck with horrible, unsafe languages.
- kaashif 3y agoThis hasn't exactly been my experience, Python isn't easier to learn than e.g. Java because of the lack of static typing, it's easier because the language is more expressive. I actually think there is literally no downside to making typing opt out rather than opt in - beginners benefit from the compiler saying they accidentally swapped two arguments around - how could they not? I have helped beginners with untyped Python codebases and it's a lot harder to tell what function parameters are supposed to be than in Java or C++, leaving aside any other considerations.
- JSavageOne 3y agoThe downside is increased short-term dev time and code, and thus increasing rigidity of the code. In most cases this tradeoff is still worth it, but sometimes speed and flexibility is a higher priority, hence why I prefer opt-in
- p1necone 3y agoAnecdotally as someone who writes a lot of Typescript - which has a very expressive type system, most of the time when I need to help someone who's "fighting" with the type system the issue is that they've written a bug but the error message just isn't particularly clear. Which is loads better than them writing a bug and then only discovering it in production. Mentally parsing the type errors is an art though - generally you want to ignore pretty much all but the bottom 1 or 2 lines - the rest is a bunch of very verbose information at a much higher level of abstraction than you care about.
- other_herbert 3y agoThis exactly… I’ve had several devs come to me with things like “react won’t let me do this” … which is really typescript and really it’s that they are doing something wrong in any lang and it is just good news that the type system caught it
- p1necone 3y agoI'd be much more interested in needing less unit tests with stronger typing than avoiding needing stronger typing by writing more unit tests. The former seems like much less work.
- raincole 3y agoIt's from 2012. I feel the general opinion about static typing changed a lot in the past decade. (for reference, it's the same year when TypeScript came out. PEP 484 came out in 2014.)
- kelnos 3y agoCame here to say exactly this. Tests are more code that has to be maintained, and more code that can be wrong. I'm sure we've all encountered a test that didn't actually test what it was supposed to test, or wouldn't actually fail in the cases it was supposed to. Obviously a compiler can't prove your code is correct in every possible way, but any time the compiler can prove something correct without me needing to write a test for it, that's a win. Stronger type systems and static type checking is a great way to do that.
- codethief 3y agoHasn't this long been settled? See e.g. https://www.destroyallsoftware.com/talks/ideology https://www.destroyallsoftware.com/talks/ideology . Spoiler alert: We need both.
- 8organicbits 3y agoThe article was published three years before this talk.
- codethief 3y agoI know but it was resubmitted and upvoted to the Front page. :)
- quickthrower2 3y agoWell "we need both" is obvious to anyone who has written software for any length of time :-) But that said, dynamically typed code is OK for small code bases you can fit in your head. The real question is does "tie yourself in knows" type systems like Haskell have a benefit over "80-20 does it" type systems like Python with type hints.
- seeknotfind 3y agoThe answer is no.
- tuckerconnelly 3y agoYes, but is either going to save you from building the wrong features? Paraphrasing Ed Catmull in his book Creativity Inc, the cost of preventing errors is usually much higher than the cost of fixing them.
- epicureanideal 3y agoI haven’t read that book, so I will, but that sounds like it would only be true if you don’t modify the code repeatedly.
- zarathustreal 3y agoThe answer is yes, obviously, due to the Curry-Howard isomorphism. Likewise static type checking obviates unit testing. Granted, most static type systems aren’t Turing complete so there are some assertions that can’t be made at the type level that can be made at the unit test level, however there are absolutely a handful of languages with type systems that are. Indeed, any language with first-class functions can implement runtime type checking via higher-order predicates and thus obviate the need for unit tests. Is this news?
- Jtsummers 3y agoStatic typing, short of dependent types (and even then you have to use them to their full extent), does not obviate any form of testing unless your programs are trivial. For any non-trivial program, testing is still needed unless you're going to go through formal proofs (which could be embedded into dependent type systems). Otherwise, you can have something which type checks but which still contains logical errors.
- zarathustreal 3y agoYour comment reads like you’re opposing what I said but you’re not actually refuting or addressing any of the points I made..do you have anything to add to the conversation or are you just yelling in the opposite direction?
- efarrer 3y agoHi, I'm the author of the blog post and the paper. I certainly do not argue that static typing obviates unit testing. I agree that both are valuable together. I did however find examples where individual unit tests could be deleted because they didn't test anything that wasn't covered by the static type system. In other words static typing didn't eliminate the need for unit tests (for catching bugs), but it did reduce the number of needed tests (in real world code).
- deleted 3y ago[deleted]
- danielam 3y agoType checking allows the compiler to prove certain properties hold for your code, and so this can be used to catch certain classes of bugs. Unit tests don't have that power. But which classes of bugs are caught depends on the type system. Not all type systems are the same. In most statically typed languages, division is not total, and so a function using division can type check, but also crash the program at runtime. In such cases, unit tests can complement types, so you can profit from both. Furthermore, in most languages, the type system is too coarse to prove the sort of correctness that unit tests try to corroborate. If types are a specification, then a sorting function with the type signature `[Int] -> [Int]` doesn't tell us much. There are, in principle, an infinite number of functions that have that type, most of which are not sorting functions. Unit tests can give us some modest, practical confidence of correctness that such a type system cannot. On the other hand, dependent types do permit us to construct return types that guarantee that some ordering relation holds, so it is possible to make the specification more precise in such cases. Such precision may not always be economical or easy to achieve, however. Type inference can assist us in this task, though. But one of the aspects of types that I find useful is the documentation they provide. In a dynamic language, you often have to guess what the types of functions are by reading the implementation, and not just of the function in question, but the functions composing that function. As a code base grows in size, this can become annoying. With code base size, modularity becomes increasingly important, and modularity benefits greatly from clearly defined and stable interfaces. Also, types can enable something like LSP to make suggestions about what can appear in a given place. With type inference, this can be quite powerful, allowing you to build code in an interactive fashion (proof assistants make use of this behavior). Unit tests can also provide value as documentation, in the sense that they can function as tested and working examples of, say, how a library could be used. So, in other words, both types and unit tests have value, and the value they offer will vary depending on the type system and how types are being leveraged in a given situation. There exists an overlap in the practical value provided by types and by unit tests, and where there is overlap, I would say types have the upper hand in principle, if not always in practice.
- magicalhippo 3y ago> But one of the aspects of types that I find useful is the documentation they provide. For me this is one of the core aspects. Self-documenting code makes me so much more productive, and being able to reason about code thanks to the types without having to consult the documentation is very powerful. Any time I work with JavaScript, Python or similar I invariably waste time having to do print(dir(foo)) or similar, just to figure out what the hell some function returned and what I can do with it.
- v3ss0n 3y agoStatic type checking is way a lot easier, help with your code discovery and learning process . Less unit testing, better typing ftw.
- ReflectedImage 3y agoStatic type checking takes a long longer developer time wise and does a much worse job than unit tests.
- steve_adams_86 3y agoI’m surprised anyone thinks types make unit tests redundant. I can know a function returns a value adhering to the correct interface, sure, but that doesn’t tell me if the values are the correct ones given specific inputs. These are different things. I do think a type system dramatically reduces the size and complexity of unit tests, though. I don’t miss writing tests to check types.
- seanmcdirmid 3y agoIt could happen given a sufficiently powerful type system…a type system no one would probably want to use (given that you would write more type-finessing code than real code, you might as well be programming with a theorem prover).
- LispSporks22 3y agoWhen I was a Haskell noob there was this weird meme being propagated that if it compiles, it’s correct.
- tome 3y agoIt's possible you heard that (there are 227 hits for that phrase on Google) but more likely you heard "if it compiles, it works" (35,000 hits). "If it compiles, it’s correct" is obviously false, and no one should be saying it about Haskell code. "If it compiles, it works" is true in certain contexts and for suitably weak definition of "works" (such as, behaves in a sensible way and provides a sensible answer, just not necessarily a correct one) and is an experience attested to by many, many Haskell programmers.
- kelnos 3y agoTypes make some tests redundant. I've seen unit tests in Python code bases that were simply there to make sure that a function did something sane if a value of the wrong type was passed to it. I think it's absolutely certain that, to achieve the same level of confidence in your code, you need more unit tests for, say, a Python code base, than for a Java code base.
- Tade0 3y agoI was part of a team maintaining 150k lines of ancient JavaScript. Doing this requires careful engineering, but is otherwise possible, though not recommended. A good friend of mine is still there. I find conversations with him enlightening because there does not exist a mainstream type system which can clearly an concisely express what he's doing there - even though these are fairly mundane things in dynamically typed languages, like arguments with correlated types. I'm a firm believer in exercise as a means of maintaining skill, so I've been doing a hobby project in vanilla JavaScript. Of course I make type errors, but they're usually the sort that makes me go "aw shit, this field is wrong" and move on. Meanwhile I relearned some characteristics of this environment, which are: -Tests are mandatory. -You have to be radically explicit, otherwise you lose track of what you've been doing. -Documentation is the cornerstone of remaining on track in the long run. -The biggest mistakes are not in the types, but, unsurprisingly, the logic behind the code. -General advice, like short-circuiting edge cases, using pure functions and immutable data structures still applies. -As per Grug (https://grugbrain.dev/ https://grugbrain.dev/) - "grug very like type systems make programming easier. for grug, type systems most value when grug hit dot on keyboard and list of things grug can do pop up magic. this 90% of value of type system or more to grug" - wise tell this, me concur Some interesting features of the codebase so far: -There's considerably less code than in my imagined type-annotated version. -I still use JSDoc to label inputs and outputs. -I'm sort of cheating, because I rely heavily on classes, which have the `myObject.constructor.name` property, which in turn serves as runtime type information. -Data structures are flatter - I have e.g. an array (named "path") where even indexes are strings, odd - numbers. There might be a way in TypeScript to express this(and there's definitely one in some über "strongly" typed language that compiles to JS), but my experience is that finding this out would be the perfect nerd snipe for me. I could have an array of objects with proper fields, but these are not initialized in the same place and a "path" ending in a string is also valid, so I would have to deal with the bureaucracy of optional fields. --- Overall you probably don't want to do the same in a team - unless you're doing a group exercise and this is not code meant for production. Static types are to me a whip for those who take the path of least resistance - every developer has such moments, some just more frequently than others. That being said you might want to occasionally try working without the whip cracking above you and see what you create then - my guess is that if you don't, you'll miss out.
- ReflectedImage 3y agoIf you spent a day writing unit tests you will eliminate more bugs than if you spent a day writing types. And that's all that matters. Unit tests are the more effective technique. Yes obviously static typing doesn't catch all the bugs caught by unit tests or vice versa. But it's a meaningless thing to say.
- kristianp 3y agoOriginal title: Unit testing isn't enough. You need static typing too. From 2012.
- smitty1e 3y agoIn the meantime, mypy has come along, inviting another trip through these projects. Have any of the python projects acquired type annotations?
- pfdietz 3y agoNot that unit testing is enough, but the unit testing that these projects happened to do isn't enough. I suggest that testing that's enough to have confidence that a non-trivial program is correct is going to cover just about everything static type checking could find as well (except in dead code, and how does that matter?) Or, to put it another way, if you didn't care enough about the correctness of your code to test it adequately, why would you care that static type checking could find some bugs? You've already determined quality is not too important.
- xupybd 3y agoEnough unit testing will test everything the type system can. However, that is way more work than a good type system.
- chriswarbo 3y ago> Enough unit testing will test everything the type system can Depends on the type system. For example, if we have parametricity then we can constrain possible values quite a lot; e.g. a value with type 'forall t. t -> t' is the identity function (or diverges, if our language is non-total). If we don't have parametricity, such values could do all sorts of shenanigans, e.g. def f[T](x: T): T = x match { case n: BigInt if isOdd(n) && isPerfect(n) => n+1 case _ => x } This acts as the identity function for all values of all types; except if we give it a BigInt which is an odd perfect number. This is probably the identity function, but ensuring there are no edge cases would require our test suite to solve a famously hard problem https://en.wikipedia.org/wiki/Perfect_number#Odd_perfect_numbers https://en.wikipedia.org/wiki/Perfect_number#Odd_perfect_num... (This sort of guarantee is known as a "free theorem", e.g. see https://bartoszmilewski.com/2014/09/22/parametricity-money-for-nothing-and-theorems-for-free https://bartoszmilewski.com/2014/09/22/parametricity-money-f... )