7 ms·

How many of the <fixnum> benefits of TDD and unit tests are achievable through the use of sufficiently advanced type systems? Disclaimer: I'm not using advanced

by z0r 13y ago

How many of the <fixnum> benefits of TDD and unit tests are achievable through the use of sufficiently advanced type systems? Disclaimer: I'm not using advanced type systems to replace unit tests myself, but the more I read about them the more I suspect that most of the tests I've ever worked with (a small sample to be sure) could be obviated by pushing the logic they test for into structured types.

prawks 13y ago
This is one thing that has irked me a little bit as I've gotten more involved with TDD and unit testing in general: the logic in many of my unit tests is near duplicate logic that is implemented within my application. My thought is that the general advantage of TDD is that you are thinking about your eventual implementation from the outside. You write tests in a different mindset based on expectations, rather than when you are implementing the production code where you are more concerned with the details of logic, etc. Unit tests also tend to be a bit more readable than the actual implementation logic, and so act as a nice form of documentation as a side-effect. In this way, test-first development provides a more gradual transition between a specification or design and the end implementation, which results in fewer missed requirements, allows for easy refactoring, etc. This is of course only related to TDD however, not unit testing in general. Hence, it still irks me that many of my tests look so similar to the logic they are testing, and thus aren't really testing for correctness at all.
- jacquesm 13y ago
  Think of unit testing as auditing the contract between two pieces of code right at the interface by replacing one of the two pieces of code (usually the client or caller but you can replace the code that provides the response just as easily) by a program that exercises the other layer. Like that you build your confidence layer by layer and with every set of tests you verify that you are still building on solid ground instead of quicksand. That's why you test at the interface level, it allows you to swap out the implementation without having to re-work all the test code.
- tieTYT 13y ago
  I went through a phase of this when I started writing automated tests. It occurred when I started using a mocking library for the first time. The library was so advanced I could basically test my implementation instead of my api. EG: "first you call foo.bar(), then you call foo.baz()" As a result, my test and my system under test looked very similar. I'm not sure if you're in the same situation I was in, but let me tell you: My situation was an anti-pattern. All I was testing was that my code does what it does. I wasn't testing that my code was working. Make sure you're not making the same mistake. If you are, a symptom would be that lots of tests fail when you're only refactoring.
  - prawks 13y ago
    Not quite, but I get what you're saying for sure. I think jacquesm said it well in his reply to me that you're more testing the "contracts between two pieces of code right at the interface", and it shouldn't be any deeper than that to avoid what you're describing. The symptom of many tests failing during refactoring is something I'll be keeping in mind, however.
nahname 13y ago
Some issues are removed with a type system and wouldn't require writing tests to verfiy (the compiler would be doing this for you). I wouldn't take that fact as being in favor of using a type system, but more of a reminder that using a dynamic language without tests is reckless.
- pauloortins 13y ago
  Type testing can be replaced by static types but you still have to test all the others stuffs in your code.
  - cgag 13y ago
    I just want to note that with an advanced type system, "type testing" can cover a lot more than people seem to give it credit for. An example I saw someone give on here the other day was that he had the types for his red black tree set up so that code that produced an unbalanced tree would never compile.
    - Skinney 13y ago
      Link? How would a type system be able to ensure that both sides of a rb tree is balanced? :S
      - deleted 13y ago
        [deleted]
      - cgag 13y ago
        https://news.ycombinator.com/item?id=5987527 https://news.ycombinator.com/item?id=5987527 I'm just getting into Haskell myself and I can't really explain those 10 lines myself (I also haven't implemented a red black tree). I watched a talk by Yaron Minksy the other day that also talked about the idea that you should "make illegal states unrepresentable" that was really good: http://vimeo.com/14313378 http://vimeo.com/14313378 (he starts talking about that at around ~17:50)
      - 13y ago
tmoertel 13y ago
Even with advanced type systems, the sweet spot will be a mix of types and tests. With types, you can prove many claims, including some you can't test for. With tests, you can gather evidence to support the claims you can't easily prove with types. So you need both. Elaboration: With advanced type systems you can prove that your code satisfies (or fails to satisfy) any property that you can encode as types. The more expressive the type system, the more properties you can encode, and the more kinds of claims you can prove. But there will always be some properties that cannot be proved or that are too expensive to encode within a given type system. If you care about these properties, you'll have to find some other way of making sure they hold. So, even with advanced type systems, you'll end up writing tests. The flip side is that there are many important properties that you cannot adequately test for (e.g., "my code is free of injection vulnerabilities"). If you're not using proofs to make sure that these properties hold, there's little reason to believe that they do hold, regardless of how much testing you do. So, even with lots of tests, you'll end up using types. You need both.
- tel 13y ago
  I find the decision tree tends to start with whether or not I even understand the problem well enough to generate constraints. After that there are serious questions (today) on whether it's worth formalization. After that, you're probably knee-deep in discovering the difficulty of any proof obligations you've given yourself. Then, finally, you can answer questions about how pervasive the encoding difficulty might be. If those fail at any point it drives you to either use a more permissive type or to just test some useful cases. Not infrequently, test cases can be upgraded into more powerful types.
  - contingencies 13y ago
    If you test within a small enough scope (even at runtime and without a type system; eg. input to each function, religiously) then you don't really have the "do I understand the problem well enough" architectural issue, which in any event is not something you can test for ;)
    - 13y ago

by z0r 13y ago

prawks 13y ago

This is one thing that has irked me a little bit as I've gotten more involved with TDD and unit testing in general: the logic in many of my unit tests is near duplicate logic that is implemented within my application. My thought is that the general advantage of TDD is that you are thinking about your eventual implementation from the outside. You write tests in a different mindset based on expectations, rather than when you are implementing the production code where you are more concerned with the details of logic, etc. Unit tests also tend to be a bit more readable than the actual implementation logic, and so act as a nice form of documentation as a side-effect. In this way, test-first development provides a more gradual transition between a specification or design and the end implementation, which results in fewer missed requirements, allows for easy refactoring, etc. This is of course only related to TDD however, not unit testing in general. Hence, it still irks me that many of my tests look so similar to the logic they are testing, and thus aren't really testing for correctness at all.

jacquesm 13y ago
Think of unit testing as auditing the contract between two pieces of code right at the interface by replacing one of the two pieces of code (usually the client or caller but you can replace the code that provides the response just as easily) by a program that exercises the other layer. Like that you build your confidence layer by layer and with every set of tests you verify that you are still building on solid ground instead of quicksand. That's why you test at the interface level, it allows you to swap out the implementation without having to re-work all the test code.
tieTYT 13y ago
I went through a phase of this when I started writing automated tests. It occurred when I started using a mocking library for the first time. The library was so advanced I could basically test my implementation instead of my api. EG: "first you call foo.bar(), then you call foo.baz()" As a result, my test and my system under test looked very similar. I'm not sure if you're in the same situation I was in, but let me tell you: My situation was an anti-pattern. All I was testing was that my code does what it does. I wasn't testing that my code was working. Make sure you're not making the same mistake. If you are, a symptom would be that lots of tests fail when you're only refactoring.
- prawks 13y ago
  Not quite, but I get what you're saying for sure. I think jacquesm said it well in his reply to me that you're more testing the "contracts between two pieces of code right at the interface", and it shouldn't be any deeper than that to avoid what you're describing. The symptom of many tests failing during refactoring is something I'll be keeping in mind, however.

nahname 13y ago

Some issues are removed with a type system and wouldn't require writing tests to verfiy (the compiler would be doing this for you). I wouldn't take that fact as being in favor of using a type system, but more of a reminder that using a dynamic language without tests is reckless.

pauloortins 13y ago
Type testing can be replaced by static types but you still have to test all the others stuffs in your code.
- cgag 13y ago
  I just want to note that with an advanced type system, "type testing" can cover a lot more than people seem to give it credit for. An example I saw someone give on here the other day was that he had the types for his red black tree set up so that code that produced an unbalanced tree would never compile.
  - Skinney 13y ago
    Link? How would a type system be able to ensure that both sides of a rb tree is balanced? :S
    - deleted 13y ago
      [deleted]
    - cgag 13y ago
      https://news.ycombinator.com/item?id=5987527 https://news.ycombinator.com/item?id=5987527 I'm just getting into Haskell myself and I can't really explain those 10 lines myself (I also haven't implemented a red black tree). I watched a talk by Yaron Minksy the other day that also talked about the idea that you should "make illegal states unrepresentable" that was really good: http://vimeo.com/14313378 http://vimeo.com/14313378 (he starts talking about that at around ~17:50)
    - 13y ago

tmoertel 13y ago

Even with advanced type systems, the sweet spot will be a mix of types and tests. With types, you can prove many claims, including some you can't test for. With tests, you can gather evidence to support the claims you can't easily prove with types. So you need both. Elaboration: With advanced type systems you can prove that your code satisfies (or fails to satisfy) any property that you can encode as types. The more expressive the type system, the more properties you can encode, and the more kinds of claims you can prove. But there will always be some properties that cannot be proved or that are too expensive to encode within a given type system. If you care about these properties, you'll have to find some other way of making sure they hold. So, even with advanced type systems, you'll end up writing tests. The flip side is that there are many important properties that you cannot adequately test for (e.g., "my code is free of injection vulnerabilities"). If you're not using proofs to make sure that these properties hold, there's little reason to believe that they do hold, regardless of how much testing you do. So, even with lots of tests, you'll end up using types. You need both.

tel 13y ago
I find the decision tree tends to start with whether or not I even understand the problem well enough to generate constraints. After that there are serious questions (today) on whether it's worth formalization. After that, you're probably knee-deep in discovering the difficulty of any proof obligations you've given yourself. Then, finally, you can answer questions about how pervasive the encoding difficulty might be. If those fail at any point it drives you to either use a more permissive type or to just test some useful cases. Not infrequently, test cases can be upgraded into more powerful types.
- contingencies 13y ago
  If you test within a small enough scope (even at runtime and without a type system; eg. input to each function, religiously) then you don't really have the "do I understand the problem well enough" architectural issue, which in any event is not something you can test for ;)
  - 13y ago