5 ms·

The real value in unit tests comes when you need to refactor a codebase. Especially if you're not the original author. Unit tests are a codified API contract.

by bbrks 7y ago

The real value in unit tests comes when you need to refactor a codebase. Especially if you're not the original author. Unit tests are a codified API contract. It's easy to write something small and correct from scratch (like a greenfield microservice). It's far harder to maintain that over several years and several changes of hands.

crimsonalucard 7y ago
A type checker should track an aspect of this thoroughly. If you find yourself relying unit tests as a contract to prevent future changes something is wrong with your coding methodology. Too many dependencies in your code, not enough modularity. If your coding methodology was good, the type checker is enough to verify these contracts. A language without a type checker will have more need of unit tests.
- mchaynes 7y ago
  For very simple programs, sure. Type checkers do not verify behavior. Unit tests verify behavior
  - continuational 7y ago
    Unit tests do not verify behavior, except for a few cases (out of a virtually infinite set of cases) that the programmer happened to think about. Types do verify behavior - e.g. this function always returns an Int, that function only returns a UserId, this other function doesn't do any I/O.
    - bluGill 7y ago
      That is a very different behavior though. If my multiply function returns ADDs the input and returns an int it passes the type checker but is still wrong.
      - morelisp 7y ago
        But if your multiply function fails to handle some special case that the tests don't catch (negatives, overflow/saturation, denorms, ...) then the tests are also wrong. What you really want is to be able to state properties, e.g. "for all x/y, mul(x, 0) = 0, mul(x, 1) = x, mul(x, 2) = x + x, mul(x, y) = mul(y, x)", and so on. And working with these kinds of statements (via e.g. quickcheck or whatever) might generate tests, but feels a lot more like working with types. You're making proofs, not examples.
        bluGill 7y ago
        Which is why I have always (well for the last 5 years or so anyway) said that you want both because they catch different issues.
- alkonaut 7y ago
  A type checker still accepts a correct but enormous interface without complaint. It’s only when you want to test it in isolation you realize that the function needs a dog+kitchen+universe you have trouble providing. So you refactor it to have a smaller interface and that lets you write the test. The act of writing the test now made you write less coupled software and it will remain and prevent anyone from adding the coupling in the future. For me this is the best side of unittests: their validation of “correctness” is just one aspect, but their job as a guarantee of decoupling and a living documentation is at least as important. Even a test that is only written and compiled but never run provides great value for architecture and documentation.
  - crimsonalucard 7y ago
    I am only arguing the fact that unit tests are unneeded FOR correctness now and in the future if you have the right programming methodology. If you want to write unit tests for documentation that's a different argument. I'm not against that.
    - alkonaut 7y ago
      While a lot of domain knowledge might be possible to encode using types, most domain knowledge is probably easier to ensure via a combination of types and unit tests. It not only needs to be possible for unit tests to b me unneeded, it needs to be practical (as in economical, possible on the language used). You can probably make a formally verified first person shooter which validates the logic of its physics/network/game logic in Agda. But you likely can’t do it in C++ and be done by the holiday season. So unit tests are probably still needed and a good idea for most situations. Unit tests famously don’t prove correctness (the absence of bugs) only the presence of bugs, so I guess it’s wrong to compare them to actual verifying actual correctness.
      - crimsonalucard 7y ago
        There's another emergent phenomenon that isn't often talked about for unit tests. It's known intuitively though nobody mentions it though. Unit tests are like statistical samples of all possible test cases. So unit tests while they don't verify correctness overall they can correlate with correctness. That is why you only need a couple unit tests to correlate with. a degree of correctness with a domain that's almost infinite. The phenomenon that enables science also enables unit tests to work. That being said, there are programming methodologies of managing complexity that negate the need for unit tests. This is my main point.
- bluGill 7y ago
  A type checker is very helpful, but type checkers catch a few different type of error from unit tests. I would not want to refactor a large program without both. Unit tests ensure that everything you thought about still works. Types ensure that ALL types are compatible. What types do well is the everything case, you cannot forget something with types. However they only cover compatibility not correctness. You can have a type correct program that is wrong. There are formal methods can can get to correctness. If you have the right constraints. The only problem is you can have the wrong constraints. Within that limit though it shows the program works correctly in all cases. Unit tests prove only the cases you thought about work. While this is far below the infinite number of cases the above prove, they are still very useful because it is generally very easy to reason about the one case you do choose. If you choose the cases well, the majority of possible cases are "close enough". Thus a few well chosen unit tests can give you assurance that your code works correctly in the important cases.
  - crimsonalucard 7y ago
    Unit tests operate in the same domain and constraints as formal methods. In terms of correctness formal methods beat unit tests hands down. For things that live outside of this domain the term integration test becomes relevant. If you are testing for unknown side effects it is no longer a unit test but an integration test. This is not my point. Unfortunately I can't go into detail about my point because it's just too long to talk about. I glossed over it and called it "Programming methodology." In the context of correctness, with the right programming methodology, programming language and type checker you can use this and your intuition alone to forego the need for unit tests. This does not apply to integration tests. Anyway without going in to deep the right programming methodology involves reducing complexity. It's similar to how if a programming doesn't have nulls you can't have runtime errors caused my nulls. If you use the right programming language or right methodology you can reduce much of the complexity of the program to the point where unit tests are redundant to your intuition. Another good example of a language that pulls this off is Rust or Haskell. You don't have to go as far as COQ but the previous two languages force you into a methodology that ensures correctness to a degree that many unit tests become redundant.

by bbrks 7y ago

crimsonalucard 7y ago

A type checker should track an aspect of this thoroughly. If you find yourself relying unit tests as a contract to prevent future changes something is wrong with your coding methodology. Too many dependencies in your code, not enough modularity. If your coding methodology was good, the type checker is enough to verify these contracts. A language without a type checker will have more need of unit tests.

mchaynes 7y ago
For very simple programs, sure. Type checkers do not verify behavior. Unit tests verify behavior
- continuational 7y ago
  Unit tests do not verify behavior, except for a few cases (out of a virtually infinite set of cases) that the programmer happened to think about. Types do verify behavior - e.g. this function always returns an Int, that function only returns a UserId, this other function doesn't do any I/O.
  - bluGill 7y ago
    That is a very different behavior though. If my multiply function returns ADDs the input and returns an int it passes the type checker but is still wrong.
    - morelisp 7y ago
      But if your multiply function fails to handle some special case that the tests don't catch (negatives, overflow/saturation, denorms, ...) then the tests are also wrong. What you really want is to be able to state properties, e.g. "for all x/y, mul(x, 0) = 0, mul(x, 1) = x, mul(x, 2) = x + x, mul(x, y) = mul(y, x)", and so on. And working with these kinds of statements (via e.g. quickcheck or whatever) might generate tests, but feels a lot more like working with types. You're making proofs, not examples.
      - bluGill 7y ago
        Which is why I have always (well for the last 5 years or so anyway) said that you want both because they catch different issues.
alkonaut 7y ago
A type checker still accepts a correct but enormous interface without complaint. It’s only when you want to test it in isolation you realize that the function needs a dog+kitchen+universe you have trouble providing. So you refactor it to have a smaller interface and that lets you write the test. The act of writing the test now made you write less coupled software and it will remain and prevent anyone from adding the coupling in the future. For me this is the best side of unittests: their validation of “correctness” is just one aspect, but their job as a guarantee of decoupling and a living documentation is at least as important. Even a test that is only written and compiled but never run provides great value for architecture and documentation.
- crimsonalucard 7y ago
  I am only arguing the fact that unit tests are unneeded FOR correctness now and in the future if you have the right programming methodology. If you want to write unit tests for documentation that's a different argument. I'm not against that.
  - alkonaut 7y ago
    While a lot of domain knowledge might be possible to encode using types, most domain knowledge is probably easier to ensure via a combination of types and unit tests. It not only needs to be possible for unit tests to b me unneeded, it needs to be practical (as in economical, possible on the language used). You can probably make a formally verified first person shooter which validates the logic of its physics/network/game logic in Agda. But you likely can’t do it in C++ and be done by the holiday season. So unit tests are probably still needed and a good idea for most situations. Unit tests famously don’t prove correctness (the absence of bugs) only the presence of bugs, so I guess it’s wrong to compare them to actual verifying actual correctness.
    - crimsonalucard 7y ago
      There's another emergent phenomenon that isn't often talked about for unit tests. It's known intuitively though nobody mentions it though. Unit tests are like statistical samples of all possible test cases. So unit tests while they don't verify correctness overall they can correlate with correctness. That is why you only need a couple unit tests to correlate with. a degree of correctness with a domain that's almost infinite. The phenomenon that enables science also enables unit tests to work. That being said, there are programming methodologies of managing complexity that negate the need for unit tests. This is my main point.
bluGill 7y ago
A type checker is very helpful, but type checkers catch a few different type of error from unit tests. I would not want to refactor a large program without both. Unit tests ensure that everything you thought about still works. Types ensure that ALL types are compatible. What types do well is the everything case, you cannot forget something with types. However they only cover compatibility not correctness. You can have a type correct program that is wrong. There are formal methods can can get to correctness. If you have the right constraints. The only problem is you can have the wrong constraints. Within that limit though it shows the program works correctly in all cases. Unit tests prove only the cases you thought about work. While this is far below the infinite number of cases the above prove, they are still very useful because it is generally very easy to reason about the one case you do choose. If you choose the cases well, the majority of possible cases are "close enough". Thus a few well chosen unit tests can give you assurance that your code works correctly in the important cases.
- crimsonalucard 7y ago
  Unit tests operate in the same domain and constraints as formal methods. In terms of correctness formal methods beat unit tests hands down. For things that live outside of this domain the term integration test becomes relevant. If you are testing for unknown side effects it is no longer a unit test but an integration test. This is not my point. Unfortunately I can't go into detail about my point because it's just too long to talk about. I glossed over it and called it "Programming methodology." In the context of correctness, with the right programming methodology, programming language and type checker you can use this and your intuition alone to forego the need for unit tests. This does not apply to integration tests. Anyway without going in to deep the right programming methodology involves reducing complexity. It's similar to how if a programming doesn't have nulls you can't have runtime errors caused my nulls. If you use the right programming language or right methodology you can reduce much of the complexity of the program to the point where unit tests are redundant to your intuition. Another good example of a language that pulls this off is Rust or Haskell. You don't have to go as far as COQ but the previous two languages force you into a methodology that ensures correctness to a degree that many unit tests become redundant.