4 ms·

It's all degrees. Unit tests are great at finding examples of errors or correct behaviours. However they prove nothing and they definitely do not demonstrate th

It's all degrees. Unit tests are great at finding examples of errors or correct behaviours. However they prove nothing and they definitely do not demonstrate the absence of errors. They are often sufficient for a great deal of projects. If all it takes to convince you it's "good enough," are a handful of examples then that's it. As much as you need and no less. However I find we programmers tend to be a dogmatic bunch and many of us out there like to cling to our favoured practices and tools. Unit tests aren't the only testing method. Integration tests are fine. Some times testing is not sufficient: you need proof. Static types are great but fast-and-loose reasoning is also useful and so you still need a few tests. What's important is that we sit down to think about specifying what it means for our programs to be, "correct." Because when someone asks, "is it correct?" You need to as, "with respect to what?" If all you have are some hastily written notes from a bunch of meetings and long-lost whiteboard sessions... then you don't really have an answer. Any behaviour is, "correct," if you haven't specified what it should be.

pfdietz 3y ago
> they prove nothing If they fail, they prove there's a bug (in either the test or the code.) This is like literally any other kind of test.
- erikpukinskis 3y ago
  A bug in test code is not a real bug. It’s just a test that’s not giving you useful information. Lots of tests don’t give you useful information. Some that fail and some that pass. It’s easy to write a test that doesn’t provide useful information across time. Harder to write a test that does.
  - baq 3y ago
    There are times for constructive advice and there are times when '...so don't do that' is the right answer. > It’s easy to write a test that doesn’t provide useful information across time. I firmly believe this is one of those times. (Currently my only issue with tests in the product I work on is that they take too long to run. Can't have it all.)
  - slothtrop 3y ago
    > A bug in test code is not a real bug. The bug is in the library that the test code invokes. Tests themselves should be simple, if there are bugs in tests they are trivial once you have a framework figured out.
- agentultra 3y ago
  I meant "prove" as in, "mathematically proven." That is, for all possible inputs your theorem holds. A unit test is only an example of one such input. They don't prove there are no bad inputs. There are many places in programming where you don't care to prove properties of your program to this level of rigor; that's fine -- sufficiency is an important distinction: if a handful of examples are enough to convince you that your implementation is correct with regards to your specifications, then it's good enough. Does the file get copied to the right place? Cool. However there are many more places where unit tests aren't sufficient. You can't express properties like, "this program can share memory and never allows information to escape to other threads." Or, with <= 10 writers all transactions will always complete. A unit test can demonstrate an example of one such case at a time... but you will never prove things one example at a time.
  - pfdietz 3y ago
    Well that's silly. You're not arguing against unit tests, you're arguing against testing in general. But empirically testing does improve the reliability of software. You're tossing the baby out with the bathwater in the interest of an academic ideal of proved correctness. I will add that if you are verifying the correctness of some code, you have a formal specification of what the code is supposed to do. That is, you have a description of valid inputs, and a formula determining if the output is correct, given those inputs. But if you have those, you can also do property-based testing: generate random inputs that satisfy the input properties, and check that the output satisfies the output condition. This is all easier that proving correctness (it requires little or no manual intervention) and gives much of the same benefit.
    - agentultra 3y ago
      Maybe you need to re-read my original comment. I’m arguing for sufficient evidence of correctness. Unit tests often provide that for a good deal of software. I think people ought to write more of them. However I think folks do get dogmatic about testing and will claim things like, “all you need is unit/integration/types.”
IggleSniggle 3y ago
The correct behavior is the behavior it has, of course! It is all the other programs that can't integrate with it that are wrong. /s Unit tests or not, so much code I interact with is like this. This is part of why I love integration tests. It's usually at the point of integrating one thing with another that things go bad, where bugs occur, and where the intention of APIs are misunderstood. I like unit tests for the way that they encourage composition and dependency injection, but if you're already doing that, then (unit tests or not) I prefer integration tests. They might not be as neat and tidy as a unit test OR as a e2e test, and they might miss important implementation edge cases, but well made integration tests can find all sorts of race conditions, configurations that we should help users avoid, and much much more precisely because they are looking for problems with the side effects that no amount of pure-function unit-tested edge-cased code will make obvious or mitigate. Integration tests are like the "explain why" comments that everyone clamors for, but in reproducible demo form. "Show me" vs "tell me"
lizardking 3y ago
They can prove that by building out some new feature you implicitly break some existing functionality that didn't occur to the person writing the requirements.

pfdietz 3y ago

> they prove nothing If they fail, they prove there's a bug (in either the test or the code.) This is like literally any other kind of test.

erikpukinskis 3y ago
A bug in test code is not a real bug. It’s just a test that’s not giving you useful information. Lots of tests don’t give you useful information. Some that fail and some that pass. It’s easy to write a test that doesn’t provide useful information across time. Harder to write a test that does.
- baq 3y ago
  There are times for constructive advice and there are times when '...so don't do that' is the right answer. > It’s easy to write a test that doesn’t provide useful information across time. I firmly believe this is one of those times. (Currently my only issue with tests in the product I work on is that they take too long to run. Can't have it all.)
- slothtrop 3y ago
  > A bug in test code is not a real bug. The bug is in the library that the test code invokes. Tests themselves should be simple, if there are bugs in tests they are trivial once you have a framework figured out.
agentultra 3y ago
I meant "prove" as in, "mathematically proven." That is, for all possible inputs your theorem holds. A unit test is only an example of one such input. They don't prove there are no bad inputs. There are many places in programming where you don't care to prove properties of your program to this level of rigor; that's fine -- sufficiency is an important distinction: if a handful of examples are enough to convince you that your implementation is correct with regards to your specifications, then it's good enough. Does the file get copied to the right place? Cool. However there are many more places where unit tests aren't sufficient. You can't express properties like, "this program can share memory and never allows information to escape to other threads." Or, with <= 10 writers all transactions will always complete. A unit test can demonstrate an example of one such case at a time... but you will never prove things one example at a time.
- pfdietz 3y ago
  Well that's silly. You're not arguing against unit tests, you're arguing against testing in general. But empirically testing does improve the reliability of software. You're tossing the baby out with the bathwater in the interest of an academic ideal of proved correctness. I will add that if you are verifying the correctness of some code, you have a formal specification of what the code is supposed to do. That is, you have a description of valid inputs, and a formula determining if the output is correct, given those inputs. But if you have those, you can also do property-based testing: generate random inputs that satisfy the input properties, and check that the output satisfies the output condition. This is all easier that proving correctness (it requires little or no manual intervention) and gives much of the same benefit.
  - agentultra 3y ago
    Maybe you need to re-read my original comment. I’m arguing for sufficient evidence of correctness. Unit tests often provide that for a good deal of software. I think people ought to write more of them. However I think folks do get dogmatic about testing and will claim things like, “all you need is unit/integration/types.”

IggleSniggle 3y ago

The correct behavior is the behavior it has, of course! It is all the other programs that can't integrate with it that are wrong. /s Unit tests or not, so much code I interact with is like this. This is part of why I love integration tests. It's usually at the point of integrating one thing with another that things go bad, where bugs occur, and where the intention of APIs are misunderstood. I like unit tests for the way that they encourage composition and dependency injection, but if you're already doing that, then (unit tests or not) I prefer integration tests. They might not be as neat and tidy as a unit test OR as a e2e test, and they might miss important implementation edge cases, but well made integration tests can find all sorts of race conditions, configurations that we should help users avoid, and much much more precisely because they are looking for problems with the side effects that no amount of pure-function unit-tested edge-cased code will make obvious or mitigate. Integration tests are like the "explain why" comments that everyone clamors for, but in reproducible demo form. "Show me" vs "tell me"

lizardking 3y ago

They can prove that by building out some new feature you implicitly break some existing functionality that didn't occur to the person writing the requirements.