3 ms·
> they prove nothing If they fail, they prove there's a bug (in either the test or the code.) This is like literally any other kind of test.
by pfdietz 3y ago
> they prove nothing
If they fail, they prove there's a bug (in either the test or the code.)
This is like literally any other kind of test.
- erikpukinskis 3y agoA bug in test code is not a real bug. It’s just a test that’s not giving you useful information. Lots of tests don’t give you useful information. Some that fail and some that pass. It’s easy to write a test that doesn’t provide useful information across time. Harder to write a test that does.
- baq 3y agoThere are times for constructive advice and there are times when '...so don't do that' is the right answer. > It’s easy to write a test that doesn’t provide useful information across time. I firmly believe this is one of those times. (Currently my only issue with tests in the product I work on is that they take too long to run. Can't have it all.)
- slothtrop 3y ago> A bug in test code is not a real bug. The bug is in the library that the test code invokes. Tests themselves should be simple, if there are bugs in tests they are trivial once you have a framework figured out.
- agentultra 3y agoI meant "prove" as in, "mathematically proven." That is, for all possible inputs your theorem holds. A unit test is only an example of one such input. They don't prove there are no bad inputs. There are many places in programming where you don't care to prove properties of your program to this level of rigor; that's fine -- sufficiency is an important distinction: if a handful of examples are enough to convince you that your implementation is correct with regards to your specifications, then it's good enough. Does the file get copied to the right place? Cool. However there are many more places where unit tests aren't sufficient. You can't express properties like, "this program can share memory and never allows information to escape to other threads." Or, with <= 10 writers all transactions will always complete. A unit test can demonstrate an example of one such case at a time... but you will never prove things one example at a time.
- pfdietz 3y agoWell that's silly. You're not arguing against unit tests, you're arguing against testing in general. But empirically testing does improve the reliability of software. You're tossing the baby out with the bathwater in the interest of an academic ideal of proved correctness. I will add that if you are verifying the correctness of some code, you have a formal specification of what the code is supposed to do. That is, you have a description of valid inputs, and a formula determining if the output is correct, given those inputs. But if you have those, you can also do property-based testing: generate random inputs that satisfy the input properties, and check that the output satisfies the output condition. This is all easier that proving correctness (it requires little or no manual intervention) and gives much of the same benefit.
- agentultra 3y agoMaybe you need to re-read my original comment. I’m arguing for sufficient evidence of correctness. Unit tests often provide that for a good deal of software. I think people ought to write more of them. However I think folks do get dogmatic about testing and will claim things like, “all you need is unit/integration/types.”