5 ms·

I feel in a few years people will look back and laugh at all the useless tests we wrote. I still haven't seen a test that can prove even an identity function

I feel in a few years people will look back and laugh at all the useless tests we wrote. I still haven't seen a test that can prove even an identity function is correct. Yet it's obvious to the programmer. We lack the technology to properly test software. I like to think tests are a way to implement twice and hope you got it right at least once. But we aren't even to that point yet.

caf 6y ago
Most tests aren't intended to prove correctness - that is a very high bar. They're instead intended to offer some evidence that increases our level of confidence in correctness. Techniques that are intended to prove correctness - like for example symbolic execution - easily handle the identity function (of course!).
- gnulinux 6y ago
  I should add that even if you have the technology to prove correctness of program, that doesn't mean it's practical to prove correctness of all behavior of the program. I write real life programs in Agda (which is a dependently typed dialect of Haskell in which you can encode arbitrary proofs in type system) and you rarely prove all intended behavior. For example, recently I was writing a parser, and wanted to prove that for all languages I define, they're not empty languages (so that I don't accidentally construct empty languages by writing something like `between 'z' and 'a'` (silly example as this is easy to prevent)). So you start with basics, proving that the language that only contains empty string is not empty. Then you define what it means to be empty (empty is empty, empty+empty is empty etc...). Then you eventually need to prove L1+L2 is not empty if either L1 is not empty or L2 is not empty. Fine, just give a "witness" to each non-empty language so that we at least know these languages have 1 word in them. Then you have to prove things like "if string in L1 then string in L1+L2". Anyway, long story short, this one extra check I wanted to make, ended up being its own project that took maybe a week. It was definitely -- and I can't stress this enough -- definitely not worth the time. Being engineers requires you to assess the risk, and if something is not worth checking, there is no reason to check it.
rooam-dev 6y ago
Imagine assembling a car engine without any QC process, perhaps just visual inspection of the parts. The end users (whoever will drive the car) will be the actual testers.
- postalrat 6y ago
  Testing and documentation are both essential but too much of either becomes harmful.
  - qayxc 6y ago
    Welcome to the land of ambiguity and subjectivity - enjoy our stay! The problem is, that there's no way to generally quantify "too much" or "not enough" when it comes to tests and documentation. Even "harmful" isn't a well defined term on its own and needs to be properly defined on a case-by-case basis. The statement is a platitude without any substance behind it and you can replace "testing and documentation" with just about anything and come to the same result: Vitamines and calorie intake are both essential but too much of either becomes harmful. How profound...
  - afarrell 6y ago
    https://agilemanifesto.org/ https://agilemanifesto.org/ > Individuals and interactions over processes and tools > Working software over comprehensive documentation > Customer collaboration over contract negotiation > Responding to change over following a plan The things on the right are only useful in service of the things on the left.
- afarrell 6y ago
  You jest, but I have been in a conversation where I proposed a user experience improvement and heard "but if we change the code, it will no longer be tested. Right now, users have reported bugs and we have fixed all of them." Because users always report bugs... If you've ever been frustrated by the package manager Conda and the UX of its command-line interface ...I'm sorry. I tried.
  - 6y ago
dragonwriter 6y ago
> I still haven't seen a test that can prove even an identity function is correct. As with any hypothesis (empirical) testing, software testing is not about proving code is correct, but about striving to falsify that claim by poking at ways it would be likely to fail if it wasn't. (There are formal analytical methods for proving code correct as opposed to empirical methods of falsifying it, as well, but those aren't tests.)
donw 6y ago
Tests do not exist to prove correctness. Tests exist to help prevent you from changing behavior unknowingly.
- qayxc 6y ago
  I'd argue that tests can also be a tool for improving your architecture (e.g. separation of concerns) and for communicating intend and usage. Tests can be a tool for documenting expected behaviour, intended use, and edge cases. In contrast to written documentation, they have to be up-to-date by their very nature, which is an advantage over code comments or external docs.
CJefferson 6y ago
What do you suggest the alternative is? Also, I agree that's exactly what testing is, in fact I often now test code like that, get someone else to do one simple implementation in Python, a second high-performance implementation in C++/Rust, and compare them. Then, as long as the same bug doesn't occur in both versions (and that seems very unlikely, as they are implemented with different algorithms), we find all the bugs in both.
gwd 6y ago
> I like to think tests are a way to implement twice and hope you got it right at least once. This to me is the biggest thing that makes writing good tests a "grind". For a simple, well-defined function, you can toss up a bunch of inputs and outputs and feel satisfied that all the corner cases work. For any moderately complex behavior, however, so far the only effective way I've found to test the behavior is to effectively duplicate the logic a second time in the testing code, and check to see that I got the same result; which is pretty discouraging when you're doing it. It's certainly nice once it's done, though, to be able to make a change, re-run your test suite, and have a pretty reasonable confidence that nothing broke.
chriswarbo 6y ago
The crucial difference between tests and implementations is that implementations have to satisfy all of the requirements at once, while tests can check a single thing at a time. As a simple example, if we need to test a 'sort' function for lists, we might write a whole bunch of tests like this: "Sorting" should "preserve length" in { forAll() { (l: List[Int]) => assert(sort(l).length == l.length) } } "Sorting" should "be idempotent" in { forAll() { (l: List[Int]) => assert(sort(l) == sort(sort(l))) } "Sorting" should "put elements in order" in { forAll() { (l: List[Int]) => whenever(l.nonEmpty) { sort(l).foldLeft((true, l.min))({ case ((result, max), elem) => (result && max <= elem, elem) }) match { case (result, max) => assert(result && max == l.max) } } } } "Sorting" should "not change elements" in { forAll() { (l: List[Int) => { val sorted = sort(l) assert(l.all(elem => sorted.contains(elem))) assert(sorted.all(elem => l.contains(elem))) } } Each of these tests can focus on one aspect of sorting and ignore everything else. Each on its own is not enough to give us confidence in the 'sort' function (e.g. the identity function would pass the idempotence test, the same-length test and the same-elements test; returning an empty list would pass the elements-in-order test; etc.), however together they are pretty good. In contrast, we can't implement the 'sort' function in such a piecemeal way. It needs to take every requirement into account, in case a step which satisfies one requirement invalidates the others. That's also why writing a whole new implementation to test against is best seen as a last resort.
rocqua 6y ago
I wrote an LU decomposition function yesterday. I then wrote a test that tries to reconstitute the original matrix for the LU decomposition, for a few randomly generated matrices. That is not 'implement twice and hope you get it right at least once'. It isn't a perfect proof of correctness either, but it gives much more guarantees than a simple double implementation. Notably, when my code to solve a linear system using LU decomposition failed, I new I didn't need to worry about my decomposition, because those tests still passed.
LeonidBugaev 6y ago
In general agree. Tests are needed, but not for the sake of the tests. You need to cover main hot paths. And thats why I prefer integration testing. And ideally automated one, for example using things like GoReplay https://goreplay.org/ https://goreplay.org/.

caf 6y ago

Most tests aren't intended to prove correctness - that is a very high bar. They're instead intended to offer some evidence that increases our level of confidence in correctness. Techniques that are intended to prove correctness - like for example symbolic execution - easily handle the identity function (of course!).

gnulinux 6y ago
I should add that even if you have the technology to prove correctness of program, that doesn't mean it's practical to prove correctness of all behavior of the program. I write real life programs in Agda (which is a dependently typed dialect of Haskell in which you can encode arbitrary proofs in type system) and you rarely prove all intended behavior. For example, recently I was writing a parser, and wanted to prove that for all languages I define, they're not empty languages (so that I don't accidentally construct empty languages by writing something like `between 'z' and 'a'` (silly example as this is easy to prevent)). So you start with basics, proving that the language that only contains empty string is not empty. Then you define what it means to be empty (empty is empty, empty+empty is empty etc...). Then you eventually need to prove L1+L2 is not empty if either L1 is not empty or L2 is not empty. Fine, just give a "witness" to each non-empty language so that we at least know these languages have 1 word in them. Then you have to prove things like "if string in L1 then string in L1+L2". Anyway, long story short, this one extra check I wanted to make, ended up being its own project that took maybe a week. It was definitely -- and I can't stress this enough -- definitely not worth the time. Being engineers requires you to assess the risk, and if something is not worth checking, there is no reason to check it.

rooam-dev 6y ago

Imagine assembling a car engine without any QC process, perhaps just visual inspection of the parts. The end users (whoever will drive the car) will be the actual testers.

postalrat 6y ago
Testing and documentation are both essential but too much of either becomes harmful.
- qayxc 6y ago
  Welcome to the land of ambiguity and subjectivity - enjoy our stay! The problem is, that there's no way to generally quantify "too much" or "not enough" when it comes to tests and documentation. Even "harmful" isn't a well defined term on its own and needs to be properly defined on a case-by-case basis. The statement is a platitude without any substance behind it and you can replace "testing and documentation" with just about anything and come to the same result: Vitamines and calorie intake are both essential but too much of either becomes harmful. How profound...
- afarrell 6y ago
  https://agilemanifesto.org/ https://agilemanifesto.org/ > Individuals and interactions over processes and tools > Working software over comprehensive documentation > Customer collaboration over contract negotiation > Responding to change over following a plan The things on the right are only useful in service of the things on the left.
afarrell 6y ago
You jest, but I have been in a conversation where I proposed a user experience improvement and heard "but if we change the code, it will no longer be tested. Right now, users have reported bugs and we have fixed all of them." Because users always report bugs... If you've ever been frustrated by the package manager Conda and the UX of its command-line interface ...I'm sorry. I tried.
- 6y ago

dragonwriter 6y ago

> I still haven't seen a test that can prove even an identity function is correct. As with any hypothesis (empirical) testing, software testing is not about proving code is correct, but about striving to falsify that claim by poking at ways it would be likely to fail if it wasn't. (There are formal analytical methods for proving code correct as opposed to empirical methods of falsifying it, as well, but those aren't tests.)

donw 6y ago

Tests do not exist to prove correctness. Tests exist to help prevent you from changing behavior unknowingly.

qayxc 6y ago
I'd argue that tests can also be a tool for improving your architecture (e.g. separation of concerns) and for communicating intend and usage. Tests can be a tool for documenting expected behaviour, intended use, and edge cases. In contrast to written documentation, they have to be up-to-date by their very nature, which is an advantage over code comments or external docs.

CJefferson 6y ago

What do you suggest the alternative is? Also, I agree that's exactly what testing is, in fact I often now test code like that, get someone else to do one simple implementation in Python, a second high-performance implementation in C++/Rust, and compare them. Then, as long as the same bug doesn't occur in both versions (and that seems very unlikely, as they are implemented with different algorithms), we find all the bugs in both.

gwd 6y ago

> I like to think tests are a way to implement twice and hope you got it right at least once. This to me is the biggest thing that makes writing good tests a "grind". For a simple, well-defined function, you can toss up a bunch of inputs and outputs and feel satisfied that all the corner cases work. For any moderately complex behavior, however, so far the only effective way I've found to test the behavior is to effectively duplicate the logic a second time in the testing code, and check to see that I got the same result; which is pretty discouraging when you're doing it. It's certainly nice once it's done, though, to be able to make a change, re-run your test suite, and have a pretty reasonable confidence that nothing broke.

chriswarbo 6y ago

The crucial difference between tests and implementations is that implementations have to satisfy all of the requirements at once, while tests can check a single thing at a time. As a simple example, if we need to test a 'sort' function for lists, we might write a whole bunch of tests like this: "Sorting" should "preserve length" in { forAll() { (l: List[Int]) => assert(sort(l).length == l.length) } } "Sorting" should "be idempotent" in { forAll() { (l: List[Int]) => assert(sort(l) == sort(sort(l))) } "Sorting" should "put elements in order" in { forAll() { (l: List[Int]) => whenever(l.nonEmpty) { sort(l).foldLeft((true, l.min))({ case ((result, max), elem) => (result && max <= elem, elem) }) match { case (result, max) => assert(result && max == l.max) } } } } "Sorting" should "not change elements" in { forAll() { (l: List[Int) => { val sorted = sort(l) assert(l.all(elem => sorted.contains(elem))) assert(sorted.all(elem => l.contains(elem))) } } Each of these tests can focus on one aspect of sorting and ignore everything else. Each on its own is not enough to give us confidence in the 'sort' function (e.g. the identity function would pass the idempotence test, the same-length test and the same-elements test; returning an empty list would pass the elements-in-order test; etc.), however together they are pretty good. In contrast, we can't implement the 'sort' function in such a piecemeal way. It needs to take every requirement into account, in case a step which satisfies one requirement invalidates the others. That's also why writing a whole new implementation to test against is best seen as a last resort.

rocqua 6y ago

I wrote an LU decomposition function yesterday. I then wrote a test that tries to reconstitute the original matrix for the LU decomposition, for a few randomly generated matrices. That is not 'implement twice and hope you get it right at least once'. It isn't a perfect proof of correctness either, but it gives much more guarantees than a simple double implementation. Notably, when my code to solve a linear system using LU decomposition failed, I new I didn't need to worry about my decomposition, because those tests still passed.

LeonidBugaev 6y ago

In general agree. Tests are needed, but not for the sake of the tests. You need to cover main hot paths. And thats why I prefer integration testing. And ideally automated one, for example using things like GoReplay https://goreplay.org/ https://goreplay.org/.