5 ms·

The issue I have with tests, is that I feel like I'm just implementing my algorithm a second time, in a different wording (language/domain), hoping that this wi

by runeks 10y ago

The issue I have with tests, is that I feel like I'm just implementing my algorithm a second time, in a different wording (language/domain), hoping that this will catch some unknown bug(s). If a test suite covers all aspects of an implementation, it's basically a second implementation, and then the question becomes whether the time spent writing this test code would actually have been better spent looking over the actual code that defines your algorithm. If we can't detect errors in code by reading it, I don't think tests will help us much. Furthermore: if our algorithms require a second "test implementation", shouldn't we also write a test for the test, to confirm that the written test actually tests correctly? What good is a broken test? If you've built a plywood door, and a machine for testing that door, wouldn't it be useful to have a machine that tests that your testing machine actually tests the door properly? All jokes aside, I really enjoy automated, high-level tests, which mimic actual user behaviour. I think this is different because we're no longer implementing an algorithm a second time around (as a test), but rather encoding user behaviour (of the algorithm) into an algorithm (new code).

sverige 10y ago
> If a test suite covers all aspects of an implementation, it's basically a second implementation, and then the question becomes whether the time spent writing this test code would actually have been better spent looking over the actual code that defines your algorithm. If we can't detect errors in code by reading it, I don't think tests will help us much. The OpenBSD code has been audited many times by very experienced and careful C experts. Fuzzing apparently found the very few things they hadn't already found by reading the code.
- cLeEOGPw 10y ago
  And it if I understand OpenBSD goal correctly, they take any kind of edge they can get to find those missed parts. Their formula for what is worth essentially boils down to "if this approach can find at least one bug no other approach would do it, we will use it".
- runeks 10y ago
  > The OpenBSD code has been audited many times by very experienced and careful C experts. Fuzzing apparently found the very few things they hadn't already found by reading the code. That's definitely comforting, but I guess my point is that the fact that bugs can even hide in plain sight in the first place is the root cause of the problem. If we can look at a specification of what something is supposed to do - the code - and not see that it does something entirely different, then the problem is with the programming/specification language. The very purpose of a programming language is to make the specification of computer programs readable (rather than reading assembly). If a specification language needs a a computer program to go through the specification and see if it specifies things that we didn't intend to, then that specification language misses the very point of being a language in which to specify things in the first place.
  - et1337 10y ago
    I don't think it's that simple. A lot of bugs are the result of disparate pieces of code -- each of which is fairly easy to verify by reading it -- interacting in an unforeseen way.
    - runeks 10y ago
      I would argue that this effect is not a product of code but of global state. If you have a pure function that transforms some input to an output, all you can do wrong is use the wrong tool for the wrong job, which the type system should, preferably, prevent you from doing. Using the wrong function for the wrong job means we haven't restricted well enough the input the function takes, through clearly defining the types of the arguments, not allowing for meaningless invariants. If a specification language requires the writer/reader to be aware of the entire specification in order to verify whether an isolated piece of specification is valid, that specification language is not of much value, I would argue. It occurs to me that the problem with traditional languages is that they do not allow the direct reference to information (values); only to variables which contain values (information). In Haskell, if you want a place to store information you create a TVar (variable), while you are forced to do this, always, in traditional languages, if you want to work with information/values properly. You're forced to keep information in a register if you want to manipulate it, and always refer to the register, when the information it contains is what you're really concerned with. Having to both think about the values/information and where you stored this information is double work. Why not reference the information directly?
BugsBunnySan 10y ago
I think the difference is that the test shows what the idea you (as the architect) have is, while the code shows how you (as the developer) implemented it. And then you run the tests and they will, hopefully, tell you if the how matches the what, and if it doesn't where the two differ, i.e. where the the code doesn't do what you intended...

by runeks 10y ago

sverige 10y ago

> If a test suite covers all aspects of an implementation, it's basically a second implementation, and then the question becomes whether the time spent writing this test code would actually have been better spent looking over the actual code that defines your algorithm. If we can't detect errors in code by reading it, I don't think tests will help us much. The OpenBSD code has been audited many times by very experienced and careful C experts. Fuzzing apparently found the very few things they hadn't already found by reading the code.

cLeEOGPw 10y ago
And it if I understand OpenBSD goal correctly, they take any kind of edge they can get to find those missed parts. Their formula for what is worth essentially boils down to "if this approach can find at least one bug no other approach would do it, we will use it".
runeks 10y ago
> The OpenBSD code has been audited many times by very experienced and careful C experts. Fuzzing apparently found the very few things they hadn't already found by reading the code. That's definitely comforting, but I guess my point is that the fact that bugs can even hide in plain sight in the first place is the root cause of the problem. If we can look at a specification of what something is supposed to do - the code - and not see that it does something entirely different, then the problem is with the programming/specification language. The very purpose of a programming language is to make the specification of computer programs readable (rather than reading assembly). If a specification language needs a a computer program to go through the specification and see if it specifies things that we didn't intend to, then that specification language misses the very point of being a language in which to specify things in the first place.
- et1337 10y ago
  I don't think it's that simple. A lot of bugs are the result of disparate pieces of code -- each of which is fairly easy to verify by reading it -- interacting in an unforeseen way.
  - runeks 10y ago
    I would argue that this effect is not a product of code but of global state. If you have a pure function that transforms some input to an output, all you can do wrong is use the wrong tool for the wrong job, which the type system should, preferably, prevent you from doing. Using the wrong function for the wrong job means we haven't restricted well enough the input the function takes, through clearly defining the types of the arguments, not allowing for meaningless invariants. If a specification language requires the writer/reader to be aware of the entire specification in order to verify whether an isolated piece of specification is valid, that specification language is not of much value, I would argue. It occurs to me that the problem with traditional languages is that they do not allow the direct reference to information (values); only to variables which contain values (information). In Haskell, if you want a place to store information you create a TVar (variable), while you are forced to do this, always, in traditional languages, if you want to work with information/values properly. You're forced to keep information in a register if you want to manipulate it, and always refer to the register, when the information it contains is what you're really concerned with. Having to both think about the values/information and where you stored this information is double work. Why not reference the information directly?

BugsBunnySan 10y ago

I think the difference is that the test shows what the idea you (as the architect) have is, while the code shows how you (as the developer) implemented it. And then you run the tests and they will, hopefully, tell you if the how matches the what, and if it doesn't where the two differ, i.e. where the the code doesn't do what you intended...