5 ms·

How come? What better way to test your function then giving them random/unexpected input and see how that works out. Would you not want to try out what happens

How come? What better way to test your function then giving them random/unexpected input and see how that works out. Would you not want to try out what happens if I pulled a door you built that is only meant for pushing? There will always be that one person who ignores the sign and tries pulling just like there is always that person who uses your function in an unexpected way.

kbenson 10y ago
I took it more as there are fundamentals that could be focused on first. If you know your door is a piece of plywood leaning against the opening, yout time may be better spent getting a real door before you start testing all the ways plywood can fail. If you aren't in the habit of using debris for structural items, sure, spend some time testing failure modes. :)
- lifeisstillgood 10y ago
  Exactly - these days table stakes is having a CI test suite with unit tests with good coverage, good integration tests (with the attendant "so I need four servers plus a couple to fire the tests ... Oh") and wait what you want UI testing ? Oh.
runeks 10y ago
The issue I have with tests, is that I feel like I'm just implementing my algorithm a second time, in a different wording (language/domain), hoping that this will catch some unknown bug(s). If a test suite covers all aspects of an implementation, it's basically a second implementation, and then the question becomes whether the time spent writing this test code would actually have been better spent looking over the actual code that defines your algorithm. If we can't detect errors in code by reading it, I don't think tests will help us much. Furthermore: if our algorithms require a second "test implementation", shouldn't we also write a test for the test, to confirm that the written test actually tests correctly? What good is a broken test? If you've built a plywood door, and a machine for testing that door, wouldn't it be useful to have a machine that tests that your testing machine actually tests the door properly? All jokes aside, I really enjoy automated, high-level tests, which mimic actual user behaviour. I think this is different because we're no longer implementing an algorithm a second time around (as a test), but rather encoding user behaviour (of the algorithm) into an algorithm (new code).
- sverige 10y ago
  > If a test suite covers all aspects of an implementation, it's basically a second implementation, and then the question becomes whether the time spent writing this test code would actually have been better spent looking over the actual code that defines your algorithm. If we can't detect errors in code by reading it, I don't think tests will help us much. The OpenBSD code has been audited many times by very experienced and careful C experts. Fuzzing apparently found the very few things they hadn't already found by reading the code.
  - cLeEOGPw 10y ago
    And it if I understand OpenBSD goal correctly, they take any kind of edge they can get to find those missed parts. Their formula for what is worth essentially boils down to "if this approach can find at least one bug no other approach would do it, we will use it".
  - runeks 10y ago
    > The OpenBSD code has been audited many times by very experienced and careful C experts. Fuzzing apparently found the very few things they hadn't already found by reading the code. That's definitely comforting, but I guess my point is that the fact that bugs can even hide in plain sight in the first place is the root cause of the problem. If we can look at a specification of what something is supposed to do - the code - and not see that it does something entirely different, then the problem is with the programming/specification language. The very purpose of a programming language is to make the specification of computer programs readable (rather than reading assembly). If a specification language needs a a computer program to go through the specification and see if it specifies things that we didn't intend to, then that specification language misses the very point of being a language in which to specify things in the first place.
    - et1337 10y ago
      I don't think it's that simple. A lot of bugs are the result of disparate pieces of code -- each of which is fairly easy to verify by reading it -- interacting in an unforeseen way.

kbenson 10y ago

I took it more as there are fundamentals that could be focused on first. If you know your door is a piece of plywood leaning against the opening, yout time may be better spent getting a real door before you start testing all the ways plywood can fail. If you aren't in the habit of using debris for structural items, sure, spend some time testing failure modes. :)

lifeisstillgood 10y ago
Exactly - these days table stakes is having a CI test suite with unit tests with good coverage, good integration tests (with the attendant "so I need four servers plus a couple to fire the tests ... Oh") and wait what you want UI testing ? Oh.

runeks 10y ago

The issue I have with tests, is that I feel like I'm just implementing my algorithm a second time, in a different wording (language/domain), hoping that this will catch some unknown bug(s). If a test suite covers all aspects of an implementation, it's basically a second implementation, and then the question becomes whether the time spent writing this test code would actually have been better spent looking over the actual code that defines your algorithm. If we can't detect errors in code by reading it, I don't think tests will help us much. Furthermore: if our algorithms require a second "test implementation", shouldn't we also write a test for the test, to confirm that the written test actually tests correctly? What good is a broken test? If you've built a plywood door, and a machine for testing that door, wouldn't it be useful to have a machine that tests that your testing machine actually tests the door properly? All jokes aside, I really enjoy automated, high-level tests, which mimic actual user behaviour. I think this is different because we're no longer implementing an algorithm a second time around (as a test), but rather encoding user behaviour (of the algorithm) into an algorithm (new code).

sverige 10y ago
> If a test suite covers all aspects of an implementation, it's basically a second implementation, and then the question becomes whether the time spent writing this test code would actually have been better spent looking over the actual code that defines your algorithm. If we can't detect errors in code by reading it, I don't think tests will help us much. The OpenBSD code has been audited many times by very experienced and careful C experts. Fuzzing apparently found the very few things they hadn't already found by reading the code.
- cLeEOGPw 10y ago
  And it if I understand OpenBSD goal correctly, they take any kind of edge they can get to find those missed parts. Their formula for what is worth essentially boils down to "if this approach can find at least one bug no other approach would do it, we will use it".
- runeks 10y ago
  > The OpenBSD code has been audited many times by very experienced and careful C experts. Fuzzing apparently found the very few things they hadn't already found by reading the code. That's definitely comforting, but I guess my point is that the fact that bugs can even hide in plain sight in the first place is the root cause of the problem. If we can look at a specification of what something is supposed to do - the code - and not see that it does something entirely different, then the problem is with the programming/specification language. The very purpose of a programming language is to make the specification of computer programs readable (rather than reading assembly). If a specification language needs a a computer program to go through the specification and see if it specifies things that we didn't intend to, then that specification language misses the very point of being a language in which to specify things in the first place.
  - et1337 10y ago
    I don't think it's that simple. A lot of bugs are the result of disparate pieces of code -- each of which is fairly easy to verify by reading it -- interacting in an unforeseen way.