4 ms·

You can’t test everything. To believe we can is just hubris; even if we get every function to have coverage, we won’t have coverage of the full range of inputs

You can’t test everything. To believe we can is just hubris; even if we get every function to have coverage, we won’t have coverage of the full range of inputs unless all that’s being done is some very simple programming. The combinatorial explosion is fast. Testing definitely has value, but when testing you always have to make the assumption that you can’t test everything, and then the trick is deciding what to test versus what not to. I think people forget this and focus on having every function tested, every interface having coverage, etc.

capableweb 6y ago
Quickcheck, clojure.spec and similar efforts are getting damn close to it, and using clojure.spec raised my confidence in my code a lot, more than unit testing ever did. Since generative/property based testing is gaining more ground, we'll get better tools as well. Not to say there is not still a place for manually written unit tests.
- stingraycharles 6y ago
  While I am a big fan of property based testing, it’s hardly able to “test everything”. It’s great for testing pure functions, but as soon as things have side effects, property based testing becomes very cumbersome. Unfortunately, this is precisely where a large amount of bugs originate from. And I say all this as someone who has pursued this very idea of property testing side effects for years and in the end had to admit failure. It just is too much work get right (eg uuids being generated not unique, to name just one example), and your test runs become incredibly slow, which is a killer in many REPL-driven workflows.
  - PopeDotNinja 6y ago
    I have a (probably horrible) joke reply to the question "what is a monad?" I say "that's where you put the real programming." I have also never done any serious with Haskell, so don't make the mistake of taking me seriously on this :)
    - gindely 6y ago
      however true the statement may be, it's not therefore an abandonment of pure functional programming. You don't have to program in `IO` or `MonadIO m => m`. Expressing your state in terms of pure functions of input events, as can be done with an FRP like Reflex, means you can test all your logic purely. You can be confident that you'll only request to launch the nukes when the nukes need to be launched. Then you have to make yourself confident that, when the system asks the nukes to launch, it sends the right signal. But that's mockable by running the system attached to device that reads the signal and reports its correctness. And making sure the nukes take off, aim correctly, and land on their target is not my department (says Werner von Braun).
  - Jtsummers 6y ago
    If you can't capture the side effects to determine if they're correct, you can't test it period. Property based testing, unit testing, or integration testing, doesn't matter.
commandlinefan 6y ago
> You can’t test everything. The problem with that (tautological) statement is that too many people read it as: "you can't test everything, so don't bother testing anything".
- ironmagma 6y ago
  People believe that even if you don’t say it, however. While TDD I’m sure started with good intentions, the reason it hasn’t seen as wide adoption as it could have, in my opinion, is that it is frequently stated as requiring that everything be developed from tests. I find it much more convincing if someone were to say, “test what’s easy” instead, that’s all.

capableweb 6y ago

Quickcheck, clojure.spec and similar efforts are getting damn close to it, and using clojure.spec raised my confidence in my code a lot, more than unit testing ever did. Since generative/property based testing is gaining more ground, we'll get better tools as well. Not to say there is not still a place for manually written unit tests.

stingraycharles 6y ago
While I am a big fan of property based testing, it’s hardly able to “test everything”. It’s great for testing pure functions, but as soon as things have side effects, property based testing becomes very cumbersome. Unfortunately, this is precisely where a large amount of bugs originate from. And I say all this as someone who has pursued this very idea of property testing side effects for years and in the end had to admit failure. It just is too much work get right (eg uuids being generated not unique, to name just one example), and your test runs become incredibly slow, which is a killer in many REPL-driven workflows.
- PopeDotNinja 6y ago
  I have a (probably horrible) joke reply to the question "what is a monad?" I say "that's where you put the real programming." I have also never done any serious with Haskell, so don't make the mistake of taking me seriously on this :)
  - gindely 6y ago
    however true the statement may be, it's not therefore an abandonment of pure functional programming. You don't have to program in `IO` or `MonadIO m => m`. Expressing your state in terms of pure functions of input events, as can be done with an FRP like Reflex, means you can test all your logic purely. You can be confident that you'll only request to launch the nukes when the nukes need to be launched. Then you have to make yourself confident that, when the system asks the nukes to launch, it sends the right signal. But that's mockable by running the system attached to device that reads the signal and reports its correctness. And making sure the nukes take off, aim correctly, and land on their target is not my department (says Werner von Braun).
- Jtsummers 6y ago
  If you can't capture the side effects to determine if they're correct, you can't test it period. Property based testing, unit testing, or integration testing, doesn't matter.

commandlinefan 6y ago

> You can’t test everything. The problem with that (tautological) statement is that too many people read it as: "you can't test everything, so don't bother testing anything".

ironmagma 6y ago
People believe that even if you don’t say it, however. While TDD I’m sure started with good intentions, the reason it hasn’t seen as wide adoption as it could have, in my opinion, is that it is frequently stated as requiring that everything be developed from tests. I find it much more convincing if someone were to say, “test what’s easy” instead, that’s all.