3 ms·

For Testing: 1. Tests aren't necessarily for you, they could be to convince somebody else that your solution is robust enough to be used for their use-cases.

by bArray 6y ago

For Testing: 1. Tests aren't necessarily for you, they could be to convince somebody else that your solution is robust enough to be used for their use-cases. 2. Test's aren't necessarily for today. They could be about preventing code regression in the future (some dev comes in and makes "performance" changes for example, not realizing they are breaking the code for others). 3. They can be about testing that the code behaves how you believe it behaves. Sometimes even simple code requires a sanity check. 4. 100% test coverage is likely impossible, but if we do find errors, we can learn from them and try to prevent them happening in the future by adding them to the tests. 5. Tests breaking are a good thing. They inform you that some change you are making is changing the behavior of something you thought to be act reliably. 6. With regards to the same mind making the tests, this could also be a useful tool for ensuring a developer of some code thought of most the edge cases when merging some code. "Ah, i see you didn't add minus numbers to the tests, how are those handled?" 7. We should be able to freely rip out tests and add testing as the requirements of the code base change. Generally though, the goal of much code doesn't really change, despite perhaps how exactly it achieves the task does.

leafboi 6y ago
>100% test coverage is likely impossible, but if we do find errors, we can learn from them and try to prevent them happening in the future by adding them to the tests. You do know there's a way to verify a function to a degree of 100% without writing a single test? You do also realize that for even a trivial function f(x) = x + 2, the only way to achieve 100% coverage is to write: assert f(0) == 2 assert f(1) == 3 assert f(2) == 4 assert f(3) == 5 .... assert f(N) == N + 2 where N = infinite. There are infinite possible inputs and infinite possible outputs so to get 100% coverage you need to write infinite tests. Because your "tests" can never even approach this number most of your "coverage" really amounts to a number close to 0%. The question is, why does testing seem to kinda sort of work even when our test coverage covers only an amount roughly equal to 0% of all inputs to the program? It's an interesting question with an interesting answer. Suffice to say "test coverage" is a garbage statistic. One way to look at testing is that you're taking a statistical sample of a population. You take a small sample and do a statistical measurement on that population and if all tests pass for a sample you assume that the correlations implies that the entire population of inputs will pass all tests. So in a sense testing is just science. We try to establish correlations among a given observational data set and we assume that this is true the entire population of data including ones that aren't seen. Except the above isn't actually true either... The reason is most test writers don't randomly sample test input data. Their methodology is very divergent from the way a statistics expert gathers data. Test writers don't write functions that randomly pick a set of data to test... instead they're highly biased in the tests that they write... for example a typical test set can look like this: F(x) = 320302/x assert F(2) == 160151 assert F(0) == error As you can see the above two tests are highly biased with the second test picked in order to deliberately cause an error. Imagine if the test data was randomly picked? It would be highly that the random picker would draw zero as an input test case indicating that statistical sampling may not be the best way to test data.... So if our tests cases are biased then why do tests kind of sort of work? Or do they not actually work? How is the programmer picking a test and how does that influence the overall correctness of their program? Perhaps it's not the test itself or the amount of tests that the programmer is writing but it's more about how the tests influence the way the programmer thinks about the function...? I would say the last micro service I wrote, (in python) was virtually 100% bug free ever since it went into prod. I also didn't write a single unit test for it. I did do some manual testing on the system but the application itself does not have a suite of unit tests to protect it and it has since had 0 bugs ever since it was placed into prod. I would still say a testing suite is still good for new programmers diving into an unfamiliar system attempting to change things haphazardly, but in terms of correctness I question this strict almost religious adherence to unit testing. Food for a thought.
- bArray 6y ago
  > You do know there's a way to verify a function to a degree > of 100% without writing a single test? I carefully tried to avoid the word "function", as mathematically they tend to be well defined. As soon as you have anything even mildly complex or some element of randomness - suddenly the number of required tests to brute force the problem can explode. > I would say the last micro service I wrote, (in python) > was virtually 100% bug free ever since it went into prod. There is never any bugs, until there is. Also there is quite some difference between code that is easy to reason about and code that is not. > I would still say a testing suite is still good for new > programmers diving into an unfamiliar system attempting to > change things haphazardly, but in terms of correctness I > question this strict almost religious adherence to unit > testing. I think relegating bugs to something only new programmers write is unfair. I doubt you have a full compiler in your head or could even begin to consider all possible states of some code that could be considered complex. If your existing code is complex enough, chances are that you already have introduced some bug. To be honest, I don't write high test coverage either for most projects, but I am sure to write tests for code that I either have trouble reasoning about or is of high enough complexity. It happens, even for veterans. Sometimes when pair programming I even spotted very seasoned programmers making such mistakes when they are tired.

by bArray 6y ago

leafboi 6y ago

>100% test coverage is likely impossible, but if we do find errors, we can learn from them and try to prevent them happening in the future by adding them to the tests. You do know there's a way to verify a function to a degree of 100% without writing a single test? You do also realize that for even a trivial function f(x) = x + 2, the only way to achieve 100% coverage is to write: assert f(0) == 2 assert f(1) == 3 assert f(2) == 4 assert f(3) == 5 .... assert f(N) == N + 2 where N = infinite. There are infinite possible inputs and infinite possible outputs so to get 100% coverage you need to write infinite tests. Because your "tests" can never even approach this number most of your "coverage" really amounts to a number close to 0%. The question is, why does testing seem to kinda sort of work even when our test coverage covers only an amount roughly equal to 0% of all inputs to the program? It's an interesting question with an interesting answer. Suffice to say "test coverage" is a garbage statistic. One way to look at testing is that you're taking a statistical sample of a population. You take a small sample and do a statistical measurement on that population and if all tests pass for a sample you assume that the correlations implies that the entire population of inputs will pass all tests. So in a sense testing is just science. We try to establish correlations among a given observational data set and we assume that this is true the entire population of data including ones that aren't seen. Except the above isn't actually true either... The reason is most test writers don't randomly sample test input data. Their methodology is very divergent from the way a statistics expert gathers data. Test writers don't write functions that randomly pick a set of data to test... instead they're highly biased in the tests that they write... for example a typical test set can look like this: F(x) = 320302/x assert F(2) == 160151 assert F(0) == error As you can see the above two tests are highly biased with the second test picked in order to deliberately cause an error. Imagine if the test data was randomly picked? It would be highly that the random picker would draw zero as an input test case indicating that statistical sampling may not be the best way to test data.... So if our tests cases are biased then why do tests kind of sort of work? Or do they not actually work? How is the programmer picking a test and how does that influence the overall correctness of their program? Perhaps it's not the test itself or the amount of tests that the programmer is writing but it's more about how the tests influence the way the programmer thinks about the function...? I would say the last micro service I wrote, (in python) was virtually 100% bug free ever since it went into prod. I also didn't write a single unit test for it. I did do some manual testing on the system but the application itself does not have a suite of unit tests to protect it and it has since had 0 bugs ever since it was placed into prod. I would still say a testing suite is still good for new programmers diving into an unfamiliar system attempting to change things haphazardly, but in terms of correctness I question this strict almost religious adherence to unit testing. Food for a thought.

bArray 6y ago
> You do know there's a way to verify a function to a degree > of 100% without writing a single test? I carefully tried to avoid the word "function", as mathematically they tend to be well defined. As soon as you have anything even mildly complex or some element of randomness - suddenly the number of required tests to brute force the problem can explode. > I would say the last micro service I wrote, (in python) > was virtually 100% bug free ever since it went into prod. There is never any bugs, until there is. Also there is quite some difference between code that is easy to reason about and code that is not. > I would still say a testing suite is still good for new > programmers diving into an unfamiliar system attempting to > change things haphazardly, but in terms of correctness I > question this strict almost religious adherence to unit > testing. I think relegating bugs to something only new programmers write is unfair. I doubt you have a full compiler in your head or could even begin to consider all possible states of some code that could be considered complex. If your existing code is complex enough, chances are that you already have introduced some bug. To be honest, I don't write high test coverage either for most projects, but I am sure to write tests for code that I either have trouble reasoning about or is of high enough complexity. It happens, even for veterans. Sometimes when pair programming I even spotted very seasoned programmers making such mistakes when they are tired.