5 ms·

Hardly "best" practices. "Best" would suggest there are no better practices. I am fed up with unit testing. There is no feedback to tell if unit testing setup

Hardly "best" practices. "Best" would suggest there are no better practices. I am fed up with unit testing. There is no feedback to tell if unit testing setup is complete and if unit tests are comprehensive. I am more and more turning to functional regression testing when entire service is tested as a black box against spec. Since we are testing API this tends to be specified much better than individual components. There is usually already some kind of documentation. API changes much less than individual components which means that if I start doing some kind of refactoring I may change a lot of internals but don't touch API at all. If I touch API then I need to update documentation and inform users so writing tests for that is not a huge additional cost.

chii 6y ago
> unit testing setup is complete coverage reports. They tell you which branch wasnt executed in your test, which can tell you what may be missing in your unit testing. > more turning to functional regression testing when entire service is tested as a black box against spec. functional/regression testing is important, but it can be difficult to use if your app is huge. And you can not easily isolate an area to test, so you always end up testing a common component repeatedly, and thus is slower to run. Not to mention some functional testing mechanisms (such as using a web browser) can be flaky. Both have their place in the automated testing paradigm.
- 2rsf 6y ago
  Although coverage reports are misleading, 100% coverage means that you passed through every line of code, that's it. You don't know if you covered edge cases, it cannot cover concurrency well and although possible it will not tell you whether you used a series of values or long-paths in your code.
  - StevenWaterman 6y ago
    You're right about edge cases in terms of input values, and right about issues with concurrency, but many of the issues you mention are solved by changing your metric. Branch-level coverage tells you what percentage of codepaths are covered. In other words, it would report 50% coverage for: if a: print "a yes" else: print "a no" if b: print "b yes" else: print "b no" test 1: a = false, b = false test 2: a = true, b = true
  - mumblemumble 6y ago
    > every line of code, that's it Yes. And it's worth noting what's left out of that. Passing through every line of code is not the same thing as following all paths through the code. Here's a sampling of code features where that distinction is particularly important: - successive if-statements - switch statements with fallthrough - loops - mutable variables with a large influence The first two are because {{a}, {b}} ≠ {{a, b}}. By that last one, I mean both mutable variables with a large scope, and mutable variables within objects that have a large scope. That's probably the trickiest thing, because, in the presence of mutability like that, it's not just which lines of code were executed that matters, it's also the order in which they were executed. That doesn't strike me as a pretty picture. Overall, it implies that automated code coverage metrics are the least useful in the kinds of codebases where they are most desirable.
- lmilcin 6y ago
  Unfortunately coverage measures are shit. Real coverage test would test coverage OF THE SPECIFICATION. So for a list of statements describing how your service should behave in various situation you would want to know which are being verified by the unit test. Checking lines of code is pointless because line of code would usually be required for multiple specification statements (requirements). There is barely any connection between having 100% line coverage and covering entire component specification. I think unit coverage are those measures where somebody thought they need to measure something but they could either not define what to measure or they deciding that measuring the right thing is too difficult so we can measure something else instead and managers will be happy anyway.
pron 6y ago
> There is no feedback to tell if unit testing setup is complete and if unit tests are comprehensive. Consider using mutation testing [1] tools like https://pitest.org/ https://pitest.org/ to test the quality of your tests. Having said that, unit tests obviously shouldn't be your only quality assurance practice. [1]: https://en.wikipedia.org/wiki/Mutation_testing https://en.wikipedia.org/wiki/Mutation_testing
RHSeeger 6y ago
> There is no feedback to tell if unit testing setup is complete and if unit tests are comprehensive. Unit tests are neither required or expected to be "complete". At no point do unit tests tell you "this code is perfect and correct". Rather, they identify where code is incorrect. This is valuable information. > I am more and more turning to functional regression testing when entire service is tested as a black box against spec. Both types of tests (and others) are useful.
- lmilcin 6y ago
  What I mean by feedback is that, when a developer works on functionality he will get a feedback from somewhere. It has to do what it is supposed to do whether the developer understood the requirements or didn't and the users are going to point it out with bug reports or incidents. On the other hand if the developer misunderstood the requirements or completely bungled the unit tests (for example they always pass) then there is no feedback to tell. Everything is fine as long as tests pass. Of course, one could say that the test should be first written to not pass and then fixed by implementing the feature, but who's going to check that? There isn't easy way to tell me if developers are doing good job with tests other than to delve into each feature and understand it and look at the tests. Also unit tests test components that typically are specified by the developer (so he makes both spec and the component and the unit test). With functional testing at least one of those is verified by the user and you get some feedback -- if the user reports the service is not working properly you make a functional test to replicate the issue and then fix the service so that it behaves correctly according to the written test. If the user is another development team it is entirely possible, in a mature org, that the other team can write the test.
  - RHSeeger 6y ago
    > It has to do what it is supposed to do whether the developer understood the requirements or didn't and the users are going to point it out with bug reports or incidents. This sounds like you're saying that one should leave testing of the code to the end user, which obviously makes no sense. I'm going to assume it really means just "things the developer missed". So that that point, we have the developer testing for things the developer thought of. We're in agreement there. When I write code, I tend to make sure the individual pieces work (sort actually sorts, it doesn't blow up on null values, maybe it has too maintain order for equivalent items, maybe it has an option to sort null values first or last). As time goes on, I may find a bug or a change (or misunderstanding of requirements). When that happens, I need to go modify the code to account for it. When I do that, I want to make sure that everything that worked before still works. If I wrote automated tests originally, all I need to do it rerun my tests to get back to "everything I knew to work before, still works". If I did not, then I need to manually test everything. Both are ways of regression testing; one of them is a lot faster.
gowld 6y ago
What is your feedback to tell if your functional tests and your spec is comprehenive?
- lmilcin 6y ago
  The feedback loop is this: 1. When the user says the service doesn't work (which means it doesn't work according to spec) we look what is at fault (is it the user that misunderstood the spec, the spec is faulty or the implementation is faulty). 2. If the spec or implementation is faulty we ensure there is a test that checks the correct behavior (and fix the spec if necessary). For example, the user might submit a piece of logic that in his view should work correctly but it does not. This might get distilled to a test. 3. We ensure the test fails (which means we were able to reproduce the error). 4. We fix the code and ensure the new version passes the test. 5. We get back to the user to confirm. Now, if you read it you will notice following: - after completion of the process we are left with a test to check, so as users complain with problems and incidents the spec and library of tests checking the spec gets updated. This is the feedback loop that updates the library with relevant tests over time ensuring more complete suite. - this is no different form the idea of TDD, just implemented for functional testing. It is not like there is something fundamentally wrong with TDD, it's that TDD cannot be effectively forced on developers. On the other hand, it can be enforced for functional testing because it is just a simple fix to incident/bug handling process.
sabujp 6y ago
I agree. You can get rid of lots of unit tests if you do functional, integration, and performance testing of your APIs.

chii 6y ago

> unit testing setup is complete coverage reports. They tell you which branch wasnt executed in your test, which can tell you what may be missing in your unit testing. > more turning to functional regression testing when entire service is tested as a black box against spec. functional/regression testing is important, but it can be difficult to use if your app is huge. And you can not easily isolate an area to test, so you always end up testing a common component repeatedly, and thus is slower to run. Not to mention some functional testing mechanisms (such as using a web browser) can be flaky. Both have their place in the automated testing paradigm.

2rsf 6y ago
Although coverage reports are misleading, 100% coverage means that you passed through every line of code, that's it. You don't know if you covered edge cases, it cannot cover concurrency well and although possible it will not tell you whether you used a series of values or long-paths in your code.
- StevenWaterman 6y ago
  You're right about edge cases in terms of input values, and right about issues with concurrency, but many of the issues you mention are solved by changing your metric. Branch-level coverage tells you what percentage of codepaths are covered. In other words, it would report 50% coverage for: if a: print "a yes" else: print "a no" if b: print "b yes" else: print "b no" test 1: a = false, b = false test 2: a = true, b = true
- mumblemumble 6y ago
  > every line of code, that's it Yes. And it's worth noting what's left out of that. Passing through every line of code is not the same thing as following all paths through the code. Here's a sampling of code features where that distinction is particularly important: - successive if-statements - switch statements with fallthrough - loops - mutable variables with a large influence The first two are because {{a}, {b}} ≠ {{a, b}}. By that last one, I mean both mutable variables with a large scope, and mutable variables within objects that have a large scope. That's probably the trickiest thing, because, in the presence of mutability like that, it's not just which lines of code were executed that matters, it's also the order in which they were executed. That doesn't strike me as a pretty picture. Overall, it implies that automated code coverage metrics are the least useful in the kinds of codebases where they are most desirable.
lmilcin 6y ago
Unfortunately coverage measures are shit. Real coverage test would test coverage OF THE SPECIFICATION. So for a list of statements describing how your service should behave in various situation you would want to know which are being verified by the unit test. Checking lines of code is pointless because line of code would usually be required for multiple specification statements (requirements). There is barely any connection between having 100% line coverage and covering entire component specification. I think unit coverage are those measures where somebody thought they need to measure something but they could either not define what to measure or they deciding that measuring the right thing is too difficult so we can measure something else instead and managers will be happy anyway.

pron 6y ago

> There is no feedback to tell if unit testing setup is complete and if unit tests are comprehensive. Consider using mutation testing [1] tools like https://pitest.org/ https://pitest.org/ to test the quality of your tests. Having said that, unit tests obviously shouldn't be your only quality assurance practice. [1]: https://en.wikipedia.org/wiki/Mutation_testing https://en.wikipedia.org/wiki/Mutation_testing

RHSeeger 6y ago

> There is no feedback to tell if unit testing setup is complete and if unit tests are comprehensive. Unit tests are neither required or expected to be "complete". At no point do unit tests tell you "this code is perfect and correct". Rather, they identify where code is incorrect. This is valuable information. > I am more and more turning to functional regression testing when entire service is tested as a black box against spec. Both types of tests (and others) are useful.

lmilcin 6y ago
What I mean by feedback is that, when a developer works on functionality he will get a feedback from somewhere. It has to do what it is supposed to do whether the developer understood the requirements or didn't and the users are going to point it out with bug reports or incidents. On the other hand if the developer misunderstood the requirements or completely bungled the unit tests (for example they always pass) then there is no feedback to tell. Everything is fine as long as tests pass. Of course, one could say that the test should be first written to not pass and then fixed by implementing the feature, but who's going to check that? There isn't easy way to tell me if developers are doing good job with tests other than to delve into each feature and understand it and look at the tests. Also unit tests test components that typically are specified by the developer (so he makes both spec and the component and the unit test). With functional testing at least one of those is verified by the user and you get some feedback -- if the user reports the service is not working properly you make a functional test to replicate the issue and then fix the service so that it behaves correctly according to the written test. If the user is another development team it is entirely possible, in a mature org, that the other team can write the test.
- RHSeeger 6y ago
  > It has to do what it is supposed to do whether the developer understood the requirements or didn't and the users are going to point it out with bug reports or incidents. This sounds like you're saying that one should leave testing of the code to the end user, which obviously makes no sense. I'm going to assume it really means just "things the developer missed". So that that point, we have the developer testing for things the developer thought of. We're in agreement there. When I write code, I tend to make sure the individual pieces work (sort actually sorts, it doesn't blow up on null values, maybe it has too maintain order for equivalent items, maybe it has an option to sort null values first or last). As time goes on, I may find a bug or a change (or misunderstanding of requirements). When that happens, I need to go modify the code to account for it. When I do that, I want to make sure that everything that worked before still works. If I wrote automated tests originally, all I need to do it rerun my tests to get back to "everything I knew to work before, still works". If I did not, then I need to manually test everything. Both are ways of regression testing; one of them is a lot faster.

gowld 6y ago

What is your feedback to tell if your functional tests and your spec is comprehenive?

lmilcin 6y ago
The feedback loop is this: 1. When the user says the service doesn't work (which means it doesn't work according to spec) we look what is at fault (is it the user that misunderstood the spec, the spec is faulty or the implementation is faulty). 2. If the spec or implementation is faulty we ensure there is a test that checks the correct behavior (and fix the spec if necessary). For example, the user might submit a piece of logic that in his view should work correctly but it does not. This might get distilled to a test. 3. We ensure the test fails (which means we were able to reproduce the error). 4. We fix the code and ensure the new version passes the test. 5. We get back to the user to confirm. Now, if you read it you will notice following: - after completion of the process we are left with a test to check, so as users complain with problems and incidents the spec and library of tests checking the spec gets updated. This is the feedback loop that updates the library with relevant tests over time ensuring more complete suite. - this is no different form the idea of TDD, just implemented for functional testing. It is not like there is something fundamentally wrong with TDD, it's that TDD cannot be effectively forced on developers. On the other hand, it can be enforced for functional testing because it is just a simple fix to incident/bug handling process.

sabujp 6y ago

I agree. You can get rid of lots of unit tests if you do functional, integration, and performance testing of your APIs.