3 ms·

What I have trouble with is even gauging the quality of existing tests.

crdoconnor 8y ago
If you track failures and categorizing them as either failures that caught a bug or false positives then you basically have a good measure for test quality.
hinkley 8y ago
Good tests are hard. Beware of anyone who tries to pretend otherwise. To build on what crdoconnor said, empirical evidence helps. Do a post mortem every time. Try to recall what was going on when the problem started and see if there are any reasonable measures you can undertake in the future. It's the typical iterative process of learning something as you go along. When you discover you made a mistake, try to figure out how not to make it again. Every time you get a regression in the code ask yourself "didn't we have a test for that?" If the answer is 'no' then think back to the last time someone (maybe you!) talked you out of spending more time on tests. Is the hassle now worth the hassle you avoided then? Occasionally this is the case, and the response is "shit happens" (if this is not rare, then you might need an attitude adjustment). If you find that prevention would have been better, try to remember that next time you're thinking about cutting back on tests. If the answer was 'yes', then something has gone wrong with your testing strategy. Was it always broken, or did subsequent code changes break it? When you fix the test(s), would the way they look now have made sense at the time you wrote them to begin with? Should new tests you write look more like these do? Should you go back right now and fix other tests to make them look this way? Just keep thinking. Verify your assumptions. Chuck the ones that don't hold water. Watch how other people deal with similar problems, and steal their ideas (or learn from their mistakes).
adrianN 8y ago
Mutation testing is a good way to assess test quality. You ideally want every mutation to be caught be a very small number of tests.
- aiCeivi9 8y ago
  As far I can tell pi-test can't scale even to medium sized projects, time complexity is close to (unit tests * classes). Are the other tools (maybe for other languages) that deal with it better, without the need to split code in smaller parts?

If you track failures and categorizing them as either failures that caught a bug or false positives then you basically have a good measure for test quality.

hinkley 8y ago

Good tests are hard. Beware of anyone who tries to pretend otherwise. To build on what crdoconnor said, empirical evidence helps. Do a post mortem every time. Try to recall what was going on when the problem started and see if there are any reasonable measures you can undertake in the future. It's the typical iterative process of learning something as you go along. When you discover you made a mistake, try to figure out how not to make it again. Every time you get a regression in the code ask yourself "didn't we have a test for that?" If the answer is 'no' then think back to the last time someone (maybe you!) talked you out of spending more time on tests. Is the hassle now worth the hassle you avoided then? Occasionally this is the case, and the response is "shit happens" (if this is not rare, then you might need an attitude adjustment). If you find that prevention would have been better, try to remember that next time you're thinking about cutting back on tests. If the answer was 'yes', then something has gone wrong with your testing strategy. Was it always broken, or did subsequent code changes break it? When you fix the test(s), would the way they look now have made sense at the time you wrote them to begin with? Should new tests you write look more like these do? Should you go back right now and fix other tests to make them look this way? Just keep thinking. Verify your assumptions. Chuck the ones that don't hold water. Watch how other people deal with similar problems, and steal their ideas (or learn from their mistakes).

adrianN 8y ago

Mutation testing is a good way to assess test quality. You ideally want every mutation to be caught be a very small number of tests.

aiCeivi9 8y ago
As far I can tell pi-test can't scale even to medium sized projects, time complexity is close to (unit tests * classes). Are the other tools (maybe for other languages) that deal with it better, without the need to split code in smaller parts?