4 ms·
> Not sure if any of the test-driven development people have thought about this, but the same principle also applies there: if you debug and fix the code until
by shajznnckfke 6y ago
> Not sure if any of the test-driven development people have thought about this, but the same principle also applies there: if you debug and fix the code until it passes the tests, you can overfit the tests. It's no longer a good measure for code quality, once it has been explicitly optimized. You'd need new, previously unseen test cases. It would be an obvious mistake in machine learning to simply add failed test examples into the training set and rejoyce that the new model can now deal with those previously difficult cases.
This is a great point, and it articulates some challenges that I’ve seen with test-driven development.
I would say that if you can write tests that fully specify the desired functionality, rather than merely check a few possible inputs, it’s less of an issue. This is a reason to try to build things with less knobs to twiddle, so the space of inputs has lower dimensionality.
- virgilp 6y agoYes but then the tests would formally describe the requirements - i.e.they would be a de-facto implementation. Congratulations, you just coded an (indirect) solution to your problem (a solution that you postulate is correct/bug-free)
- tonyhb 6y agoAre you saying this like it's a bad thing? Tests describing requirements acts as documentation, validation, and regression prevention with future refactors.
- virgilp 6y agoMy beef is with the assumption that tests can fully encode the requirements - to the point where any bug would trigger a test failure. Having such a test suite is no simpler than having a perfect implementation of the requirements - i.e. it's probably only feasible, at all, in the simplest/ "didactic" cases. Having a perfect implementation is not bad, of course. It's just not realistic.