3 ms·

Not sure if any of the test-driven development people have thought about this, but the same principle also applies there: if you debug and fix the code until it

Not sure if any of the test-driven development people have thought about this, but the same principle also applies there: if you debug and fix the code until it passes the tests, you can overfit the tests. It's no longer a good measure for code quality, once it has been explicitly optimized. You'd need new, previously unseen test cases. It would be an obvious mistake in machine learning to simply add failed test examples into the training set and rejoyce that the new model can now deal with those previously difficult cases. It's related to Goodheart's law: "When a measure becomes a target, it ceases to be a good measure". It also works in one's personal life. If you find that you made a mistake, don't just fix that particular thing, go back and see what other similar things may need fixing. It connects to the idea of fixing the deeper reason than fixing the symptoms. "Deeper reason" just means, something that generalizes. By the way, you need to be careful about your data split. This always depends on how you intend to use the model. If you intend to use it on new, unseen patients, then the train and test data cannot overlap with respect to people. Another obvious case is videos: You can't just take every even frame as a training sample and every odd frame as test. Even though technically you would be testing on non-overlapping data, the performance measure would be biased compared to real world performance. Or if you want to classify burglars from security camera footage, you may need to test it on new camera setups from different houses if you intend to deploy it to new houses. If your scenario is such that you'd perform training on each new site and run a location specific model, you can test on images from the same site. You always have to use your brain to decide what to do.

shajznnckfke 6y ago
> Not sure if any of the test-driven development people have thought about this, but the same principle also applies there: if you debug and fix the code until it passes the tests, you can overfit the tests. It's no longer a good measure for code quality, once it has been explicitly optimized. You'd need new, previously unseen test cases. It would be an obvious mistake in machine learning to simply add failed test examples into the training set and rejoyce that the new model can now deal with those previously difficult cases. This is a great point, and it articulates some challenges that I’ve seen with test-driven development. I would say that if you can write tests that fully specify the desired functionality, rather than merely check a few possible inputs, it’s less of an issue. This is a reason to try to build things with less knobs to twiddle, so the space of inputs has lower dimensionality.
- virgilp 6y ago
  Yes but then the tests would formally describe the requirements - i.e.they would be a de-facto implementation. Congratulations, you just coded an (indirect) solution to your problem (a solution that you postulate is correct/bug-free)
  - tonyhb 6y ago
    Are you saying this like it's a bad thing? Tests describing requirements acts as documentation, validation, and regression prevention with future refactors.
    - virgilp 6y ago
      My beef is with the assumption that tests can fully encode the requirements - to the point where any bug would trigger a test failure. Having such a test suite is no simpler than having a perfect implementation of the requirements - i.e. it's probably only feasible, at all, in the simplest/ "didactic" cases. Having a perfect implementation is not bad, of course. It's just not realistic.
throwaway4007 6y ago
I was super into TDD and never thought that there was such a thing as "TDD going too far" until I read that post by an ex-Oracle employee: https://news.ycombinator.com/item?id=18442941 https://news.ycombinator.com/item?id=18442941
mumblemumble 6y ago
This is a big reason why I prefer property testing to unit testing. One can think of focusing on the abstract behavior and invariants of the code, rather than just the expected outputs for specific inputs, as being somewhat akin to taking the time to plan out your experiment and the statistical tests you want to perform ahead of time, do your power analysis, etc. It's a fair bit more up-front work, but helps to ensure that you have a crystal clear idea of what you're going to do before you start doing it.

shajznnckfke 6y ago

> Not sure if any of the test-driven development people have thought about this, but the same principle also applies there: if you debug and fix the code until it passes the tests, you can overfit the tests. It's no longer a good measure for code quality, once it has been explicitly optimized. You'd need new, previously unseen test cases. It would be an obvious mistake in machine learning to simply add failed test examples into the training set and rejoyce that the new model can now deal with those previously difficult cases. This is a great point, and it articulates some challenges that I’ve seen with test-driven development. I would say that if you can write tests that fully specify the desired functionality, rather than merely check a few possible inputs, it’s less of an issue. This is a reason to try to build things with less knobs to twiddle, so the space of inputs has lower dimensionality.

virgilp 6y ago
Yes but then the tests would formally describe the requirements - i.e.they would be a de-facto implementation. Congratulations, you just coded an (indirect) solution to your problem (a solution that you postulate is correct/bug-free)
- tonyhb 6y ago
  Are you saying this like it's a bad thing? Tests describing requirements acts as documentation, validation, and regression prevention with future refactors.
  - virgilp 6y ago
    My beef is with the assumption that tests can fully encode the requirements - to the point where any bug would trigger a test failure. Having such a test suite is no simpler than having a perfect implementation of the requirements - i.e. it's probably only feasible, at all, in the simplest/ "didactic" cases. Having a perfect implementation is not bad, of course. It's just not realistic.

throwaway4007 6y ago

I was super into TDD and never thought that there was such a thing as "TDD going too far" until I read that post by an ex-Oracle employee: https://news.ycombinator.com/item?id=18442941 https://news.ycombinator.com/item?id=18442941

mumblemumble 6y ago

This is a big reason why I prefer property testing to unit testing. One can think of focusing on the abstract behavior and invariants of the code, rather than just the expected outputs for specific inputs, as being somewhat akin to taking the time to plan out your experiment and the statistical tests you want to perform ahead of time, do your power analysis, etc. It's a fair bit more up-front work, but helps to ensure that you have a crystal clear idea of what you're going to do before you start doing it.