3 ms·
Technical (Honest) Solution: two holdouts 1. Involved in the build process 2. Never touched until paper metrics are being written, only run once Realisticall
by tomrod 4y ago
Technical (Honest) Solution: two holdouts
1. Involved in the build process
2. Never touched until paper metrics are being written, only run once
Realistically, unlikely to occur however due to the incentives causing publication bias.
- jerpint 4y agoThird (better) option: have a regulating body have a separate, undisclosed test set. If you can't beat it, you can't deploy your model. If you can beat it, you still need to have your models peer reviewed and scrutinized
- tomrod 4y agoThis sounds simple yet I expect data governance will be the bottleneck.
- cm2187 4y agoso the models that fail this one test never get published and the models that succeed get published. And all you have done is to publish a model that predicts that particular history, in other words data fitting.
- tomrod 4y ago> the models that fail this one test never get published Not necessarily -- for comparison purposes one should include, and a negative result is not a bad outcome. But consider Edison's light bulb. Most of the failures don't matter, and a few might have interesting properties to reconsider or tweak down the line. But the major one folks care about is the one that worked. > data fitting Yes, models trained on data and not logic alone fit data.