3 ms·
One thing I appreciate in supervised machine learning is that it's widely acknowledged that one should not test on training/tuning data. For both computer bench
by gok 6y ago
One thing I appreciate in supervised machine learning is that it's widely acknowledged that one should not test on training/tuning data. For both computer benchmarks and car crash tests, this is not the case. Both fields routinely work to optimize for a test, which leads to dramatic overfitting. This would be like passing out a standardized test years before students take it.
This seems solvable. For benchmarks, there should be held out tests that don't get distributed to (say) compiler authors until after the results are published. For crash tests, parameters like overlap and speed should be randomized each year so that manufacturers are forced to optimize for a range of possible crash scenarios.
- ghaff 6y ago>For benchmarks, there should be held out tests that don't get distributed to (say) compiler authors until after the results are published. The way you really "solve" the problem is to have a third-party test organization run the benchmark and only provide the final result--and you only get one shot for a given generation of hardware/software. (This is sort of how standardized tests work.) The problem is that you're now benchmarking with a totally opaque test that you have to trust is reasonably representative of the type of workload it claims to represent as well as trust that the benchmark isn't favoring specific vendors or design decisions that are irrelevant to real world workloads. I assume there have been benchmarks at various times that were similar to this. Certainly, I'm pretty sure source code for benchmark suites hasn't always been available.