4 ms·
Many of the benchmarks I have seen in this space suffer from the Texas Sharpshooter fallacy, where you shoot first and then paint a target around the hole. If
by 29athrowaway 2y ago
Many of the benchmarks I have seen in this space suffer from the Texas Sharpshooter fallacy, where you shoot first and then paint a target around the hole.
If you create a benchmark and your product outperforms everything else, it could mean many things. Overfitting being one of them.
- prats226 2y agoThat's an interesting point. The bias might or might not be intentional. From the benchmarks I have seen, lot of tools solve slightly different problems altogether and also target slightly different data distribution and in the end have to build best solution around it. Which is why publishing open benchmarks is first step where there is public scrutiny around whether then benchmark itself irrespective of the results is fair. In the end, the end user will choose the benchmark that's best fit for their usecase or mostly will create a variation of their own, do their own unbiased evaluations.