3 ms·

edited to better explain So... If the variance of a particular implementation from run to run is significant then one has to question how real any gain vs the

by sgt101 11y ago

edited to better explain So... If the variance of a particular implementation from run to run is significant then one has to question how real any gain vs the bottom result obtained is? In the old days we used to do things like cross validate 30+ times and give results x vs y vs z with some confidence interval. I believe that any implementation of a particular algorithm should be almost exactly the same as any other implementation if it is correct. Therefore, over a number of runs the results should be the same.

Analog24 11y ago
I agree in the context of the non-deterministic algorithms. If there is too much variance between runs then the ones that do well are probably just overfitting the data and won't actually perform well when deployed. However, I wouldn't consider differences in each implementation as a measure of variance. This is likely the result of differences in parameter choices or subtleties in the algorithms. There are multiple ways to train a NN, for example. So I doubt any of the examples shown in the study are true apples-to-apples comparisons.
Analog24 11y ago
I think you have a stricter definition of what an algorithm is than what is used in the context of this study. The general concept of the random forest algorithm (for example) is the same in each implementation but the exact details of how that general algorithm is implemented (the exact algorithm) is most likely not the same. Therefor, you shouldn't expect to get completely identical results regardless of how much data you train them with. They should all be in the same neighborhood though, which they are, for the most part, in the results from the study.