4 ms·

These algorithms are all fairly complicated with numerous parameters involved in the calculations. Most of the packages (I can't speak for all of them since I'm

These algorithms are all fairly complicated with numerous parameters involved in the calculations. Most of the packages (I can't speak for all of them since I'm not familiar with all of them) make it easy to implement these algorithms by providing reasonable default values for most of the necessary parameters. It's very unlikely that they are set to identical values, thus leading to differing results. In addition, some of the algorithms are non-deterministic. Random forests, as the name implies, involves randomly setting the decision tree parameters numerous times. You can run the same algorithm with the same implementation and get different results. Likewise with NN's, it all depends how you set the weights initially, which is usually a non-deterministic process as well. To sum it up, I don't think that obtaining different results from different implementations of the same ML algorithm is indicative of a bug. It's actually expected in many situations.

sgt101 11y ago
edited to better explain So... If the variance of a particular implementation from run to run is significant then one has to question how real any gain vs the bottom result obtained is? In the old days we used to do things like cross validate 30+ times and give results x vs y vs z with some confidence interval. I believe that any implementation of a particular algorithm should be almost exactly the same as any other implementation if it is correct. Therefore, over a number of runs the results should be the same.
- Analog24 11y ago
  I agree in the context of the non-deterministic algorithms. If there is too much variance between runs then the ones that do well are probably just overfitting the data and won't actually perform well when deployed. However, I wouldn't consider differences in each implementation as a measure of variance. This is likely the result of differences in parameter choices or subtleties in the algorithms. There are multiple ways to train a NN, for example. So I doubt any of the examples shown in the study are true apples-to-apples comparisons.
- Analog24 11y ago
  I think you have a stricter definition of what an algorithm is than what is used in the context of this study. The general concept of the random forest algorithm (for example) is the same in each implementation but the exact details of how that general algorithm is implemented (the exact algorithm) is most likely not the same. Therefor, you shouldn't expect to get completely identical results regardless of how much data you train them with. They should all be in the same neighborhood though, which they are, for the most part, in the results from the study.

sgt101 11y ago

edited to better explain So... If the variance of a particular implementation from run to run is significant then one has to question how real any gain vs the bottom result obtained is? In the old days we used to do things like cross validate 30+ times and give results x vs y vs z with some confidence interval. I believe that any implementation of a particular algorithm should be almost exactly the same as any other implementation if it is correct. Therefore, over a number of runs the results should be the same.

Analog24 11y ago
I agree in the context of the non-deterministic algorithms. If there is too much variance between runs then the ones that do well are probably just overfitting the data and won't actually perform well when deployed. However, I wouldn't consider differences in each implementation as a measure of variance. This is likely the result of differences in parameter choices or subtleties in the algorithms. There are multiple ways to train a NN, for example. So I doubt any of the examples shown in the study are true apples-to-apples comparisons.
Analog24 11y ago
I think you have a stricter definition of what an algorithm is than what is used in the context of this study. The general concept of the random forest algorithm (for example) is the same in each implementation but the exact details of how that general algorithm is implemented (the exact algorithm) is most likely not the same. Therefor, you shouldn't expect to get completely identical results regardless of how much data you train them with. They should all be in the same neighborhood though, which they are, for the most part, in the results from the study.