4 ms·

It's always good to see proper evalations and I am sure that this will provide food for thought and improvement for the tool owners. However - a couple of poin

• lqdc13 11y ago
  Not really a bug in most cases but the assumptions made by the lib that cannot be changed. Also, sometimes more data doesn't help much so this is somewhat relevant even for spark users.
• Analog24 11y ago
  These algorithms are all fairly complicated with numerous parameters involved in the calculations. Most of the packages (I can't speak for all of them since I'm not familiar with all of them) make it easy to implement these algorithms by providing reasonable default values for most of the necessary parameters. It's very unlikely that they are set to identical values, thus leading to differing results. In addition, some of the algorithms are non-deterministic. Random forests, as the name implies, involves randomly setting the decision tree parameters numerous times. You can run the same algorithm with the same implementation and get different results. Likewise with NN's, it all depends how you set the weights initially, which is usually a non-deterministic process as well. To sum it up, I don't think that obtaining different results from different implementations of the same ML algorithm is indicative of a bug. It's actually expected in many situations.
  • sgt101 11y ago
    edited to better explain So... If the variance of a particular implementation from run to run is significant then one has to question how real any gain vs the bottom result obtained is? In the old days we used to do things like cross validate 30+ times and give results x vs y vs z with some confidence interval. I believe that any implementation of a particular algorithm should be almost exactly the same as any other implementation if it is correct. Therefore, over a number of runs the results should be the same.
    • Analog24 11y ago
      I agree in the context of the non-deterministic algorithms. If there is too much variance between runs then the ones that do well are probably just overfitting the data and won't actually perform well when deployed. However, I wouldn't consider differences in each implementation as a measure of variance. This is likely the result of differences in parameter choices or subtleties in the algorithms. There are multiple ways to train a NN, for example. So I doubt any of the examples shown in the study are true apples-to-apples comparisons.
    • Analog24 11y ago
      I think you have a stricter definition of what an algorithm is than what is used in the context of this study. The general concept of the random forest algorithm (for example) is the same in each implementation but the exact details of how that general algorithm is implemented (the exact algorithm) is most likely not the same. Therefor, you shouldn't expect to get completely identical results regardless of how much data you train them with. They should all be in the same neighborhood though, which they are, for the most part, in the results from the study.
• micro_cam 11y ago
  I've implemented a random forest library that achieves similar or better results to some of the ones described [1] and in my experience there is a lot of room for interpretation in the algorithm. In most of these cases there isn't necessarily a right way, one implementation may handle clean data better while another handles dirty/noisy data better. Some examples of spots where interpretations differ are: 1) Do you exhaustively or randomly search for categorical splits, require one hot encoding or use some other heuristic? 2) Do you use 32 or 64 bit floats? Or do you support both but stop tree growth but have a minimum impurity decrease constant to make sure tests pass on both (scikit does this, it is essentially an unexposed hyper parameter). 3) How do you handle features that have become constant in the branch of a tree being extended? Scikit insists that at least one non constant feature be examined for each potential split resulting in bigger trees, other implementations will examine only m features even if they are all constant. I made this switchable in mine, scikit behavior works better on many test data sets, the other way on some noisy overfitting prone datasets. 4) If default paramaters are used do you default to gini impurity or entropy? Do you round to the nearest number or just up (so you always get at least one) after taking the square root of the number of features to set the number of features examined for each split. All of this just means it is important to try different implementations and do hyperparameter searches etc on the data you actually care about. [1] https://github.com/ryanbressler/CloudForest https://github.com/ryanbressler/CloudForest