3 ms·

If you have a collection of algorithms which are each trying to interpret the same effect or data, but in different ways, combining them could make things worse

If you have a collection of algorithms which are each trying to interpret the same effect or data, but in different ways, combining them could make things worse. Can it? I wouldn't think so if you are combining them properly... at least, I would expect the overall average to be better even if its worse in some predictions. Obviously if you are doing something like evenly weighing the predictions you could end up with something worse but thats not really a smart combination.

JacobAldridge 17y ago
Don't get me wrong - it won't definitely make things worse, but it can. I felt the article over-simplistically inferred that adding other teams and algorithms would improve results almost every time, which isn't the case - it took a lot of smart combinations to win.
- roundsquare 17y ago
  Maybe we agree... but I guess my point is that if you are doing things correctly/intelligently, it should never make things worse (in aggregate)... In other words, if combining algorithms is making things worse than you are probably combining them wrong. I'm sorta pulling this out of my ass (in the sense that I haven't done this before), but thats what I would thing...
trapper 17y ago
There are some interesting papers on the equivalence of different learning algorithms (e.g. decision trees, logistic regression, svm, FF nn's). When it comes down to it, a lot of the "different" methods actually model the same thing in roughly the same way, meaning the only difference in the algorithm's boiled down to ability to interpret the algorithms internal rules (e.g. nn is black box, decision tree isn't) and training time. No references handy unfortunately.
- roundsquare 17y ago
  Really? Makes sense I guess, but it would be great if you do run across any papers. So I guess the real art comes down to 1) Picking your learning algo 2) Some details with each method. For nn's - do you use sigmoid squashing, etc... For knn - what is your distance metric.
  - trapper 17y ago
    Yes, I was surprised as well but if you think about it, it does makes sense. I'll dig around and see if I can find the paper.

JacobAldridge 17y ago

Don't get me wrong - it won't definitely make things worse, but it can. I felt the article over-simplistically inferred that adding other teams and algorithms would improve results almost every time, which isn't the case - it took a lot of smart combinations to win.

roundsquare 17y ago
Maybe we agree... but I guess my point is that if you are doing things correctly/intelligently, it should never make things worse (in aggregate)... In other words, if combining algorithms is making things worse than you are probably combining them wrong. I'm sorta pulling this out of my ass (in the sense that I haven't done this before), but thats what I would thing...

trapper 17y ago

There are some interesting papers on the equivalence of different learning algorithms (e.g. decision trees, logistic regression, svm, FF nn's). When it comes down to it, a lot of the "different" methods actually model the same thing in roughly the same way, meaning the only difference in the algorithm's boiled down to ability to interpret the algorithms internal rules (e.g. nn is black box, decision tree isn't) and training time. No references handy unfortunately.

roundsquare 17y ago
Really? Makes sense I guess, but it would be great if you do run across any papers. So I guess the real art comes down to 1) Picking your learning algo 2) Some details with each method. For nn's - do you use sigmoid squashing, etc... For knn - what is your distance metric.
- trapper 17y ago
  Yes, I was surprised as well but if you think about it, it does makes sense. I'll dig around and see if I can find the paper.