6 ms·

Sure. Random forests are a very powerful machine learning method that has proven to give excellent performance with very little tuning. Neural networks with m

by Fede_V 11y ago

Sure. Random forests are a very powerful machine learning method that has proven to give excellent performance with very little tuning. Neural networks with many layers (aka deep learning) is the new frontier of machine learning - they are very powerful but require extensive tuning (the architecture of the network) and lots of examples before they become practical. In this paper, the authors combine the two methods - they basically stack a random forest on top of a neural network, and obtain some cutting edge results. One of the main theoretical results in the paper is that they propose a differentiable (which means you can use gradient-based optimization techniques) technique to optimize the parameters of the trees (which are the building blocks of the random forest).

JD557 11y ago
Also, it might be worth adding that random forests are based on decision trees, which can be interpreted by humans. While it's hard to interpret a random forest, it's still much more interpretable than a neural network. Maybe this model is also easier to interpret by humans.
- Fede_V 11y ago
  Random forests are very difficult to interpret especially once you start having lots of trees.
- rm999 11y ago
  >While it's hard to interpret a random forest, it's still much more interpretable than a neural network Depends on the domain. The first layer or two of a neural network can be fairly interpretable on image or spectrogram (sound) data, for example.
UmDieWelt 11y ago
New frontier? They've been popular for over 20 years.
- robotresearcher 11y ago
  That's AI. Every generation invents everything again. Probably works this way in other fields too. (Yes this is hyperbole, but any old timers out there will know the feeling)
  - rdtsc 11y ago
    > That's AI. Every generation invents everything again. Also "Past decades was AI winter. But no worries, now we'll start seeing human intelligence in a about next year" probably heard that since late 90's.
- Fede_V 11y ago
  They really came to the forefront with alex.net - GPUs plus lots of labeled examples made them suddenly very practical.
  - simonhughes22 11y ago
    Yes and No. Layer to layer training of deep belief nets were doing very well before alex.net came along, although alex.net was seminal for images. People often miss the big 'advances' in RNN's, although those go back to the 90's and earlier. A lot of this (but not all) is also due to more data, better and faster hardware (including GPU's). Although there have been some important algorithmic enhancements too.
- mtrimpe 11y ago
  Regular neural networks were popular until they stopped delivering best-in-class results for a lot of problems. Recent hardware made it possible to train more layers and they're now getting cutting-edge results in many areas again so they're now getting more attention again.
- eva1984 11y ago
  Old time neural networks not even came close to current models in terms of performance. I clearly remember in my machine learning class, one professor mentions neural networks and says it is slow and impossible to tame when the layers goes up thus loses its popularity. That is just 3-4 years ago.
obastani 11y ago
I think there are a few additional important details. It's not so much that they stack a random forest on top of a neural network, as that they are trying to train neural networks that have tree-like properties. Deep neural networks can be expensive to evaluate because every layer is needed to compute the resulting label. On the other hand, in a decision tree you only need to evaluate one of a nodes children (i.e., evaluation is linear in the depth of the tree). They combine these ideas by allowing some nodes in the network to branch (so for a given image, you choose one of two neural networks to evaluate). These nodes don't have to occur at the top level, but can occur in the middle of the network as well. The difficulty is how you train these nodes; that's where the differential technique for optimizing trees come in. Because you have a unified training algorithm, you can mix and match tree-like nodes and neural network like nodes and train the whole thing. All in all, the hope is that getting a label for a given node is much faster, so you can run these on smaller devices such as phones without sacrificing accuracy.
YeGoblynQueenne 11y ago
>> One of the main theoretical results in the paper is that they propose a differentiable .. technique to optimize the parameters of the trees. Yay! Now Decision Trees too can get stuck in local optima! :D (totally tongue in cheek)
- argonaut 11y ago
  Even though this is tongue-in-cheek, it makes no sense. Decision trees are traditionally trained in a greedy, non-globally-optimal fashion.
  - YeGoblynQueenne 11y ago
    What I mean is that decision trees are not seen as optimising a function so it doesn't make much sense to think of them as finding optima, or getting stuck in sub-optima (if I may). That's the traditional view. I guess it's always possible to see decision trees as optimising a binary function, but that's probably to be filed under "original research". Or maybe not- I'm not sure. In any case, I've never heard of anyone worried that a tree learner would get stuck anywhere. The concern is usually with overfitting.

by Fede_V 11y ago

JD557 11y ago

Also, it might be worth adding that random forests are based on decision trees, which can be interpreted by humans. While it's hard to interpret a random forest, it's still much more interpretable than a neural network. Maybe this model is also easier to interpret by humans.

Fede_V 11y ago
Random forests are very difficult to interpret especially once you start having lots of trees.
rm999 11y ago
>While it's hard to interpret a random forest, it's still much more interpretable than a neural network Depends on the domain. The first layer or two of a neural network can be fairly interpretable on image or spectrogram (sound) data, for example.

UmDieWelt 11y ago

New frontier? They've been popular for over 20 years.

robotresearcher 11y ago
That's AI. Every generation invents everything again. Probably works this way in other fields too. (Yes this is hyperbole, but any old timers out there will know the feeling)
- rdtsc 11y ago
  > That's AI. Every generation invents everything again. Also "Past decades was AI winter. But no worries, now we'll start seeing human intelligence in a about next year" probably heard that since late 90's.
Fede_V 11y ago
They really came to the forefront with alex.net - GPUs plus lots of labeled examples made them suddenly very practical.
- simonhughes22 11y ago
  Yes and No. Layer to layer training of deep belief nets were doing very well before alex.net came along, although alex.net was seminal for images. People often miss the big 'advances' in RNN's, although those go back to the 90's and earlier. A lot of this (but not all) is also due to more data, better and faster hardware (including GPU's). Although there have been some important algorithmic enhancements too.
mtrimpe 11y ago
Regular neural networks were popular until they stopped delivering best-in-class results for a lot of problems. Recent hardware made it possible to train more layers and they're now getting cutting-edge results in many areas again so they're now getting more attention again.
eva1984 11y ago
Old time neural networks not even came close to current models in terms of performance. I clearly remember in my machine learning class, one professor mentions neural networks and says it is slow and impossible to tame when the layers goes up thus loses its popularity. That is just 3-4 years ago.

obastani 11y ago

I think there are a few additional important details. It's not so much that they stack a random forest on top of a neural network, as that they are trying to train neural networks that have tree-like properties. Deep neural networks can be expensive to evaluate because every layer is needed to compute the resulting label. On the other hand, in a decision tree you only need to evaluate one of a nodes children (i.e., evaluation is linear in the depth of the tree). They combine these ideas by allowing some nodes in the network to branch (so for a given image, you choose one of two neural networks to evaluate). These nodes don't have to occur at the top level, but can occur in the middle of the network as well. The difficulty is how you train these nodes; that's where the differential technique for optimizing trees come in. Because you have a unified training algorithm, you can mix and match tree-like nodes and neural network like nodes and train the whole thing. All in all, the hope is that getting a label for a given node is much faster, so you can run these on smaller devices such as phones without sacrificing accuracy.

YeGoblynQueenne 11y ago

>> One of the main theoretical results in the paper is that they propose a differentiable .. technique to optimize the parameters of the trees. Yay! Now Decision Trees too can get stuck in local optima! :D (totally tongue in cheek)

argonaut 11y ago
Even though this is tongue-in-cheek, it makes no sense. Decision trees are traditionally trained in a greedy, non-globally-optimal fashion.
- YeGoblynQueenne 11y ago
  What I mean is that decision trees are not seen as optimising a function so it doesn't make much sense to think of them as finding optima, or getting stuck in sub-optima (if I may). That's the traditional view. I guess it's always possible to see decision trees as optimising a binary function, but that's probably to be filed under "original research". Or maybe not- I'm not sure. In any case, I've never heard of anyone worried that a tree learner would get stuck anywhere. The concern is usually with overfitting.