4 ms·

This is a nice article, but there's some caveats with this approach that deserve a note: 1) Random forests are by nature supervised learners. They will compl

This is a nice article, but there's some caveats with this approach that deserve a note: 1) Random forests are by nature supervised learners. They will completely ignore information that they can't use to predict their target output. This will have a big impact on any similarity scores derived from these types of models. At best, you'll see which observations the model considers to be similar in the training set (for the purposes of predicting the given output). This is still useful, but not as generalizable. 2) Even when considering the above, the distribution of splits is commonly skewed or biased in some way. The depth of the trees in the forest will vary. The distribution of training data at leaf nodes will vary. Categorical features may cause more splits than numeric features, etc. Some manner of re-weighting (a la information gain ratio) needs to be used to balance feature importance on the overall model behavior. I think this is a useful technique, and I would like to see more tree-based metrics used. The metrics just need some adjustments to account for all the distributional kinks in the trees, and the users need to know what kind of caveats are applicable for their data.

tvladeck 9y ago
> 1) Random forests are by nature supervised learners You can run unsupervised random forests. The way this procedure works is to first make a "copy" of the original dataset (the reason for the quotes will be evident in a second), and then train the forest to distinguish the original data from the copy with a binary target vector. The "copy" is made by sampling column-wise from the original dataset, so that, in the copy, each variable has the same univariate distribution as the original, but any interdependencies are destroyed. The forest then learns to distinguish the original from the copy, and the only information that it has at its disposal are interdependencies in the original dataset.
RMarcus 9y ago
Both good considerations! This approach can be used to build a kernel that measures the similarity of two items with respect to the labels learned by the forest. I don't doubt that the specific algorithm used to grow the forest. Luckily, any kernel made from any forest will have a positive semi-definite Gram matrix. But that doesn't mean it'll be a good one. I would summarize this approach as "learning a kernel that compares items w.r.t. the labels learned by the tree."

tvladeck 9y ago

> 1) Random forests are by nature supervised learners You can run unsupervised random forests. The way this procedure works is to first make a "copy" of the original dataset (the reason for the quotes will be evident in a second), and then train the forest to distinguish the original data from the copy with a binary target vector. The "copy" is made by sampling column-wise from the original dataset, so that, in the copy, each variable has the same univariate distribution as the original, but any interdependencies are destroyed. The forest then learns to distinguish the original from the copy, and the only information that it has at its disposal are interdependencies in the original dataset.

RMarcus 9y ago

Both good considerations! This approach can be used to build a kernel that measures the similarity of two items with respect to the labels learned by the forest. I don't doubt that the specific algorithm used to grow the forest. Luckily, any kernel made from any forest will have a positive semi-definite Gram matrix. But that doesn't mean it'll be a good one. I would summarize this approach as "learning a kernel that compares items w.r.t. the labels learned by the tree."