5 ms·

I've started studying ML (as someone who's hoping to transition at 47 to a different career from teaching and music), and some concepts I've found easy to get,

I've started studying ML (as someone who's hoping to transition at 47 to a different career from teaching and music), and some concepts I've found easy to get, others not so much (I'm old, so give me a break!). Often I need to have a concept explained in a different manner before I'll have a 'Oh, I see!' moment, and because I don't have a great grasp on the maths needed (which I'm working on, but it's s-l-o-w), as soon as equations I can't get appear, I tend to lose focus and think I can't do it. This post covers things that I've seen in the past, and seems to sum up my internal understanding of the concept, which is good for me as reading through it I had a number of 'see, I do get it' moments. The only criticism I have of it is that it seems to gloss over what differences the trees within the random forest have - as I understand it, they are all slightly different, and this gives them greater accuracy? Anyway, thanks for posting it - I'll read the other posts when I get a chance.

eden_h 8y ago
> it seems to gloss over what differences the trees within the random forest have - as I understand it, they are all slightly different, and this gives them greater accuracy? They kinda cover it in the section in 3.1 and 3.2 with Bagging and Bagging -> RandomForest, but it'd be good for them to explain Boosted Trees here as well. As far as I understand it, random forests are an aggregation of trained trees based on randomly sampled data points from the original data set. It doesn't necessarily make them more accurate on the training dataset, but it makes them more generalised and less likely to overfit (https://en.wikipedia.org/wiki/Overfitting https://en.wikipedia.org/wiki/Overfitting), because the different trees are likely to focus on different characteristics of the dataset. Boosted trees do become more accurate, as they resample, but give more priority to data points that weren't correctly classified by the earlier models.
- psandersen 8y ago
  Just to add here that the columns and not just the rows are also sampled at each node so the trees are deliberately prevented from learning too much. This helps improve diversity and reduce overfitting. This is typically around 1/3rd of the number of columns and controlled by the mtry parameter.
mabbo 8y ago
> what differences the trees within the random forest have - as I understand it, they are all slightly different, and this gives them greater accuracy? I can give you an example from my own work. We have a random forest on a 400+ attribute input (ie: 400 variables). All we want at the end is a probability from 0.0 to 1.0. Our random forest model will build around 500 trees. Each tree randomly selected a small subset of those 400+ input attributes and says "what's the best I can do using only these attributes?". Generally, it does okay. But when you average the 500 trees, the accuracy is pretty darned good. Edit later: To be clear, each new tree is generated using the random subset of variables. The point is that each tree may glean some insight about that small combination of variables.
thanatropism 8y ago
> as soon as equations I can't get That's the point where you should backtrack. Since this is a blog post it may not be kosher copacetic on all the details. But if you're studying from books and papers all the notation will either be way too well-known (set theory, cartesian products, R^d vector spaces, lp/Lp norms) or explicitly explained. If you're behind or fuzzy on the more basic stuff (what's an equivalence class? What's a Cartesian product?) I recommend the first two chapters in Munkres' book of topology. The book builds pretty far out into uncharted territory, but its recap of the basics is rigorous and superbly explained with copious illuminating prose.
khawkins 8y ago
If you understand how CART decision trees are trained, then you can see why random forests are powerful by thinking through their training process. In the first step of the decision tree training we pick the best feature split, divide the data into two groups, then train each of the two data groups independently. But what about the 2nd-best feature split? In some sense, we lose the information the other splits could provide. To see this, when testing queries, the first step is to look at that best split and pass the query to one of the two sub-trees. But those trees have only been trained with half of the training set data, and thus have weaker discriminatory power. Every split down the tree has diminishing returns in terms of how much information it provides. Now think about what the random forest does. If the feature which contains the best split is chosen for a particular tree, the split will be the same. But if it doesn't, then if the feature for the second split is present then it will be chosen. If the top 2 features aren't present then the third best split will be chosen, and so on. Thus, across our forest we have representatives of a range of feature-splits, each trained on more data and thus have more discriminatory power per split. The aggregation step at the end combines the information gleaned from these different models. Each one of them is weaker than the original CART decision tree, but has gotten more information out of the data for the features it was given. Thus, together, they are much better predictors than by themselves.

eden_h 8y ago

> it seems to gloss over what differences the trees within the random forest have - as I understand it, they are all slightly different, and this gives them greater accuracy? They kinda cover it in the section in 3.1 and 3.2 with Bagging and Bagging -> RandomForest, but it'd be good for them to explain Boosted Trees here as well. As far as I understand it, random forests are an aggregation of trained trees based on randomly sampled data points from the original data set. It doesn't necessarily make them more accurate on the training dataset, but it makes them more generalised and less likely to overfit (https://en.wikipedia.org/wiki/Overfitting https://en.wikipedia.org/wiki/Overfitting), because the different trees are likely to focus on different characteristics of the dataset. Boosted trees do become more accurate, as they resample, but give more priority to data points that weren't correctly classified by the earlier models.

psandersen 8y ago
Just to add here that the columns and not just the rows are also sampled at each node so the trees are deliberately prevented from learning too much. This helps improve diversity and reduce overfitting. This is typically around 1/3rd of the number of columns and controlled by the mtry parameter.

mabbo 8y ago

> what differences the trees within the random forest have - as I understand it, they are all slightly different, and this gives them greater accuracy? I can give you an example from my own work. We have a random forest on a 400+ attribute input (ie: 400 variables). All we want at the end is a probability from 0.0 to 1.0. Our random forest model will build around 500 trees. Each tree randomly selected a small subset of those 400+ input attributes and says "what's the best I can do using only these attributes?". Generally, it does okay. But when you average the 500 trees, the accuracy is pretty darned good. Edit later: To be clear, each new tree is generated using the random subset of variables. The point is that each tree may glean some insight about that small combination of variables.

thanatropism 8y ago

> as soon as equations I can't get That's the point where you should backtrack. Since this is a blog post it may not be kosher copacetic on all the details. But if you're studying from books and papers all the notation will either be way too well-known (set theory, cartesian products, R^d vector spaces, lp/Lp norms) or explicitly explained. If you're behind or fuzzy on the more basic stuff (what's an equivalence class? What's a Cartesian product?) I recommend the first two chapters in Munkres' book of topology. The book builds pretty far out into uncharted territory, but its recap of the basics is rigorous and superbly explained with copious illuminating prose.

khawkins 8y ago

If you understand how CART decision trees are trained, then you can see why random forests are powerful by thinking through their training process. In the first step of the decision tree training we pick the best feature split, divide the data into two groups, then train each of the two data groups independently. But what about the 2nd-best feature split? In some sense, we lose the information the other splits could provide. To see this, when testing queries, the first step is to look at that best split and pass the query to one of the two sub-trees. But those trees have only been trained with half of the training set data, and thus have weaker discriminatory power. Every split down the tree has diminishing returns in terms of how much information it provides. Now think about what the random forest does. If the feature which contains the best split is chosen for a particular tree, the split will be the same. But if it doesn't, then if the feature for the second split is present then it will be chosen. If the top 2 features aren't present then the third best split will be chosen, and so on. Thus, across our forest we have representatives of a range of feature-splits, each trained on more data and thus have more discriminatory power per split. The aggregation step at the end combines the information gleaned from these different models. Each one of them is weaker than the original CART decision tree, but has gotten more information out of the data for the features it was given. Thus, together, they are much better predictors than by themselves.