3 ms·

Let me add another often-overlooked tweak that you can use with a random forest: applying random rotations before one learns the trees. This is extremely power

by srean 9y ago

Let me add another often-overlooked tweak that you can use with a random forest: applying random rotations before one learns the trees. This is extremely powerful. To give you some intuition on why this is so, recall that simple trees do axis parallel splits of the space. If the trees have to approximate an inclined line, they have to do a high resolution stair stepping to approximate it, -- like so -- __ | ------ | --------- |_______ this ends up requiring a lot of trees or a very deep one. Now if you equip the algorithm with a few inclined splits, which is exactly what a random rotations would do, you gain a lot of power.

haeffin 9y ago
How is that different from allowing non-axis-aligned splits?
- tlarkworthy 9y ago
  its all much of a muchness. You can use a linear function at the nodes to project the space before applying a >. However, all these tweaks tend to just consume CPU for marginal benefit. You can still appoximate a non axis aligned decision boundary with a series of stepwise aligned splits, albeit less compactly. But if your aim is to just generate a ton of non linear distance functions ... a la random forest, focusing on tree expressivity is the wrong thing. Why not spend your CPU cycles on more trees instead instead, which leads to smoother decision boundaries in general when mixing their responses. There is a ton of tweaks you can do, but unless you are expressly researching ML algorithms, the pragmatist should not care and just try a few options from whatever comes in scipy.
- srean 9y ago
  That is a very interesting question and one can go deep into the woods with that, not everything is known about this question. There are two distinct effects at play here, computational cost and learning efficiency. A brutally shortened response would be that searching for non-axis aligned splits in the nodes can be very expensive CPU wise. This addresses the first effect at play. Applying a random rotation on the other hand is quite cheap. The second effect at play is that ensembling loses its power when you optimize over oblique splits because of lost diversity. It comes down to a trade off.

by srean 9y ago

haeffin 9y ago

How is that different from allowing non-axis-aligned splits?

tlarkworthy 9y ago
its all much of a muchness. You can use a linear function at the nodes to project the space before applying a >. However, all these tweaks tend to just consume CPU for marginal benefit. You can still appoximate a non axis aligned decision boundary with a series of stepwise aligned splits, albeit less compactly. But if your aim is to just generate a ton of non linear distance functions ... a la random forest, focusing on tree expressivity is the wrong thing. Why not spend your CPU cycles on more trees instead instead, which leads to smoother decision boundaries in general when mixing their responses. There is a ton of tweaks you can do, but unless you are expressly researching ML algorithms, the pragmatist should not care and just try a few options from whatever comes in scipy.
srean 9y ago
That is a very interesting question and one can go deep into the woods with that, not everything is known about this question. There are two distinct effects at play here, computational cost and learning efficiency. A brutally shortened response would be that searching for non-axis aligned splits in the nodes can be very expensive CPU wise. This addresses the first effect at play. Applying a random rotation on the other hand is quite cheap. The second effect at play is that ensembling loses its power when you optimize over oblique splits because of lost diversity. It comes down to a trade off.