4 ms·
Good to see more research exploring the connection between trees, ensembles, and smoothing. Way back in Trevor Hastie's ESL book there's a section on how gradie
by levocardia 2y ago
Good to see more research exploring the connection between trees, ensembles, and smoothing. Way back in Trevor Hastie's ESL book there's a section on how gradient boosting using "stumps" (trees with only one split) is equivalent to an additive spline model (GAM, technically) with a step function as a spline basis and adaptive knot placement.
I've always thought there should be a deep connection between ReLU neural nets and regularized adaptive smoothers as well, since the ReLU function is itself a spline basis (a so-called truncated linear spline) and happens to span the same functional basis as B-splines of the same degree.
- almostgotcaught 2y agoOne of my biggest pet peeves is flagrant overuse of "deep". Everything is so deep around around here these days... > since the ReLU function is itself a spline basis (a so-called truncated linear spline) and happens to span the same functional basis as B-splines of the same degree. ... you literally just spelled out the entire "depth" of it.
- abhgh 2y agoI don't know if you have come across this: A Spline Theory of Deep Networks [1]. Has been on my To-Read list forever :( [1] http://proceedings.mlr.press/v80/balestriero18b/balestriero18b.pdf http://proceedings.mlr.press/v80/balestriero18b/balestriero1...