4 ms·

My thesis is related to trees, forests, and ensemble of forests. This is pretty concise but mostly for decision tree but only half of it. CART is the framewor

My thesis is related to trees, forests, and ensemble of forests. This is pretty concise but mostly for decision tree but only half of it. CART is the framework for decision tree for classification and regression. This article only addresses the Classification part which usually use Gini which is a class of split that split along parallel axises (there are oblique trees). The regression part uses more traditional statistical linear regression to calculate split point (SSTO). Very light on Random Forests though, doesn't talk about out of bagging, implication of bootstrap and ordinal data, etc.. but overall I think it's a neat introduction. > we only try a subset of the features, You bootstrap features without replacement at every split. Instead of just bootstrapping rows/observations like in bagging. The concept of weak learners ensemble together to become a strong learner is done by Dr. Ho work under her paper Random Subspace where she does it with decision tree and basically proposed Random Forest before Dr. Leo Breiman (both independently came to Random Forest). Her advisor have the theoretical paper for proof of weak learner, stocastic discrimination.

sandGorgon 8y ago
is there a better tutorial/course for a beginner into this field ? the end goal not being academia, but being able to think and write reasonable production code.
- anthony_doan 8y ago
  > is there a better tutorial/course for a beginner into this field ? I have no clue. I have a degree in Computer Science and am finishing my master in Applied Statistic. It just happen my skill set, statistical modeling, have many overlap with Machine Learning. General overview maybe https://r4ds.had.co.nz/ https://r4ds.had.co.nz/ But for straight up modeling I would start and recommend this book: https://otexts.org/fpp2/ https://otexts.org/fpp2/ (update: this book doesn't go over EDA, outlier, and such... r4ds does EDA. For outlier and imputation some statistic book can cover that.) It's for time series modeling (statistical slanted) but I think it goes over modeling aspect really well and it very intro (coursera have a step above that book for a little more depth in time series). To be fair... statistical modeling for univariate time series statistical model is number one-ish see m4 competition or the uber blog on time series. > the end goal not being academia, but being able to think and write reasonable production code. For the thinking part which affect writing code, from my experiences (I can be wrong), data science/ML approach to modeling is different than statistic. The vast majority of the time Data Science/ML are given the data which is why I believe AI algorithm can be bias (see Gyfcat and Asian computer vision classification problem). Where as statistic you usually figure out your hypothesis or what you are trying to answer and then you create a design experiment or strategy on how to collect the data without being bias and hopefully controlling factors. But statistic also do given data but the vast majority of models out there is slanted toward explanatory vs forecasting/prediction. DS/ML seems to care more about forecasting/prediction. I also think how each discipline approach modeling affect the way a person think within that field too. I have not figure out the unified thinking of how ML/DS discipline approach model but I am confidence that it's not the same as statistic. But speaking from statistic vs applied math, I can give an example. For time series data, statistician model on the assumption that all we're given is the data and we'll try to extract every bit of information out of the data and explain away the variance with models (each predictor can remove away the noise by explaining it whatever left is error/chances). It's more data focus. For applied math, they figure out how the data is generated eg their model assume this is how the data is generated so they got these stochastic processes and is uses more toward probability than statistic. So thinking would affect code... So how statistician code stuff is probably different than ML/DS. I've seen people calling imputation witchcraft and throw temporal data in random forest >___>. > write reasonable production code. I do R for modeling. If you're not creating any new fancy algorithm, you can just use packages. You train them on the data set, and just ship the trained model. You can wrap it as a REST service via https://www.rplumber.io/ https://www.rplumber.io/. I like to think Python have something similar? Do note I know very little about deep learning or how to ship that. I've chosen to specialize in statistical modeling and R for modeling.
- savagedata 8y ago
  I wrote this regression tree tutorial a few years back that might be a good complement to the tutorial above since it covers regression instead of classification and goes on to talk about bagging vs random forest, out-of-bag samples, and tuning parameters: https://github.com/savagedata/regression-tree-tutorial https://github.com/savagedata/regression-tree-tutorial I wrote it at the start of my career and haven't shared it beyond my study group, so I'm happy to hear feedback.
  - anthony_doan 8y ago
    It's a really good tutorial. I like how you talk about Conditional Inference. My thesis is suppose to overcome the brute force of exhaustive search for best splits that Random Forest does (I use Dr. Loh's GUIDE trees) using statistical methods. > Many implementations of random forest default to 1/3 of your predictor variables. This is interesting. I hear it was sqroot(total number of predictors). > Ensemble methods combine many individual trees to create one better, more stable model. I think stable can be more clarify to having good training accuracy and low generalize error (unseen data error rate) compare to individual tree. This is what Dr. Ho talk about with forest. But other than that I think it's an awesome tutorial. I've seen what other tree and forest do for better generalization with unseen data is pruning is using CV and choosing 0.5 to 1.0 std error as a cut off point. That may be a thing to talk about if you are interested in that.
    - savagedata 8y ago
      Thank you for the useful feedback! I'll have to look up GUIDE trees. > This is interesting. I hear it was sqroot(total number of predictors). I was probably looking at the randomForest R package documentation [1], which says: > mtry Number of variables randomly sampled as candidates at each split. Note that the default values are different for classification (sqrt(p) where p is number of variables in x) and regression (p/3) I checked the H2O implementation of random forest [2] and they use the same defaults. I'll add a note about the one third default being specific to regression since that seems like an important distinction. [1] https://www.rdocumentation.org/packages/randomForest/versions/4.6-14/topics/randomForest https://www.rdocumentation.org/packages/randomForest/version... [2] http://docs.h2o.ai/h2o/latest-stable/h2o-docs/data-science/drf.html http://docs.h2o.ai/h2o/latest-stable/h2o-docs/data-science/d...
- claytonjy 8y ago
  For decision trees, I really like http://www.r2d3.us/visual-intro-to-machine-learning-part-1/ http://www.r2d3.us/visual-intro-to-machine-learning-part-1/ and https://explained.ai/decision-tree-viz/index.html https://explained.ai/decision-tree-viz/index.html. For Random Forests, I like this one: https://www.gormanalysis.com/blog/random-forest-from-top-to-bottom/ https://www.gormanalysis.com/blog/random-forest-from-top-to-..., which also has a link to a decision-tree post. That blog also has the best GBM explainer I've seen yet (Gradient Boosted Machines are the _other_ tree-ensembling method in common use, where the trees are _stacked_ instead of _bagged_) Your goal should not be to know enough to write an RF implementation, but rather to have some intuition behind how it works, so you can better choose when to use it or not. The likelihood of it ever making sense for you to write and RF algorithm for production use is extremely unlikely; use the great code that already exists for most languages.
vzhou842 8y ago
Hey! Author here. Appreciate the feedback. You sound like you know what you're talking about. My site is open source, and if you want you can make a few edits to this article and submit a pull request! https://github.com/vzhou842/victorzhou.com/blob/master/content/posts/2019-04-10-intro-to-random-forests.md https://github.com/vzhou842/victorzhou.com/blob/master/conte...
- GaryNumanVevo 8y ago
  honest question: who does this benefit but yourself? I don't see a commit log or blame on your site for who wrote what. Aren't you effectively claiming authorship for other's work?
  - vzhou842 8y ago
    My blog is still pretty new, so I haven't had to think about this situation yet - I'm the only one who's committed so far. If someone were to contribute I'd be more than happy to figure out a fair way to attribute them, though!
  - chrisshroba 8y ago
    I think this assumes writing an article only benefits the author, when in fact I think it benefits readers far more (if the blog isn't monetized, the author only really gets satisfaction out of it). If I had expertise in some area I would be happy to contribute to someone else's blog, provided I were properly credited (which the author stated he would do).

sandGorgon 8y ago

is there a better tutorial/course for a beginner into this field ? the end goal not being academia, but being able to think and write reasonable production code.

anthony_doan 8y ago
> is there a better tutorial/course for a beginner into this field ? I have no clue. I have a degree in Computer Science and am finishing my master in Applied Statistic. It just happen my skill set, statistical modeling, have many overlap with Machine Learning. General overview maybe https://r4ds.had.co.nz/ https://r4ds.had.co.nz/ But for straight up modeling I would start and recommend this book: https://otexts.org/fpp2/ https://otexts.org/fpp2/ (update: this book doesn't go over EDA, outlier, and such... r4ds does EDA. For outlier and imputation some statistic book can cover that.) It's for time series modeling (statistical slanted) but I think it goes over modeling aspect really well and it very intro (coursera have a step above that book for a little more depth in time series). To be fair... statistical modeling for univariate time series statistical model is number one-ish see m4 competition or the uber blog on time series. > the end goal not being academia, but being able to think and write reasonable production code. For the thinking part which affect writing code, from my experiences (I can be wrong), data science/ML approach to modeling is different than statistic. The vast majority of the time Data Science/ML are given the data which is why I believe AI algorithm can be bias (see Gyfcat and Asian computer vision classification problem). Where as statistic you usually figure out your hypothesis or what you are trying to answer and then you create a design experiment or strategy on how to collect the data without being bias and hopefully controlling factors. But statistic also do given data but the vast majority of models out there is slanted toward explanatory vs forecasting/prediction. DS/ML seems to care more about forecasting/prediction. I also think how each discipline approach modeling affect the way a person think within that field too. I have not figure out the unified thinking of how ML/DS discipline approach model but I am confidence that it's not the same as statistic. But speaking from statistic vs applied math, I can give an example. For time series data, statistician model on the assumption that all we're given is the data and we'll try to extract every bit of information out of the data and explain away the variance with models (each predictor can remove away the noise by explaining it whatever left is error/chances). It's more data focus. For applied math, they figure out how the data is generated eg their model assume this is how the data is generated so they got these stochastic processes and is uses more toward probability than statistic. So thinking would affect code... So how statistician code stuff is probably different than ML/DS. I've seen people calling imputation witchcraft and throw temporal data in random forest >___>. > write reasonable production code. I do R for modeling. If you're not creating any new fancy algorithm, you can just use packages. You train them on the data set, and just ship the trained model. You can wrap it as a REST service via https://www.rplumber.io/ https://www.rplumber.io/. I like to think Python have something similar? Do note I know very little about deep learning or how to ship that. I've chosen to specialize in statistical modeling and R for modeling.
savagedata 8y ago
I wrote this regression tree tutorial a few years back that might be a good complement to the tutorial above since it covers regression instead of classification and goes on to talk about bagging vs random forest, out-of-bag samples, and tuning parameters: https://github.com/savagedata/regression-tree-tutorial https://github.com/savagedata/regression-tree-tutorial I wrote it at the start of my career and haven't shared it beyond my study group, so I'm happy to hear feedback.
- anthony_doan 8y ago
  It's a really good tutorial. I like how you talk about Conditional Inference. My thesis is suppose to overcome the brute force of exhaustive search for best splits that Random Forest does (I use Dr. Loh's GUIDE trees) using statistical methods. > Many implementations of random forest default to 1/3 of your predictor variables. This is interesting. I hear it was sqroot(total number of predictors). > Ensemble methods combine many individual trees to create one better, more stable model. I think stable can be more clarify to having good training accuracy and low generalize error (unseen data error rate) compare to individual tree. This is what Dr. Ho talk about with forest. But other than that I think it's an awesome tutorial. I've seen what other tree and forest do for better generalization with unseen data is pruning is using CV and choosing 0.5 to 1.0 std error as a cut off point. That may be a thing to talk about if you are interested in that.
  - savagedata 8y ago
    Thank you for the useful feedback! I'll have to look up GUIDE trees. > This is interesting. I hear it was sqroot(total number of predictors). I was probably looking at the randomForest R package documentation [1], which says: > mtry Number of variables randomly sampled as candidates at each split. Note that the default values are different for classification (sqrt(p) where p is number of variables in x) and regression (p/3) I checked the H2O implementation of random forest [2] and they use the same defaults. I'll add a note about the one third default being specific to regression since that seems like an important distinction. [1] https://www.rdocumentation.org/packages/randomForest/versions/4.6-14/topics/randomForest https://www.rdocumentation.org/packages/randomForest/version... [2] http://docs.h2o.ai/h2o/latest-stable/h2o-docs/data-science/drf.html http://docs.h2o.ai/h2o/latest-stable/h2o-docs/data-science/d...
claytonjy 8y ago
For decision trees, I really like http://www.r2d3.us/visual-intro-to-machine-learning-part-1/ http://www.r2d3.us/visual-intro-to-machine-learning-part-1/ and https://explained.ai/decision-tree-viz/index.html https://explained.ai/decision-tree-viz/index.html. For Random Forests, I like this one: https://www.gormanalysis.com/blog/random-forest-from-top-to-bottom/ https://www.gormanalysis.com/blog/random-forest-from-top-to-..., which also has a link to a decision-tree post. That blog also has the best GBM explainer I've seen yet (Gradient Boosted Machines are the _other_ tree-ensembling method in common use, where the trees are _stacked_ instead of _bagged_) Your goal should not be to know enough to write an RF implementation, but rather to have some intuition behind how it works, so you can better choose when to use it or not. The likelihood of it ever making sense for you to write and RF algorithm for production use is extremely unlikely; use the great code that already exists for most languages.

vzhou842 8y ago

Hey! Author here. Appreciate the feedback. You sound like you know what you're talking about. My site is open source, and if you want you can make a few edits to this article and submit a pull request! https://github.com/vzhou842/victorzhou.com/blob/master/content/posts/2019-04-10-intro-to-random-forests.md https://github.com/vzhou842/victorzhou.com/blob/master/conte...

GaryNumanVevo 8y ago
honest question: who does this benefit but yourself? I don't see a commit log or blame on your site for who wrote what. Aren't you effectively claiming authorship for other's work?
- vzhou842 8y ago
  My blog is still pretty new, so I haven't had to think about this situation yet - I'm the only one who's committed so far. If someone were to contribute I'd be more than happy to figure out a fair way to attribute them, though!
- chrisshroba 8y ago
  I think this assumes writing an article only benefits the author, when in fact I think it benefits readers far more (if the blog isn't monetized, the author only really gets satisfaction out of it). If I had expertise in some area I would be happy to contribute to someone else's blog, provided I were properly credited (which the author stated he would do).