6 ms·

An Introduction to Hierarchical Modeling

112 points by andyxor 6y ago

deleted 6y ago
[deleted]
plouffy 6y ago
Very cool animation.
antipaul 6y ago
How might this be combined with a training/testing data splitting paradigm, or even using an ML model?
- causalmodels 6y ago
  Probabilistic machine learning is still machine learning. If you're referring to one of the popular tensor frameworks, pretty much all of them have probabilistic features or add-ons [1][2][3]. There are also frameworks developed specifically for this kind of work [4][5]. This [6] is a more in depth look at hierarchical / multi-level modeling. The prediction section [7] specifically goes over cross validation and inference. [1] https://www.tensorflow.org/probability/examples/Multilevel_Modeling_Primer https://www.tensorflow.org/probability/examples/Multilevel_M... [2] https://pyro.ai/examples/forecasting_iii.html https://pyro.ai/examples/forecasting_iii.html [3] http://edwardlib.org/ http://edwardlib.org/ [4] https://mc-stan.org/ https://mc-stan.org/ [5] https://docs.pymc.io/ https://docs.pymc.io/ [6] https://docs.pymc.io/notebooks/multilevel_modeling.html https://docs.pymc.io/notebooks/multilevel_modeling.html [7] https://docs.pymc.io/notebooks/multilevel_modeling.html#Prediction https://docs.pymc.io/notebooks/multilevel_modeling.html#Pred...
- RA_Fisher 6y ago
  ML are algos not models (estimators to be precise). This model could be fit by many of them. However, fitting the model directly with the linear model will be advantageous for computational reasons.
abeppu 6y ago
Though the visualization is cool here, I think this is actually not a good example. The final "hierarchical" model presented doesn't say anything about a relation between the coefficients for the different departments, so one can view this as really just having separate models, one for each department. To me, the point of a hierarchical model is that one can assume relationships between these parameters (e.g. suppose that they are from the same distribution). Then when one department has very few data points, some of the information in your beliefs about the parameters for that department come from what you learned in other departments. E.g. the base salary is likely to be similar to base salaries of other departments.
- jaygray0919 6y ago
  Agree. The example is not a Class hierarchy - it is a group with different coefficients. But the concept is useful, and the related visualization (for Class comparisons) would be useful.
antipaul 6y ago
What if you simply add department and department*salary interaction terms to a standard regression model?
- wodenokoto 6y ago
  Then for any given data point, salary for all other departments will be zero (dep1salery = dep2salary = ... = 0) so data for each department will become separate and you will in effect have i number of different models, although they all share the same bias term / intersect.
  - plouffy 6y ago
    You could argue that a hierarchical model is already multiple models. If each department has a dummy variable then they wouldn't share the same intersect. The only difference is how you interpret that intercept. The department intercept would now not be the salary of an individual with 0 experience, but the difference between the salary of an individual with 0 experience in a department compared to the all other employees with 0 experience.
    - wodenokoto 6y ago
      Well, two things here: You wouldn't actually add salary as an interaction, as it is salary you are trying to predict. You'd use an interaction between experience and department to predict salary, as you correctly point out. But usually when doing interactions, you don't apply it to the bias term, so you will get this: salary = b0 + b1*d1*exp + b2*d2*exp ... And this is essentially a model for each department, but all models share the bias term, b0. If you also interact the bias term with the dummy variable for department, you will in effect have completely independent models, salary = b0,1*d1 + b1*d1*exp + b0,2*d2 * b2*d2*exp ... But I don't think that is what is normally done when applying interactions between features. Anyway, I think we both agree, once we get our terminology aligned.
- aseerdbnarng 6y ago
  You'll run out of data pretty quickly. https://statmodeling.stat.columbia.edu/2018/03/15/need-16-times-sample-size-estimate-interaction-estimate-main-effect/ https://statmodeling.stat.columbia.edu/2018/03/15/need-16-ti...
ZephyrBlu 6y ago
I love the visualization and animation, but I think it should be specified that this is Linear Hierarchical Modelling. I was expecting something bayesian-related given that bayesian statistics lends itself to hierarchical models quite nicely.
gegtik 6y ago
I don't quite understand what's hierarchical about this example. seems like a few different mutually exclusive segments that can then be independently analysed?
- ZephyrBlu 6y ago
  It's hierarchical because of the nature of the data, not because of the model itself.
deleted 6y ago
[deleted]
alisonkisk 6y ago
Is this a narrow special case of an SVM? Why not use a SVM?

112 points by andyxor 6y ago

deleted 6y ago

[deleted]

plouffy 6y ago

Very cool animation.

antipaul 6y ago

How might this be combined with a training/testing data splitting paradigm, or even using an ML model?

causalmodels 6y ago
Probabilistic machine learning is still machine learning. If you're referring to one of the popular tensor frameworks, pretty much all of them have probabilistic features or add-ons [1][2][3]. There are also frameworks developed specifically for this kind of work [4][5]. This [6] is a more in depth look at hierarchical / multi-level modeling. The prediction section [7] specifically goes over cross validation and inference. [1] https://www.tensorflow.org/probability/examples/Multilevel_Modeling_Primer https://www.tensorflow.org/probability/examples/Multilevel_M... [2] https://pyro.ai/examples/forecasting_iii.html https://pyro.ai/examples/forecasting_iii.html [3] http://edwardlib.org/ http://edwardlib.org/ [4] https://mc-stan.org/ https://mc-stan.org/ [5] https://docs.pymc.io/ https://docs.pymc.io/ [6] https://docs.pymc.io/notebooks/multilevel_modeling.html https://docs.pymc.io/notebooks/multilevel_modeling.html [7] https://docs.pymc.io/notebooks/multilevel_modeling.html#Prediction https://docs.pymc.io/notebooks/multilevel_modeling.html#Pred...
RA_Fisher 6y ago
ML are algos not models (estimators to be precise). This model could be fit by many of them. However, fitting the model directly with the linear model will be advantageous for computational reasons.

abeppu 6y ago

Though the visualization is cool here, I think this is actually not a good example. The final "hierarchical" model presented doesn't say anything about a relation between the coefficients for the different departments, so one can view this as really just having separate models, one for each department. To me, the point of a hierarchical model is that one can assume relationships between these parameters (e.g. suppose that they are from the same distribution). Then when one department has very few data points, some of the information in your beliefs about the parameters for that department come from what you learned in other departments. E.g. the base salary is likely to be similar to base salaries of other departments.

jaygray0919 6y ago
Agree. The example is not a Class hierarchy - it is a group with different coefficients. But the concept is useful, and the related visualization (for Class comparisons) would be useful.

antipaul 6y ago

What if you simply add department and department*salary interaction terms to a standard regression model?

wodenokoto 6y ago
Then for any given data point, salary for all other departments will be zero (dep1salery = dep2salary = ... = 0) so data for each department will become separate and you will in effect have i number of different models, although they all share the same bias term / intersect.
- plouffy 6y ago
  You could argue that a hierarchical model is already multiple models. If each department has a dummy variable then they wouldn't share the same intersect. The only difference is how you interpret that intercept. The department intercept would now not be the salary of an individual with 0 experience, but the difference between the salary of an individual with 0 experience in a department compared to the all other employees with 0 experience.
  - wodenokoto 6y ago
    Well, two things here: You wouldn't actually add salary as an interaction, as it is salary you are trying to predict. You'd use an interaction between experience and department to predict salary, as you correctly point out. But usually when doing interactions, you don't apply it to the bias term, so you will get this: salary = b0 + b1*d1*exp + b2*d2*exp ... And this is essentially a model for each department, but all models share the bias term, b0. If you also interact the bias term with the dummy variable for department, you will in effect have completely independent models, salary = b0,1*d1 + b1*d1*exp + b0,2*d2 * b2*d2*exp ... But I don't think that is what is normally done when applying interactions between features. Anyway, I think we both agree, once we get our terminology aligned.
aseerdbnarng 6y ago
You'll run out of data pretty quickly. https://statmodeling.stat.columbia.edu/2018/03/15/need-16-times-sample-size-estimate-interaction-estimate-main-effect/ https://statmodeling.stat.columbia.edu/2018/03/15/need-16-ti...

ZephyrBlu 6y ago

I love the visualization and animation, but I think it should be specified that this is Linear Hierarchical Modelling. I was expecting something bayesian-related given that bayesian statistics lends itself to hierarchical models quite nicely.

gegtik 6y ago

I don't quite understand what's hierarchical about this example. seems like a few different mutually exclusive segments that can then be independently analysed?

ZephyrBlu 6y ago
It's hierarchical because of the nature of the data, not because of the model itself.

deleted 6y ago

[deleted]

alisonkisk 6y ago

Is this a narrow special case of an SVM? Why not use a SVM?