27 ms·

Hm, I agree with what you're saying, but your example is pretty whack: (1) in a logistic regression the coefficients are on a log scale ergo the ratio between

Hm, I agree with what you're saying, but your example is pretty whack: (1) in a logistic regression the coefficients are on a log scale ergo the ratio between exp(2) and exp(1) is actually x2.7, not x2; the bigger issue is that you have to compare the strength of the association to how easy it is to move that lever, e.g. men might like our marketing message more than women, but it's not like we're going to get people to change gender. (2) moderately correlated predictors do not bias or otherwise complicate the interpretation of regression parameters, it's only unmeasured confounders that do, that is, correlations between variables where one of the variables is not included in the model.

claytonjy 8y ago
In (2) it's actually worse than that; any relevant predictor left out of the model affects the betas of all other predictors, even if it's uncorrelated to all other predictors. This is a big thing most people miss when trying to interpret logistic regression the way they do linear regression. Logistic regression estimates are _conditional_ on the model spec in a way linear regression estimates are not.
- gbrown 8y ago
  I'm not sure I follow your second point. In both multiple linear regression and multiple logistic regression, the coefficients are interpreted conditionally - a one unit increase in X^{(i)} is associated with a B^{(i)} unit change in f(E(Y)), conditional on the values of the other covariates. Using a logit link function does remove the result from the raw scale of Y (thus, f), but in my mind the "model" is more than just the data distribution - it's whatever form your linear predictor takes. This is the problem the poster originally pointed to - interpreting a variable conditional on the fixed values of all others doesn't always make sense when there's strong correlation among predictors.
  - zwaps 8y ago
    I still don't get your issue at all. There are tons of different models and implications, and it depends entirely on your question how you interpret the model. Often you will go after average partial effects in your sample. Or you have some correlated variables in mind such that you can just plot the partial effects. Sometimes you have different models, where you will plot a distribution of partial effects around a mean based on some prior assumption... I mean, it really depends. If you impose a more complex relationship, then of course you can not put everything in a single number. But no matter what you are interested in, such a model will give you the possibility to exactly determine the measure. And what you are saying is also not correct. I may very often be interested in fixed values. Doing this regression, and not a quantile regression, for example, means I am somehow interested in a conditional expectation. That means I probably care about some sort of average impact, perhaps for some fixed subgroups. But those averages are fixed values... I think the point is that in these models, we know exactly what we go after, how to get there, and what it means. We know exactly when our inference may fail. If we both care about average effects, and I can convince you of my identification assumptions, then there is really no mystery left as to what my estimates mean. In with Deep Learning, this is still more difficult.
agentofoblivion 8y ago
Your point on (2) and bias may be correct, but I disagree about interpretation. If there's a one-way causal link between A and B (e.g. B causes A, but A does not cause B), then should you interpret the presence of B as having an impact that sums the coefficients, or ignore the correlation and pretend the impact came from A? Or what if A always caused B, but the impact was slightly less than if B occurred without A? In that case, the sign on A might be negative, but its presence would actually tend to increase the probability of class=1, it's just that the positive impact has already been counted by variable B. Maybe you try to avoid this situation by adding in an explicit interaction term of A*B, but then how do you interpret the impact of A since you now have more than one coefficient? If you feel confident making assertive statements about what has been learned by looking at an equation fit on multi-correlated data, then your mathematical intuition is much stronger than mine!
- stdbrouw 8y ago
  The interpretation of interaction terms (and first-order terms in the presence of interactions) is something that is taught to bachelor students in the social sciences all the time. I'm not going to say it's entirely straightforward and I'm sure lots of people get it wrong, but it's not rocket science. If people jump in and use a technique without having a clue about how it's supposed to be used (and I'm sure there are plenty -- I agree with you on that), then that's ultimately their own stupid fault.

claytonjy 8y ago

In (2) it's actually worse than that; any relevant predictor left out of the model affects the betas of all other predictors, even if it's uncorrelated to all other predictors. This is a big thing most people miss when trying to interpret logistic regression the way they do linear regression. Logistic regression estimates are _conditional_ on the model spec in a way linear regression estimates are not.

gbrown 8y ago
I'm not sure I follow your second point. In both multiple linear regression and multiple logistic regression, the coefficients are interpreted conditionally - a one unit increase in X^{(i)} is associated with a B^{(i)} unit change in f(E(Y)), conditional on the values of the other covariates. Using a logit link function does remove the result from the raw scale of Y (thus, f), but in my mind the "model" is more than just the data distribution - it's whatever form your linear predictor takes. This is the problem the poster originally pointed to - interpreting a variable conditional on the fixed values of all others doesn't always make sense when there's strong correlation among predictors.
- zwaps 8y ago
  I still don't get your issue at all. There are tons of different models and implications, and it depends entirely on your question how you interpret the model. Often you will go after average partial effects in your sample. Or you have some correlated variables in mind such that you can just plot the partial effects. Sometimes you have different models, where you will plot a distribution of partial effects around a mean based on some prior assumption... I mean, it really depends. If you impose a more complex relationship, then of course you can not put everything in a single number. But no matter what you are interested in, such a model will give you the possibility to exactly determine the measure. And what you are saying is also not correct. I may very often be interested in fixed values. Doing this regression, and not a quantile regression, for example, means I am somehow interested in a conditional expectation. That means I probably care about some sort of average impact, perhaps for some fixed subgroups. But those averages are fixed values... I think the point is that in these models, we know exactly what we go after, how to get there, and what it means. We know exactly when our inference may fail. If we both care about average effects, and I can convince you of my identification assumptions, then there is really no mystery left as to what my estimates mean. In with Deep Learning, this is still more difficult.

agentofoblivion 8y ago

Your point on (2) and bias may be correct, but I disagree about interpretation. If there's a one-way causal link between A and B (e.g. B causes A, but A does not cause B), then should you interpret the presence of B as having an impact that sums the coefficients, or ignore the correlation and pretend the impact came from A? Or what if A always caused B, but the impact was slightly less than if B occurred without A? In that case, the sign on A might be negative, but its presence would actually tend to increase the probability of class=1, it's just that the positive impact has already been counted by variable B. Maybe you try to avoid this situation by adding in an explicit interaction term of A*B, but then how do you interpret the impact of A since you now have more than one coefficient? If you feel confident making assertive statements about what has been learned by looking at an equation fit on multi-correlated data, then your mathematical intuition is much stronger than mine!

stdbrouw 8y ago
The interpretation of interaction terms (and first-order terms in the presence of interactions) is something that is taught to bachelor students in the social sciences all the time. I'm not going to say it's entirely straightforward and I'm sure lots of people get it wrong, but it's not rocket science. If people jump in and use a technique without having a clue about how it's supposed to be used (and I'm sure there are plenty -- I agree with you on that), then that's ultimately their own stupid fault.