4 ms·

> Yes, a linear regression is explainable. I don't agree that this is, in general, the case. Dense linear models trained on a lot of parameters aren't exactly

by qsort 6y ago

> Yes, a linear regression is explainable. I don't agree that this is, in general, the case. Dense linear models trained on a lot of parameters aren't exactly simple to trace, and even the weights aren't as intuitive as they might seem.What about regularization? What about pre-processing? What about feature selection? There are a lot of examples of linear regressions where, for example, you can reverse the sign of the correlation by picking some features rather than others. Assigning meanings to features of your model is something that requires extreme care and deep understanding of both the model and your dataset.

free_rms 6y ago
Ok, fair enough, the more sophistication you add, the less explainable it gets. That being said, you can look at magnitude and direction of various coefficients and conclude certain obvious things. Even the case where reversing the correlation by picking some features instead of others says something -- those features point different ways but one has a higher magnitude so it 'wins'. They're ultimately pretty simple statistical observations about the training set. A deep net, in comparison, is a totally unexplainable black box between the cross-linking and the sigmoid functions.
- qsort 6y ago
  I agree that there is an intuitive concept of "explainability", which is why I said that a formal model for it would be interesting to see. I also agree that "the average explainability of a deep net is lower than the average explainability of a linear model" (whatever that sentence means), my point is that it's extremely hard to pinpoint what explainability means, because the usual characterization of "you can fit the model in your head" is extremely subjective and brittle. I'm all for more research in this field. Heck, I'm all for more research in any field, but because our definition of "explainability" is so primitive and unrefined, arguing that we should build ethical or regulatory frameworks on top of it seems premature.
- glial 6y ago
  > you can look at magnitude and direction of various coefficients and conclude certain obvious things. Sometimes, but if you have correlated predictors, the 'explainability' can be distributed across coefficients of those predictors in unpredictable ways. Training a linear model twice with two different subsets of training data can give you very different coefficient weights.

by qsort 6y ago

free_rms 6y ago

Ok, fair enough, the more sophistication you add, the less explainable it gets. That being said, you can look at magnitude and direction of various coefficients and conclude certain obvious things. Even the case where reversing the correlation by picking some features instead of others says something -- those features point different ways but one has a higher magnitude so it 'wins'. They're ultimately pretty simple statistical observations about the training set. A deep net, in comparison, is a totally unexplainable black box between the cross-linking and the sigmoid functions.

qsort 6y ago
I agree that there is an intuitive concept of "explainability", which is why I said that a formal model for it would be interesting to see. I also agree that "the average explainability of a deep net is lower than the average explainability of a linear model" (whatever that sentence means), my point is that it's extremely hard to pinpoint what explainability means, because the usual characterization of "you can fit the model in your head" is extremely subjective and brittle. I'm all for more research in this field. Heck, I'm all for more research in any field, but because our definition of "explainability" is so primitive and unrefined, arguing that we should build ethical or regulatory frameworks on top of it seems premature.
glial 6y ago
> you can look at magnitude and direction of various coefficients and conclude certain obvious things. Sometimes, but if you have correlated predictors, the 'explainability' can be distributed across coefficients of those predictors in unpredictable ways. Training a linear model twice with two different subsets of training data can give you very different coefficient weights.