4 ms·

Black box model is a perfectly well understood term of art. It means a model which has a somewhat opaque internal working. A lot of modern ML approaches treat

by brmgb 7y ago

Black box model is a perfectly well understood term of art. It means a model which has a somewhat opaque internal working. A lot of modern ML approaches treat the model as a black box in that it is not particularly clear how the features actually interact to reach the prediction.

mlthoughts2018 7y ago
I work professionally in machine learning for over ten years now and I deeply dispute what you say, and many other practitioners do as well. > “not particularly clear how the features actually interact to reach the prediction.“ This is quite true of many models, such as linear regression in the wild. You may also have a clear but wrong picture of how features interact, eg looking at coefficients of interaction terms in a misspecified linear model. See for example, “The Mythos of Model Interpretability” - https://arxiv.org/abs/1606.03490 https://arxiv.org/abs/1606.03490 Without a coherent definition of what “not black box” or “explainable” means scientifically, the buzzword of “black box” is also meaningless and is more of a political game to be a gatekeeper over what models are allowed to be used than any kind of honest intellectual inquiry. I’ve worked professionally on systems where misspecified linear models and text models using simple ngram boosting were vastly more inscrutable than comparable neural networks or dimensionality reduction models for the same applications. Nobody has any scientifically cogent idea of what makes a model “explainable” other than arguing semantics.
- SQueeeeeL 7y ago
  >Nobody has any scientifically cogent idea of what makes a model “explainable” other than arguing semantics. Bullshit, any client you show a regression to can easily fully understand how the inputs change the output.
  - ovi256 7y ago
    That's only true in some limited cases. Section 4.1 of the paper he linked discusses extensively the limitations of linear model interpretability. Here's an example: "With respect to algorithmic transparency, [the claim that linear models are more interpretable] seems uncontroversial, but given high dimensional or heavily engineered features, linear models lose simulatability or decomposability, respectively." Given how many interesting problems are high-dimensional or solved through heavy feature engineering, you must be lucky to live in the happy space where problems are either low-dimensional enough to not need heavy feature engineering or where the client is already familiar with the high-dimensional features you're going to limit the model to using.
  - mlthoughts2018 7y ago
    I call bullshit on your comment, not mine. - http://www.saramitchell.org/achen04.pdf http://www.saramitchell.org/achen04.pdf The functional mechanism of a regression has nearly nothing to do with whether the regression explains the phenomenon it’s being used to model or not.
- natechols 7y ago
  The fact that the software in question is not publicly available or runnable certainly makes it a "black box", regardless of whether the underlying models are human-interpretable. EDIT: my apologies, apparently it is available: https://github.com/deepmind/deepmind-research/tree/master/alphafold_casp13 https://github.com/deepmind/deepmind-research/tree/master/al... I must have gotten mixed this up with a different black-box-advertised-in-scientific-journal story.
- perl4ever 7y ago
  It may be because I'm not an expert that I think I understand, but the gist of making a model explainable sounds straightforward from what I've read. You train a model that does who knows what, and then you define a very limited language and use ML to match the first model as well as possible. If the language is simple enough, humans can understand it, and according to what I read, it likely generalizes better than the original model as a bonus. I assume it's harder than it sounds though. Quote: "1. There’s a tiny functional language based on a small number of side-effect free combinators 2. For a given task, a program template (which the authors call a sketch), further constrains the set of programs that can be learned for the problem in hand. This also very handily constrains the search space of course, helping to make learning a suitable policy program tractable. 3. To help guide the search within the set of programs conforming to the sketch, a standard reinforcement learning algorithm is used to learn a (black box) policy. 4. The black box policy is used as an oracle (the Neural Policy Oracle), and a neurally directed program search (NDPS) tries to find the sketch-conforming program that behaves as closely to the oracle as possible." The paper described: http://proceedings.mlr.press/v80/verma18a/verma18a.pdf http://proceedings.mlr.press/v80/verma18a/verma18a.pdf ...this seems kind of similar to the recent success in solving mathematical equations using deep learning. The language translation paradigm seems to have a lot more potential than some realized.
  - mlthoughts2018 7y ago
    Yes, you are oversimplifying what is required to make a model explainable. The logical mechanism of the functional form of the model has very little to do with explainability. Explainability involves various kinds of model checking, overfitting analysis, tests for confounding variables, multicollinearity, interaction effects, model misspecification, etc. Merely decomposing a prediction into say a linear combination of predictors does not, itself, provide any type of explanation unless a variety of more complicated assumptions about statistical model checking turn out to be true. Even worse, because the naive approach is to just treat those regression coefficients as if they do automatically give explanatory power or feature-wise attribution, people misrepresent things unwittingly and don’t carry out enough robust model checking, leading to wrong explanations that appear to have deceptive degrees of confidence associated with them.

by brmgb 7y ago

mlthoughts2018 7y ago

I work professionally in machine learning for over ten years now and I deeply dispute what you say, and many other practitioners do as well. > “not particularly clear how the features actually interact to reach the prediction.“ This is quite true of many models, such as linear regression in the wild. You may also have a clear but wrong picture of how features interact, eg looking at coefficients of interaction terms in a misspecified linear model. See for example, “The Mythos of Model Interpretability” - https://arxiv.org/abs/1606.03490 https://arxiv.org/abs/1606.03490 Without a coherent definition of what “not black box” or “explainable” means scientifically, the buzzword of “black box” is also meaningless and is more of a political game to be a gatekeeper over what models are allowed to be used than any kind of honest intellectual inquiry. I’ve worked professionally on systems where misspecified linear models and text models using simple ngram boosting were vastly more inscrutable than comparable neural networks or dimensionality reduction models for the same applications. Nobody has any scientifically cogent idea of what makes a model “explainable” other than arguing semantics.

SQueeeeeL 7y ago
>Nobody has any scientifically cogent idea of what makes a model “explainable” other than arguing semantics. Bullshit, any client you show a regression to can easily fully understand how the inputs change the output.
- ovi256 7y ago
  That's only true in some limited cases. Section 4.1 of the paper he linked discusses extensively the limitations of linear model interpretability. Here's an example: "With respect to algorithmic transparency, [the claim that linear models are more interpretable] seems uncontroversial, but given high dimensional or heavily engineered features, linear models lose simulatability or decomposability, respectively." Given how many interesting problems are high-dimensional or solved through heavy feature engineering, you must be lucky to live in the happy space where problems are either low-dimensional enough to not need heavy feature engineering or where the client is already familiar with the high-dimensional features you're going to limit the model to using.
- mlthoughts2018 7y ago
  I call bullshit on your comment, not mine. - http://www.saramitchell.org/achen04.pdf http://www.saramitchell.org/achen04.pdf The functional mechanism of a regression has nearly nothing to do with whether the regression explains the phenomenon it’s being used to model or not.
natechols 7y ago
The fact that the software in question is not publicly available or runnable certainly makes it a "black box", regardless of whether the underlying models are human-interpretable. EDIT: my apologies, apparently it is available: https://github.com/deepmind/deepmind-research/tree/master/alphafold_casp13 https://github.com/deepmind/deepmind-research/tree/master/al... I must have gotten mixed this up with a different black-box-advertised-in-scientific-journal story.
perl4ever 7y ago
It may be because I'm not an expert that I think I understand, but the gist of making a model explainable sounds straightforward from what I've read. You train a model that does who knows what, and then you define a very limited language and use ML to match the first model as well as possible. If the language is simple enough, humans can understand it, and according to what I read, it likely generalizes better than the original model as a bonus. I assume it's harder than it sounds though. Quote: "1. There’s a tiny functional language based on a small number of side-effect free combinators 2. For a given task, a program template (which the authors call a sketch), further constrains the set of programs that can be learned for the problem in hand. This also very handily constrains the search space of course, helping to make learning a suitable policy program tractable. 3. To help guide the search within the set of programs conforming to the sketch, a standard reinforcement learning algorithm is used to learn a (black box) policy. 4. The black box policy is used as an oracle (the Neural Policy Oracle), and a neurally directed program search (NDPS) tries to find the sketch-conforming program that behaves as closely to the oracle as possible." The paper described: http://proceedings.mlr.press/v80/verma18a/verma18a.pdf http://proceedings.mlr.press/v80/verma18a/verma18a.pdf ...this seems kind of similar to the recent success in solving mathematical equations using deep learning. The language translation paradigm seems to have a lot more potential than some realized.
- mlthoughts2018 7y ago
  Yes, you are oversimplifying what is required to make a model explainable. The logical mechanism of the functional form of the model has very little to do with explainability. Explainability involves various kinds of model checking, overfitting analysis, tests for confounding variables, multicollinearity, interaction effects, model misspecification, etc. Merely decomposing a prediction into say a linear combination of predictors does not, itself, provide any type of explanation unless a variety of more complicated assumptions about statistical model checking turn out to be true. Even worse, because the naive approach is to just treat those regression coefficients as if they do automatically give explanatory power or feature-wise attribution, people misrepresent things unwittingly and don’t carry out enough robust model checking, leading to wrong explanations that appear to have deceptive degrees of confidence associated with them.