4 ms·

What do you mean by “black box” model? How do you know when a model “explains” something or not? How do you know if you are missing a key feature or degree of f

by mlthoughts2018 7y ago

What do you mean by “black box” model? How do you know when a model “explains” something or not? How do you know if you are missing a key feature or degree of freedom in a model? Suppose you have two models, A and B, where A gives much poorer predictions than B. Is it possible for A to “explain” things better when it is not capable of producing adequate predictions? Since nature itself is not boiled down to logical elements like math proofs, what about the inherent ambiguity of language or mental models when assessing explainability? Did we really explain anything, or just decompose a problem into units of thought that artificially feel compelling for human minds at this moment of history? What would distinguish a “real explanation” from a convenient fiction, if not purely predictive capability?

brmgb 7y ago
Black box model is a perfectly well understood term of art. It means a model which has a somewhat opaque internal working. A lot of modern ML approaches treat the model as a black box in that it is not particularly clear how the features actually interact to reach the prediction.
- mlthoughts2018 7y ago
  I work professionally in machine learning for over ten years now and I deeply dispute what you say, and many other practitioners do as well. > “not particularly clear how the features actually interact to reach the prediction.“ This is quite true of many models, such as linear regression in the wild. You may also have a clear but wrong picture of how features interact, eg looking at coefficients of interaction terms in a misspecified linear model. See for example, “The Mythos of Model Interpretability” - https://arxiv.org/abs/1606.03490 https://arxiv.org/abs/1606.03490 Without a coherent definition of what “not black box” or “explainable” means scientifically, the buzzword of “black box” is also meaningless and is more of a political game to be a gatekeeper over what models are allowed to be used than any kind of honest intellectual inquiry. I’ve worked professionally on systems where misspecified linear models and text models using simple ngram boosting were vastly more inscrutable than comparable neural networks or dimensionality reduction models for the same applications. Nobody has any scientifically cogent idea of what makes a model “explainable” other than arguing semantics.
  - SQueeeeeL 7y ago
    >Nobody has any scientifically cogent idea of what makes a model “explainable” other than arguing semantics. Bullshit, any client you show a regression to can easily fully understand how the inputs change the output.
    - ovi256 7y ago
      That's only true in some limited cases. Section 4.1 of the paper he linked discusses extensively the limitations of linear model interpretability. Here's an example: "With respect to algorithmic transparency, [the claim that linear models are more interpretable] seems uncontroversial, but given high dimensional or heavily engineered features, linear models lose simulatability or decomposability, respectively." Given how many interesting problems are high-dimensional or solved through heavy feature engineering, you must be lucky to live in the happy space where problems are either low-dimensional enough to not need heavy feature engineering or where the client is already familiar with the high-dimensional features you're going to limit the model to using.
    - mlthoughts2018 7y ago
      I call bullshit on your comment, not mine. - http://www.saramitchell.org/achen04.pdf http://www.saramitchell.org/achen04.pdf The functional mechanism of a regression has nearly nothing to do with whether the regression explains the phenomenon it’s being used to model or not.
  - natechols 7y ago
    The fact that the software in question is not publicly available or runnable certainly makes it a "black box", regardless of whether the underlying models are human-interpretable. EDIT: my apologies, apparently it is available: https://github.com/deepmind/deepmind-research/tree/master/alphafold_casp13 https://github.com/deepmind/deepmind-research/tree/master/al... I must have gotten mixed this up with a different black-box-advertised-in-scientific-journal story.
  - perl4ever 7y ago
    It may be because I'm not an expert that I think I understand, but the gist of making a model explainable sounds straightforward from what I've read. You train a model that does who knows what, and then you define a very limited language and use ML to match the first model as well as possible. If the language is simple enough, humans can understand it, and according to what I read, it likely generalizes better than the original model as a bonus. I assume it's harder than it sounds though. Quote: "1. There’s a tiny functional language based on a small number of side-effect free combinators 2. For a given task, a program template (which the authors call a sketch), further constrains the set of programs that can be learned for the problem in hand. This also very handily constrains the search space of course, helping to make learning a suitable policy program tractable. 3. To help guide the search within the set of programs conforming to the sketch, a standard reinforcement learning algorithm is used to learn a (black box) policy. 4. The black box policy is used as an oracle (the Neural Policy Oracle), and a neurally directed program search (NDPS) tries to find the sketch-conforming program that behaves as closely to the oracle as possible." The paper described: http://proceedings.mlr.press/v80/verma18a/verma18a.pdf http://proceedings.mlr.press/v80/verma18a/verma18a.pdf ...this seems kind of similar to the recent success in solving mathematical equations using deep learning. The language translation paradigm seems to have a lot more potential than some realized.

brmgb 7y ago

Black box model is a perfectly well understood term of art. It means a model which has a somewhat opaque internal working. A lot of modern ML approaches treat the model as a black box in that it is not particularly clear how the features actually interact to reach the prediction.

mlthoughts2018 7y ago
I work professionally in machine learning for over ten years now and I deeply dispute what you say, and many other practitioners do as well. > “not particularly clear how the features actually interact to reach the prediction.“ This is quite true of many models, such as linear regression in the wild. You may also have a clear but wrong picture of how features interact, eg looking at coefficients of interaction terms in a misspecified linear model. See for example, “The Mythos of Model Interpretability” - https://arxiv.org/abs/1606.03490 https://arxiv.org/abs/1606.03490 Without a coherent definition of what “not black box” or “explainable” means scientifically, the buzzword of “black box” is also meaningless and is more of a political game to be a gatekeeper over what models are allowed to be used than any kind of honest intellectual inquiry. I’ve worked professionally on systems where misspecified linear models and text models using simple ngram boosting were vastly more inscrutable than comparable neural networks or dimensionality reduction models for the same applications. Nobody has any scientifically cogent idea of what makes a model “explainable” other than arguing semantics.
- SQueeeeeL 7y ago
  >Nobody has any scientifically cogent idea of what makes a model “explainable” other than arguing semantics. Bullshit, any client you show a regression to can easily fully understand how the inputs change the output.
  - ovi256 7y ago
    That's only true in some limited cases. Section 4.1 of the paper he linked discusses extensively the limitations of linear model interpretability. Here's an example: "With respect to algorithmic transparency, [the claim that linear models are more interpretable] seems uncontroversial, but given high dimensional or heavily engineered features, linear models lose simulatability or decomposability, respectively." Given how many interesting problems are high-dimensional or solved through heavy feature engineering, you must be lucky to live in the happy space where problems are either low-dimensional enough to not need heavy feature engineering or where the client is already familiar with the high-dimensional features you're going to limit the model to using.
  - mlthoughts2018 7y ago
    I call bullshit on your comment, not mine. - http://www.saramitchell.org/achen04.pdf http://www.saramitchell.org/achen04.pdf The functional mechanism of a regression has nearly nothing to do with whether the regression explains the phenomenon it’s being used to model or not.
- natechols 7y ago
  The fact that the software in question is not publicly available or runnable certainly makes it a "black box", regardless of whether the underlying models are human-interpretable. EDIT: my apologies, apparently it is available: https://github.com/deepmind/deepmind-research/tree/master/alphafold_casp13 https://github.com/deepmind/deepmind-research/tree/master/al... I must have gotten mixed this up with a different black-box-advertised-in-scientific-journal story.
- perl4ever 7y ago
  It may be because I'm not an expert that I think I understand, but the gist of making a model explainable sounds straightforward from what I've read. You train a model that does who knows what, and then you define a very limited language and use ML to match the first model as well as possible. If the language is simple enough, humans can understand it, and according to what I read, it likely generalizes better than the original model as a bonus. I assume it's harder than it sounds though. Quote: "1. There’s a tiny functional language based on a small number of side-effect free combinators 2. For a given task, a program template (which the authors call a sketch), further constrains the set of programs that can be learned for the problem in hand. This also very handily constrains the search space of course, helping to make learning a suitable policy program tractable. 3. To help guide the search within the set of programs conforming to the sketch, a standard reinforcement learning algorithm is used to learn a (black box) policy. 4. The black box policy is used as an oracle (the Neural Policy Oracle), and a neurally directed program search (NDPS) tries to find the sketch-conforming program that behaves as closely to the oracle as possible." The paper described: http://proceedings.mlr.press/v80/verma18a/verma18a.pdf http://proceedings.mlr.press/v80/verma18a/verma18a.pdf ...this seems kind of similar to the recent success in solving mathematical equations using deep learning. The language translation paradigm seems to have a lot more potential than some realized.