4 ms·

Both a fault of the employees that worked on this, and the manager. Your deliverable should be an interpretable model. You can (and probably should) make neura

Both a fault of the employees that worked on this, and the manager. Your deliverable should be an interpretable model. You can (and probably should) make neural network models interpretable. If upper management does not trust your performance evaluation enough to bet on it, either the evaluation was weak (and no model should be deployed, however simple and interpretable) or upper management doesn't know enough about modern ML to have to make these decisions. I have sympathy for the manager in charge for making a decision on a complex model (while all they ever knew was simple survival models and basic statistical models). But you got to move with the times. Your competitors will use the most powerful models available (and some may go under due to improper risk management). Your employees don't want to build logistic regression models until eternity.

typeformer 8y ago
Not even Google has come anywhere close to being able to make complex NN models that have a human interperable "receipt" for decisions. In fact, for certain classes of problem solving it's likely impossible and that is already a huge problem.
- jmalicki 8y ago
  They've come up with plenty (e.g. https://blog.openai.com/adversarial-example-research/ https://blog.openai.com/adversarial-example-research/ ) - noone broadcasts them widely because noone likes the answers
- 5minbreak 8y ago
  I posit that complex NN models can achieve the same level of interpretability as logistic regression. In part, because some interpretability methods use logistic regression as a white box model to explain black boxes. In other words: If you are comfortable OK'ing a logistic regression model (because you looked at the coefficients and they made sense), you should be comfortable OK'ing a complex NN model (because the evaluation and interpretability modeling makes sense). Nitpicking, but significant: Most models don't output decisions, they output predictions. Decision scientists then build a policy on top of the model. Key issue here is that the policy makers don't trust the predictions. But I posit they have no reason to trust the predictions of a logistic regression model any more than the predictions of a complex black box. Provided, of course, you deliver interpretability UX, confidence estimates, and strong statistical guarantees and tests. Which is possible for even the blackest of boxes. If automatic justification is impossible for computers/black boxes, I believe it is impossible for humans too (as per Church-Turing). But let's say it is impossible. Do you think Google would use a white box model to optimize Adsense, because they can't interpret powerful deep learning solutions (like risk management for BlackRock: a very critical part of their business)? I'd say Google came pretty close with https://distill.pub/2018/building-blocks/ https://distill.pub/2018/building-blocks/ (they are not the only players in the interpretability field, and plenty of methods are becoming available, in large part driven by academia not business: interpretability and fairness are not too important for the bottom line).

typeformer 8y ago

Not even Google has come anywhere close to being able to make complex NN models that have a human interperable "receipt" for decisions. In fact, for certain classes of problem solving it's likely impossible and that is already a huge problem.

jmalicki 8y ago
They've come up with plenty (e.g. https://blog.openai.com/adversarial-example-research/ https://blog.openai.com/adversarial-example-research/ ) - noone broadcasts them widely because noone likes the answers
5minbreak 8y ago
I posit that complex NN models can achieve the same level of interpretability as logistic regression. In part, because some interpretability methods use logistic regression as a white box model to explain black boxes. In other words: If you are comfortable OK'ing a logistic regression model (because you looked at the coefficients and they made sense), you should be comfortable OK'ing a complex NN model (because the evaluation and interpretability modeling makes sense). Nitpicking, but significant: Most models don't output decisions, they output predictions. Decision scientists then build a policy on top of the model. Key issue here is that the policy makers don't trust the predictions. But I posit they have no reason to trust the predictions of a logistic regression model any more than the predictions of a complex black box. Provided, of course, you deliver interpretability UX, confidence estimates, and strong statistical guarantees and tests. Which is possible for even the blackest of boxes. If automatic justification is impossible for computers/black boxes, I believe it is impossible for humans too (as per Church-Turing). But let's say it is impossible. Do you think Google would use a white box model to optimize Adsense, because they can't interpret powerful deep learning solutions (like risk management for BlackRock: a very critical part of their business)? I'd say Google came pretty close with https://distill.pub/2018/building-blocks/ https://distill.pub/2018/building-blocks/ (they are not the only players in the interpretability field, and plenty of methods are becoming available, in large part driven by academia not business: interpretability and fairness are not too important for the bottom line).