3 ms·
We're not going to get very far letting machines guess what we want by making models we can't interpret. Explainability is important, and critical in applicati
by robius 8y ago
We're not going to get very far letting machines guess what we want by making models we can't interpret.
Explainability is important, and critical in applications where lives are on the line.
Explainability where we're guessing what's happening in a black box model won't do either. Nothing but complete transparency of the model and why it's doing what it's doing. Its source code, that makes sense to humans, is needed. Full on model audit. No guessing.
I can think of only one company that's attempting to do this, and it's not anyone you hear working on explainability, including DARPA.
- davidbrent 8y agoBut when you’re in advertising all that matters are the results.
- stochastic_monk 8y agoTommi Jaakkola at CSAIL, in particular, comes to mind. [0] I think a number of others are moving towards it. [0] https://www.csail.mit.edu/research/interpretability-complex-machine-learning-models https://www.csail.mit.edu/research/interpretability-complex-...
- Tenoke 8y ago1. There are a lot of researchers who are working on interpretability (check on arxiv). 2. This rant has very little to do with the article, and feels like a borderline meme that some people post on all that is ML-related.
- tensor 8y agoIf the visual cortex were a machine model, people would be complaining about how we can't explain it and how it's a dangerous black box. They'd probably tout the many optical illusions as demonstrations of this danger. Yet we don't demand that other humans explain how their visual cortex work. There is a double standard here.
- AstralStorm 8y agoI do not approve of the red herring. Either it is visual cortex, or the intelligence and decision making/planning. One works very differently from the other.
- visarga 8y agoYou can't make a full on model audit on a neural net, but there are architectures such as the Transformer (an attention scheme) that can give a lot of insight into what the neural net thinks. We can also visualise what inputs maximally activate a deep neuron in a CNN. Not all DL models are truly black boxes.
- AstralStorm 8y agoThe problem is the models have no reflection capabilities unlike people. The explanation is always done by a really different external system and sometimes by an actual intelligence. State models (Markovian) are sometimes able to explain things but not always really, especially in complex cases.
- visarga 8y agoOn the other hand, humans tend first to take a decision and later retrofit an explanation to it, even if it is completely wrong, and we assume the explanation is the reason and not the effect.
- jorgemf 8y agoWhat's the difference between the output of a network and what an expert says? Unless you can probe mathematically why networks or human mind works, there is no explanation for any of both methods. I can argue that humans learnt from examples the same ways neural networks do, you can say the opposite. But we have no way to say that that any claim is false or true.