3 ms·
This particular example for visualizing neural networks doesn't seem particularly enlightening but I would argue the "black box" term is a appropriate and estab
by joe_the_user 9y ago
This particular example for visualizing neural networks doesn't seem particularly enlightening but I would argue the "black box" term is a appropriate and established term for describing current neural networks. In fact, researchers in the field acknowledge the problem of prediction without clear explanation and sometimes use this term to the describe the difficulty involved in reasoning about neural networks - it's some journalistic simplification or urban myth.
For example, "Many recent advances in NLP problems have come
from formulating and training expressive and elabo-
rate neural models. This includes models for senti-
ment classification, parsing, and machine translation
among many others. The gains in accuracy have,
however, come at the cost of interpretability since
complex neural models offer little transparency con-
cerning their inner workings."
Rationalizing Neural Predictions, Tao Lei, Regina Barzilay and Tommi Jaakkola
Also:
"When people say, "Neural networks are black boxes", what they mean is that it is hard to look "into" the network and figure out exactly what it has learnt.
"In a hand-crafted pipeline, you know precisely what you are building. So you may say, "my face detector will first look for eyebrows, then the mouth, and only if both are present, it will say 'face' ".
"For simple machine learning models, like linear SVMs or linear regression, you can "look" inside the model and see what it is actually doing (well, sort of). The model in this case is just a weighted linear combination of the features and so a feature the model weights highly is probably important to the task, while a feature not weighted highly is probably irrelevant (this is a big oversimplification).
"However if your model is learnt, end-to-end, as well as highly non-linear, as in a neural network, you can't do this. You know the model is some non-linear combination of some neurons, each of which is some non-linear combination of some other neurons, but it is near impossible to say what each neuron is doing."
https://www.quora.com/Why-are-artificial-neural-nets-black-boxes https://www.quora.com/Why-are-artificial-neural-nets-black-b...