5 ms·

Waste of a read! Reasoning that going from many input nodes to few output nodes results in compression (or shedding of nonessential data) isn't a brilliant theo

Waste of a read! Reasoning that going from many input nodes to few output nodes results in compression (or shedding of nonessential data) isn't a brilliant theory, it's common sense. The author even beats the dead horse of AI being a potential existential threat.

Zee2 9y ago
That's exactly what I was thinking. Wow, a neural network discards information at each layer by reducing the number of cells in each matrix? Water is wet. How else would a network make a final decision? If it never discarded any information, it would be just as heterogeneous of a dataset as when it started. It would be as if a linear regression algorithm "fit" a dataset by connecting the dots, except even more useless. I truly don't understand what this "new" theory is trying to say.
- carlmr 9y ago
  It's so basic I was thinking they could have called me, someone who studied neural networks only for half a year in uni, ten years ago. I could have told them that. Either the journalist compressed the information a bit too much and lost the essential amazing insight. Or it's just not amazing.
- nerdponx 9y ago
  Everything is obvious in hindsight. Too bad you didn't write up and publish your brilliant thoughts before this guy did. The actual talk is pretty technical and sophisticated. The conclusion is not just that neural networks discard information; it's a rigorous derivation of exactly how they do so.
Nomentatus 9y ago
The insight may be to show why image recognition has turned out to be a fairly easy task for deep learning - because just discarding pixels (the right pixels, but still) gets you a very long way toward your solution. Other problems that a brain handles quite well may not yield to such a simple process/analysis.
alew1 9y ago
Did you look at the actual research, or just this author's summary? I was also skeptical after reading this summary, but I found the presentation in the linked video to be far from trivial.
- nerdponx 9y ago
  Indeed. I'm glad the grandparent saved me from reading the article, but I'm even gladder that you got me to watch the talk. Yes, it might be common sense (to some). But the talk is a very lucid presentation, and also very technical. Definitely not a "water is wet" scenario.

Zee2 9y ago

That's exactly what I was thinking. Wow, a neural network discards information at each layer by reducing the number of cells in each matrix? Water is wet. How else would a network make a final decision? If it never discarded any information, it would be just as heterogeneous of a dataset as when it started. It would be as if a linear regression algorithm "fit" a dataset by connecting the dots, except even more useless. I truly don't understand what this "new" theory is trying to say.

carlmr 9y ago
It's so basic I was thinking they could have called me, someone who studied neural networks only for half a year in uni, ten years ago. I could have told them that. Either the journalist compressed the information a bit too much and lost the essential amazing insight. Or it's just not amazing.
nerdponx 9y ago
Everything is obvious in hindsight. Too bad you didn't write up and publish your brilliant thoughts before this guy did. The actual talk is pretty technical and sophisticated. The conclusion is not just that neural networks discard information; it's a rigorous derivation of exactly how they do so.

Nomentatus 9y ago

The insight may be to show why image recognition has turned out to be a fairly easy task for deep learning - because just discarding pixels (the right pixels, but still) gets you a very long way toward your solution. Other problems that a brain handles quite well may not yield to such a simple process/analysis.

alew1 9y ago

Did you look at the actual research, or just this author's summary? I was also skeptical after reading this summary, but I found the presentation in the linked video to be far from trivial.

nerdponx 9y ago
Indeed. I'm glad the grandparent saved me from reading the article, but I'm even gladder that you got me to watch the talk. Yes, it might be common sense (to some). But the talk is a very lucid presentation, and also very technical. Definitely not a "water is wet" scenario.