3 ms·

Saying that neural networks are "similar to" or that they "mimic" the human brain can be misleading. Today's architectures are the byproduct of years of researc

by fzliu 4y ago

Saying that neural networks are "similar to" or that they "mimic" the human brain can be misleading. Today's architectures are the byproduct of years of research and countless GPU-hours dedicated to training and testing architecture variants. Many neuroscience-based architectures that mimic the brain better than transformers end up performing much worse. The Quanta article is overall pretty reasonable, but I've unfortunately seen other news outlets regurgitate this kind of blanket statement for the better part of a decade. The very first models were perhaps 100% inspired by the brain, but today's ML research more or less follow a "whatever works best" principle.

trhway 4y ago
The convolution kernels in the first layers of AlexNet and all its DL image processing descendants converge to the Gabor filters (or some variation of) which are the response functions of the neurons in the first layer of the visual cortex. About 15 years before AlexNet there were works showing that such type of filter is kind of mathematically optimally encoding for the feature based image processing. (So, theoretically one could have just pre-generated the first layers in the net and use them fixed thus cutting significant time/effort on the training - i myself wanted to do it 20 years ago, yet just didn't get to it :) I'm pretty sure that for middle layers in image DL as well as for transformers in lang we have a kind of similar optimality, ie. something like maximum entropy filter(separator/aggregator at higher levels?) at a given level of granularity/scale like Gabors at the first feature level.
- data_maan 4y ago
  Could you provide references for your statements > the first layers of AlexNet and all its DL image processing descendants converge to the Gabor filters (or some variation of) and > 15 years before AlexNet there were works showing that such type of filter is kind of mathematically optimally encoding for the feature based image processing ?
  - trhway 4y ago
    For the first - you can just look at the original AlexNet paper. The kernels are unmistakably strikingly Gabor-like. Some differences, like cross-color, are kind of giving rise to possibly interesting questions - is it improvement or deficiency(i.e. more training would correct) over biology? or may be it is just real-valued projection from the [plausible] fact that the optimal is complex-valued? >? I don't have that specific reference i had in mind that was published 15-20 years ago, yet you can trace that line of thought development through the works like these for example (there have been a bunch of them in the 199x and into 200x) : 1990 - https://opg.optica.org/josaa/abstract.cfm?uri=josaa-7-8-1362 https://opg.optica.org/josaa/abstract.cfm?uri=josaa-7-8-1362 1998 - https://pubmed.ncbi.nlm.nih.gov/12662821/ https://pubmed.ncbi.nlm.nih.gov/12662821/

by fzliu 4y ago

trhway 4y ago

The convolution kernels in the first layers of AlexNet and all its DL image processing descendants converge to the Gabor filters (or some variation of) which are the response functions of the neurons in the first layer of the visual cortex. About 15 years before AlexNet there were works showing that such type of filter is kind of mathematically optimally encoding for the feature based image processing. (So, theoretically one could have just pre-generated the first layers in the net and use them fixed thus cutting significant time/effort on the training - i myself wanted to do it 20 years ago, yet just didn't get to it :) I'm pretty sure that for middle layers in image DL as well as for transformers in lang we have a kind of similar optimality, ie. something like maximum entropy filter(separator/aggregator at higher levels?) at a given level of granularity/scale like Gabors at the first feature level.

data_maan 4y ago
Could you provide references for your statements > the first layers of AlexNet and all its DL image processing descendants converge to the Gabor filters (or some variation of) and > 15 years before AlexNet there were works showing that such type of filter is kind of mathematically optimally encoding for the feature based image processing ?
- trhway 4y ago
  For the first - you can just look at the original AlexNet paper. The kernels are unmistakably strikingly Gabor-like. Some differences, like cross-color, are kind of giving rise to possibly interesting questions - is it improvement or deficiency(i.e. more training would correct) over biology? or may be it is just real-valued projection from the [plausible] fact that the optimal is complex-valued? >? I don't have that specific reference i had in mind that was published 15-20 years ago, yet you can trace that line of thought development through the works like these for example (there have been a bunch of them in the 199x and into 200x) : 1990 - https://opg.optica.org/josaa/abstract.cfm?uri=josaa-7-8-1362 https://opg.optica.org/josaa/abstract.cfm?uri=josaa-7-8-1362 1998 - https://pubmed.ncbi.nlm.nih.gov/12662821/ https://pubmed.ncbi.nlm.nih.gov/12662821/