5 ms·
Here is the special sauce: “We can consider the the discrete Fourier transform (DFT) to be an artificial neural network: it is a single layer network, with no b
by omarhaneef 5y ago
Here is the special sauce: “We can consider the the discrete Fourier transform (DFT) to be an artificial neural network: it is a single layer network, with no bias, no activation function, and particular values for the weights. The number of output nodes is equal to the number of frequencies we evaluate.”
A single layer neural network is a sum of products, the basic Fourier equation is a sum of products.
In this view there are lots of single layer neural networks out there. For me, it’s the training algorithm (backprop) that sets apart the neural net.
- 29athrowaway 5y agoBiological neural networks do not use backpropagation, or at least, there's no evidence of that yet. Backpropagation is a placeholder for stuff we don't understand.
- deleted 5y ago[deleted]
- cl3misch 5y agoIsn't "backpropagation" just a synonym for the partial derivative of the scalar cost function with respect to the weights? And in the mathematical formulation of biological neural networks these derivatives can't be computed analytically. Your comment sounds like "backpropagation" is some kind of natural phenomenon.
- 29athrowaway 5y ago> Your comment sounds like "backpropagation" is some kind of natural phenomenon. No it's not, read again.
- ta988 5y agoNot exactly the same kind of backpropagation but signals fly the other way in neurons as well https://en.wikipedia.org/wiki/Neural_backpropagation https://en.wikipedia.org/wiki/Neural_backpropagation
- etienne618 5y agoIt seems that backpropagation might be a good abstraction for biological learing rules after all: "local predictive coding converges asymptotically (and in practice rapidly) to exact backprop gradients on arbitrary computation graphs using only local learning rules" from https://arxiv.org/abs/2006.04182 https://arxiv.org/abs/2006.04182
- 29athrowaway 5y agoIn biology there are excitatory networks and inhibitory networks.
- nerdponx 5y agoIn my mind, the layers make the network "non-trivial". A Fourier transform is only a neural network in a trivial, definitional sense.
- hyperman1 5y agoThe activation function is what keeps layers separated. When it isn't there, a pair of layers devolves to a matrix multiplication, which can be replaced by its resulting matrix, a single layer.
- omarhaneef 5y agoThat is a thoughtful distinction, too. I think you're right.