3 ms·

The mathematical explanation is surprisingly simple. A 2 layer neural network consists of 2 matrices A and B and an activation function f. The neural network co

by jules 16y ago

The mathematical explanation is surprisingly simple. A 2 layer neural network consists of 2 matrices A and B and an activation function f. The neural network computes this function where in and out are vectors: out = f(A*f(B*in)) where f is applied to each element of a vector. You can see how this generalizes to more layers. Neural network learning algorithms are given (in,out) pairs and try to find A and B to minimize the mean square error, for example by using stochastic gradient descent.

sqrt17 16y ago
This is a good summary. Now, here's what you need to know to have it work: * initialize weights to small random values (otherwise it will quickly get stuck in a non-desirable local optimum) * you need to decrease the learning rate of the stochastic descent according to a 1/(n+n0) or similar schedule, otherwise there's a risk of the whole thing swinging back and forth * it often helps to normalize the input values so that they have a standard deviation of 1 and an expectation of zero. Or at least it's not good to have input values differ too much in scale. * if you're training something that should give yes/no answers, you don't really want to minimize the mean square error, but a logistic loss. Some of these parts are usually built into your NN toolkit, so you don't have to worry about them. Or they're not (e.g., choosing a good learning rate), in which case you're just screwed if you don't know what happens and why. But you're right, mathematically ANNs are quite simple (and, as one has found out, don't really do similar things to what actual neurons do).
gtt 16y ago
Could you provide a link to mathematical sound introduction into neural networks? I was always turned out because it was always explained in some magical manner...
- dododo 16y ago
  if f is the logistic function, then neural networks basically correspond to logistic regression. http://en.wikipedia.org/wiki/Logistic_regression http://en.wikipedia.org/wiki/Logistic_regression which has been around since the 1940s... (multilayer neural networks correspond to hierarchical logistic regression--just plug them together). this should be in any reasonable stats book. if you mean mathematically sound in terms of learning, etc, then chris bishops "neural networks and pattern recognition" is pretty good, full of sage advice and justification.

by jules 16y ago

sqrt17 16y ago

This is a good summary. Now, here's what you need to know to have it work: * initialize weights to small random values (otherwise it will quickly get stuck in a non-desirable local optimum) * you need to decrease the learning rate of the stochastic descent according to a 1/(n+n0) or similar schedule, otherwise there's a risk of the whole thing swinging back and forth * it often helps to normalize the input values so that they have a standard deviation of 1 and an expectation of zero. Or at least it's not good to have input values differ too much in scale. * if you're training something that should give yes/no answers, you don't really want to minimize the mean square error, but a logistic loss. Some of these parts are usually built into your NN toolkit, so you don't have to worry about them. Or they're not (e.g., choosing a good learning rate), in which case you're just screwed if you don't know what happens and why. But you're right, mathematically ANNs are quite simple (and, as one has found out, don't really do similar things to what actual neurons do).

gtt 16y ago

Could you provide a link to mathematical sound introduction into neural networks? I was always turned out because it was always explained in some magical manner...

dododo 16y ago
if f is the logistic function, then neural networks basically correspond to logistic regression. http://en.wikipedia.org/wiki/Logistic_regression http://en.wikipedia.org/wiki/Logistic_regression which has been around since the 1940s... (multilayer neural networks correspond to hierarchical logistic regression--just plug them together). this should be in any reasonable stats book. if you mean mathematically sound in terms of learning, etc, then chris bishops "neural networks and pattern recognition" is pretty good, full of sage advice and justification.