Here's a few things that really helped me out: http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/ http://mattmazur.com/2015/03/17/a-step-b

Here's a few things that really helped me out: http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/ http://mattmazur.com/2015/03/17/a-step-by-step-backpropagati... http://brenocon.com/blog/2013/10/tanh-is-a-rescaled-logistic-sigmoid-function/ http://brenocon.com/blog/2013/10/tanh-is-a-rescaled-logistic... Sort of a high-level overview (some of this is particular to neural nets as they're implemented in JMP, a product made by SAS, I don't know how well they generalize): • Neural nets consist of an input layer, an output layer, and some number of hidden layers between the input and output layers • The hidden layers consist of a number of nodes, which can be thought of as “on-off” switches (called a “step function” if we’re using the proper term), although in practice they’re represented by a smooth sigmoid curve with an upper and lower bound that represent on and off. In JMP, this takes the form of a hyperbolic tangent function, which is bound between -1 and 1. • Each node has a value that’s called the bias, which if you’re thinking of the nodes as an on-off switch, is the threshold a linear combination of the inputs needs to reach to turn the node on. In the sigmoid curves output by JMP, the bias represents the point at which the hyperbolic tangent function returns 0, given a linear combination of the inputs. • A “deep learning” model is one that has more than one hidden layer; the lowest hidden layer serves as the input layer to the next hidden layer, and so on. • The output layer takes the nodes of the highest hidden layer and produces an output. When dealing with probability, the output layer can be thought of as behaving identically to the hidden layer – a number of nodes that takes the inputs and produces a sigmoid curve. In JMP, the output layer is represented by the logistic sigmoid function, which is bound at 0 and 1. There is an equivalent logistic sigmoid function for every hyperbolic tangent function, so despite using different formulas, they behave the same way in practice. • Rather than fitting to an entire data set, neural nets are “trained” one data point at a time, where an initial sets of weights are determined (I believe randomly), and then for each data point in the training data, the amount of error is “backpropagated” among the various inputs, adjusting them. There’s a “training weight,” so the weights only move about 10% of the amount needed to reduce the error to zero.