I'm not sure if this provides any insight, but I think it's interesting to visualize this type of thing for the much simpler case of a two-class problem in two

I'm not sure if this provides any insight, but I think it's interesting to visualize this type of thing for the much simpler case of a two-class problem in two dimensions: http://i.imgur.com/NdzdH5j.png http://i.imgur.com/NdzdH5j.png On the left are the training vectors, color-coded by label; the background color-codes the probability output by the learned network at each point in the plane. On the right is the gradient field of the network's output corresponding to the blue class. The gradient field shows, at any point, the local direction of greatest increase toward the blue class.