3 ms·

Well the jacobian is a numerical matrix unique at each point in the input space. So its kinda hard to visualize the changing Jacobian matrix over the input spac

Well the jacobian is a numerical matrix unique at each point in the input space. So its kinda hard to visualize the changing Jacobian matrix over the input space. People do interpret ANNs. Normally by visualising the weight matrices on the input layer, which have a 1-1 mapping to input vector attributes (so you can label them). It gets a bit hard on subsequent layers though. decision trees are an order of magnitude easier to interpret though, they are very compact and consist of a number of binary decision in the input space. For a given classification you only need to look at log(d) decisions to work out how it got to its conclusion. Its not usually that hard to how it arrived at the tree from training. The relationship between the Jacobian and the training is fairly convoluted, given back propagation and cross validation. PS so back-prorogation uses weights differentiated against outputs. Jacobian is input differentiated against outputs. You can do smart things with Jacobian aware neural architectures. See "Forward models: Supervised learning with a distal teacher" which trains two networks in parallel, one a forward model, one an inverse model, and uses the Jacobian to circumnavigate the problematic non-convexity of the inverse model.

foobarqux 13y ago
> Well the jacobian is a numerical matrix unique at each point in the input space. So its kinda hard to visualize the changing Jacobian matrix over the input space. Yeah it doesn't seem like it would be easy to gain much insight since the partial derivatives are a function of the (other) inputs. > People do interpret ANNs. Normally by visualising the weight matrices on the input layer, which have a 1-1 mapping to input vector attributes (so you can label them). What do the first-layer weights tell me?
- tlarkworthy 13y ago
  If the weight is positive that feature helps the neuron fire and vice versa. If the neural network is processing images, the weights form and image too. You can tell what a lot of the units are "looking" for by plotting the weights as an image. page 14, Figure 3: jmlr.org/papers/volume11/erhan10a/erhan10a.pdf You can clearly see various digit deformations in the weights. (white is positive weight, black is negative weight typically) Lots of papers do this, this one was happened to be the first one I managed to google. So the first layer is normally readable because weights are in the same space as the input feature space. The second layer is normally a jumble as its randomly initialised before convergence. But in the paper's example you can imagine the 2nd layer output unit representing a final classification of a 4, is probably summing up all the 4 deformation detectors in layer 1 and negatively summing up everything else.
  - tlarkworthy 13y ago
    Actually I just read that paper properly and I see they did visualise the higher level layer weights too: "For visualizing what units do on the 2nd and 3rd layer, we used the activation maxi- mization technique described by Erhan et al. (2009)"[1] So there you have it! You can visualise all of a neural network! But you have to implement some specific numerics to get the data out. Reading the first layer is trivial, and subsequent layers need some analysis. [1] Erhan 2009 Visualizing higher layer features of a deep network (good find, thanks for asking the right questions!)
  - foobarqux 13y ago
    > If the weight is positive that feature helps the neuron fire and vice versa. But that doesn't tell you much does it? It's effect could be reversed or minimized in subsequent layers. Haven't read it yet but thanks for the paper.
    - tlarkworthy 13y ago
      Just look at the one figure in the paper and you will see. Because your final layer is positively encoded, that polarity trickles down through back propogation. I also think the positive weights thing is to do with the way features normally work. It makes sense to look for corners, not anti corners for object detection.

foobarqux 13y ago

> Well the jacobian is a numerical matrix unique at each point in the input space. So its kinda hard to visualize the changing Jacobian matrix over the input space. Yeah it doesn't seem like it would be easy to gain much insight since the partial derivatives are a function of the (other) inputs. > People do interpret ANNs. Normally by visualising the weight matrices on the input layer, which have a 1-1 mapping to input vector attributes (so you can label them). What do the first-layer weights tell me?

tlarkworthy 13y ago
If the weight is positive that feature helps the neuron fire and vice versa. If the neural network is processing images, the weights form and image too. You can tell what a lot of the units are "looking" for by plotting the weights as an image. page 14, Figure 3: jmlr.org/papers/volume11/erhan10a/erhan10a.pdf You can clearly see various digit deformations in the weights. (white is positive weight, black is negative weight typically) Lots of papers do this, this one was happened to be the first one I managed to google. So the first layer is normally readable because weights are in the same space as the input feature space. The second layer is normally a jumble as its randomly initialised before convergence. But in the paper's example you can imagine the 2nd layer output unit representing a final classification of a 4, is probably summing up all the 4 deformation detectors in layer 1 and negatively summing up everything else.
- tlarkworthy 13y ago
  Actually I just read that paper properly and I see they did visualise the higher level layer weights too: "For visualizing what units do on the 2nd and 3rd layer, we used the activation maxi- mization technique described by Erhan et al. (2009)"[1] So there you have it! You can visualise all of a neural network! But you have to implement some specific numerics to get the data out. Reading the first layer is trivial, and subsequent layers need some analysis. [1] Erhan 2009 Visualizing higher layer features of a deep network (good find, thanks for asking the right questions!)
- foobarqux 13y ago
  > If the weight is positive that feature helps the neuron fire and vice versa. But that doesn't tell you much does it? It's effect could be reversed or minimized in subsequent layers. Haven't read it yet but thanks for the paper.
  - tlarkworthy 13y ago
    Just look at the one figure in the paper and you will see. Because your final layer is positively encoded, that polarity trickles down through back propogation. I also think the positive weights thing is to do with the way features normally work. It makes sense to look for corners, not anti corners for object detection.