3 ms·

I find it counter-intuitive that thought vectors in should have "Linear Structure" in multilevel autoencoders. Since the whole appeal of neural networks is tha

• antognini 10y ago
  The goal of a neural network is to take a complicated manifold and, through each of its layers, flatten it out into progressively more and more linear manifolds. If you are building a classifier, then the inputs to the last layer will necessarily have to be linearly separable because the final layer is essentially linear --- the softmax operation just transforms the pre-activation values from logits to probabilities. So if the NN is well trained, the second to last layer will have linearly separated the different classes (as much as possible, anyway). Earlier layers may not have completely linearly separated their inputs, but they are probably going to lie along simpler manifolds than even earlier inputs. There's a good blog post on this here: http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/ http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/
  • mooneater 10y ago
    Yes I understand that conceptually. But the middle autoencoder layer is not really an output layer, what would constrain it to linear representations? I assume normal output layers are constrained to linear representations by the supervised training process.