6 ms·
It's at most O(E) not at least. The capacity of a deep network could be much smaller than the number of weights and this is where the VC theory stops being usef
by backpropaganda 9y ago
It's at most O(E) not at least. The capacity of a deep network could be much smaller than the number of weights and this is where the VC theory stops being useful.
Deep networks can generalise to situations where even humans cannot. So the memorizing narrative doesn't survive any scrutiny.
- hamilyon2 9y agoIs there any good research about VC dimension of deep neural nets and usefulness of it?
- yters 9y agoCan you cite a source? It depends on the activation function, but as far as I know only the perceptron has a decent VC dimension due to its use of the sign function. The tanh and sigmoid result in O(E) and O(E^2) according to Wikipedia.
- backpropaganda 9y agoI don't really have a source, and am speaking from what is hearsay in the deep learning community. The results you cite are valid only for shallow networks. As you increase depth, you don't get the same increase in capacity, so even though millions of params are being used in deep networks, the capacity is not O(million). The capacity of a million-sized shallow net might be O(million), but noone's using such a model.
- yters 9y agoI saw the formula in Abu-Mustafa's Learning from Data. I don't think it only applied to single hidden layer networks, but I may be wrong. Additionally, the book said the VC dimension is infinite in the general case. I asked the question on CS stack exchange and no one took issue with the statement that DL had such a large VC dimension. The only counter response was that it didn't matter in practice due to DL's good error scores. But, that still doesn't mean DL is generalizing. Good error is only a necessary condition for generalization, not a sufficient condition.