4 ms·
The qualifier there is "intrinsic" dimensionality. There is smoothness to the image such that there is a lower dimensional embedding than 1m dimensions for a 1
by emcq 11y ago
The qualifier there is "intrinsic" dimensionality. There is smoothness to the image such that there is a lower dimensional embedding than 1m dimensions for a 1 megapixel image, particularly with any applied setting.
That said there is a bigger problem with those bounds because it doesn't incorporate model complexity. The VC dimension is much more insightful because the complexity of your model and the hypothesis space it represents is important for proper training. As an example, add a regularization term to your model and you're no longer doing anything like N/e^D. Convolutions, dropout, etc all prevent NN models from becoming too complex to train.
- tgflynn 11y agoDo you know of a way to measure "intrinsic dimensionality" ?
- emcq 11y agoIn general it's an abstract concept like Kolmogrov complexity but there are some practical approaches. People often try to intuit the intrinsic dimensionality of a dataset by using techniques like looking at singular values above some threshold or reconstruction error versus changing the output dimensionality of a dimensionality reduction/unsupervised technique like PCA, matrix factorization, or an autoencoder. An info theory person might argue entropy and compression ratios are also insightful.
- TTPrograms 11y agoI'd also note that for the case of typical classification that there's an additional concept of "intrinsic boundary dimensionality" that should be low due to the presence of invariants like scale, rotation and translation which drastically lower the intrinsic boundary dimensionality.