3 ms·
The intrinsic dimension idea seemed interesting, but I didn't follow this bit: >By searching through the value of d=1,2,…,D, the corresponding d when the solut
by qPM9l3XJrF 6y ago
The intrinsic dimension idea seemed interesting, but I didn't follow this bit:
>By searching through the value of d=1,2,…,D, the corresponding d when the solution emerges is defined as the intrinsic dimension.
Is the idea to increase the "intrinsic dimension" until the network is able to learn? I'm not sure I buy the part of the argument where because the network is able to learn when the "intrinsic dimension" is kept low, it therefore follows that the "intrinsic dimension" is also low even when we aren't forcing it to be low. It seems a bit like saying "because we can obtain 90% accuracy on MNIST with a 10-parameter model, it therefore follows that a 1000-parameter model for MNIST only has 10 'intrinsic' parameters". Seems like a dubious / handwavey argument to me.
- rocqua 6y agoI think it is related to the lottery ticket idea. Essentially, you try to find the lowest d such that you can find a 'winning ticket' network of size d.
- yxhuvud 6y agoWell it is not the first time the idea of dimensionality reduction have been used in the field. It is for example the idea behind latent semantic methods (LSI and LSA).
- samcodes 6y agothis is a good explanation of intrinsic dimension https://eng.uber.com/intrinsic-dimension/ https://eng.uber.com/intrinsic-dimension/
- qPM9l3XJrF 6y agoThanks. The implication seems to be that if you restrict the # of parameters so it's equal to intrinsic dimension, learning isn't possible, but it is possible with this random projections method. Wonder why. It seems like with both methods, the number of possibilities being explored is the same, but the higher-parameter model space is richer with solutions for some reason.
- belgian_guy 6y agoThe intrinsic dimension paper indeed doesn't really show that big networks also search in small subspaces (and neither does it claim to), but this has already been shown in related papers like https://arxiv.org/abs/1812.04754 https://arxiv.org/abs/1812.04754