4 ms·
You'd need to increase width (dimensionality) if you make these models shallow. My point still stands: The fact that models with sufficient capacity can approx
by cs702 4y ago
You'd need to increase width (dimensionality) if you make these models shallow.
My point still stands: The fact that models with sufficient capacity can approximate any function does not imply that two models trained independently of each other on different tasks will learn to approximate functions that relate to each other only by a linear transformation.
- sigmoid10 4y agoThe UAT states that depth is fundamentally not important, at least theoretically. It only has immense practical uses. So adding an intermediate linear layer + some nonlinearity already gets you an error scaling like O(1/N) for width N (in theory), regardless of what you are actually mapping. At least as long as it's somewhat continuous.