3 ms·
I see a lot of commenters up in arms about Universal Approximation for NN’s, and I think the issue is that it’s often framed as a superpower rather than table s
by kevinventullo 5y ago
I see a lot of commenters up in arms about Universal Approximation for NN’s, and I think the issue is that it’s often framed as a superpower rather than table stakes for any kind of general purpose algorithm.
I posit that any modeling technique which does not have the universal approximation property will be guaranteed to fail on large classes of problems no matter how much elbow grease (say feature engineering) one puts into it. That is, UA is a necessary but not sufficient condition for a modeling technique to be fully general (i.e. could form the basis of whatever AGI is).
- DiggyJohnson 5y agoReally well said. Is there a term or concept in the AI literature++ that captures this point/conjecture?
- kevinventullo 5y agoI’m not sure. My intuition comes from real analysis, where one can think of the “space of all continuous functions” as being kind of geometric like Euclidean space. Then “Universal Approximation” for some Special set of functions (say those coming from neural networks) is equivalent to saying those special functions are “dense” in the space of all continuous functions. My geometric reasoning is that if a special collection is not dense, then it’s probably something like a lower-dimensional subspace, which really means you’re missing most of the ambient space in some sense. It’s all a bit handwavy but I hope that paints a picture. Getting more in the weeds, there are theorems like https://en.wikipedia.org/wiki/Stone%E2%80%93Weierstrass_theorem#Stone%E2%80%93Weierstrass_theorem,_real_version https://en.wikipedia.org/wiki/Stone%E2%80%93Weierstrass_theo... which suggest that being “dense in the space of continuous functions” is not as hard to achieve as one might guess.