5 ms·
*any continuous function
by aportnoy 7y ago
*any continuous function
- YesProcrast 7y agoCame here to suggest the function that is 1 on the rationals and 0 on the irrationals.
- adrianhel 7y agoNot continuous.
- throwawaymath 7y agoI think that was the point.
- analog31 7y agoAlso can't be computed.
- nightcracker 7y agoThat depends on the input format.
- dual_basis 7y agoIt can represent any arbitrary function. If you are using gradient descent, then you'll need the desired loss function to be differentiable with respect to the parameters, but that's a totally different matter.
- braised_babbage 7y agoNo, neural networks cannot "represent any arbitrary function". Find a theorem that you think states otherwise, and then read what the theorem actually states.
- dual_basis 7y agoYou're correct, the statement of the theorem refers to continuous functions. In practice, however, the input to neural networks is represented by floating point values, which is a discrete set. So pick whatever arbitrary function you would like, there is some continuous approximation to that function which is actually equal to it on every floating point value, and that function can be approximated arbitrarily closely by a neural network.
- throwawaymath 7y agoWhile we're at it, doesn't the "universality theorem" (as the article calls it) basically follow immediately from the fact that the set of all continuous functions comprises a vector space? If the continuous function is additive, it's linear. If it's nonlinear, you can differentiate it to obtain a linear approximation. A neural network computes linear transformations, so unless I'm missing something I'm a little surprised there's a substantive theorem for this. Is it not a corollary on the fact that we can construct a vector space of all continuous functions?
- thethirdone 7y ago> While we're at it, doesn't the "universality theorem" (as the article calls it) basically follow immediately from the fact that the set of all continuous functions comprises a vector space? Pretty much, but you have to show that neural networks can create a basis in that vector space which is essentially the proof presented in the article. > If the continuous function is additive, it's linear. If it's nonlinear, you can differentiate it to obtain a linear approximation. Differentiating to obtain a linear approximation does not give you an arbitrarily good approximation like the theorem does. > A neural network computes linear transformations, so unless I'm missing something I'm a little surprised there's a substantive theorem for this. Is it not a corollary on the fact that we can construct a vector space of all continuous functions? Neural networks using sigmoid transfer functions do not compute linear transformations anymore. Importantly this theorem also states that you can approximate any function with only two hidden layers. A similar proof could not be made for a single hidden layer so it seems that the non-linearity of a single layer is not enough to form a basis for all continuous functions.
- braised_babbage 7y agoOne of the things about continuous functions is that they aren't necessarily differentiable, cf. https://en.wikipedia.org/wiki/Weierstrass_function https://en.wikipedia.org/wiki/Weierstrass_function
- throwawaymath 7y agoTrue, but that was kind of an afterthought of my point. All continuous functions still comprise a vector space.
- AnaniasAnanas 7y agoAren't all total functions continuous?
- deleted 7y ago[deleted]
- zeroonetwothree 7y agoWeierstrauss function ;)?