3 ms·

While we're at it, doesn't the "universality theorem" (as the article calls it) basically follow immediately from the fact that the set of all continuous functi

While we're at it, doesn't the "universality theorem" (as the article calls it) basically follow immediately from the fact that the set of all continuous functions comprises a vector space? If the continuous function is additive, it's linear. If it's nonlinear, you can differentiate it to obtain a linear approximation. A neural network computes linear transformations, so unless I'm missing something I'm a little surprised there's a substantive theorem for this. Is it not a corollary on the fact that we can construct a vector space of all continuous functions?

thethirdone 7y ago
> While we're at it, doesn't the "universality theorem" (as the article calls it) basically follow immediately from the fact that the set of all continuous functions comprises a vector space? Pretty much, but you have to show that neural networks can create a basis in that vector space which is essentially the proof presented in the article. > If the continuous function is additive, it's linear. If it's nonlinear, you can differentiate it to obtain a linear approximation. Differentiating to obtain a linear approximation does not give you an arbitrarily good approximation like the theorem does. > A neural network computes linear transformations, so unless I'm missing something I'm a little surprised there's a substantive theorem for this. Is it not a corollary on the fact that we can construct a vector space of all continuous functions? Neural networks using sigmoid transfer functions do not compute linear transformations anymore. Importantly this theorem also states that you can approximate any function with only two hidden layers. A similar proof could not be made for a single hidden layer so it seems that the non-linearity of a single layer is not enough to form a basis for all continuous functions.
braised_babbage 7y ago
One of the things about continuous functions is that they aren't necessarily differentiable, cf. https://en.wikipedia.org/wiki/Weierstrass_function https://en.wikipedia.org/wiki/Weierstrass_function
- throwawaymath 7y ago
  True, but that was kind of an afterthought of my point. All continuous functions still comprise a vector space.
  - soVeryTired 7y ago
    I don't see how that helps? > A neural network computes linear transformations They compute a nonlinear function of an affine transform. Neural networks are nonlinear functions.
tnecniv 7y ago
Continuous != differentiable. The standard proof uses some functional analysis techniques but nothing too complicated to show you can get arbitrarily close to any continuous function with an NN. That includes things like step functions whose derivatives are not defined everywhere.
- throwawaymath 7y ago
  You could also just cite the Weierstrass function, which is continuous everywhere and differentiable nowhere. But that's separate from the meat of my point, which is that C(R) is a vector space.
contravariant 7y ago
A vector space with a countable basis to be precise. The set of all (real) functions is also linear, but is a lot trickier to work with.

thethirdone 7y ago

> While we're at it, doesn't the "universality theorem" (as the article calls it) basically follow immediately from the fact that the set of all continuous functions comprises a vector space? Pretty much, but you have to show that neural networks can create a basis in that vector space which is essentially the proof presented in the article. > If the continuous function is additive, it's linear. If it's nonlinear, you can differentiate it to obtain a linear approximation. Differentiating to obtain a linear approximation does not give you an arbitrarily good approximation like the theorem does. > A neural network computes linear transformations, so unless I'm missing something I'm a little surprised there's a substantive theorem for this. Is it not a corollary on the fact that we can construct a vector space of all continuous functions? Neural networks using sigmoid transfer functions do not compute linear transformations anymore. Importantly this theorem also states that you can approximate any function with only two hidden layers. A similar proof could not be made for a single hidden layer so it seems that the non-linearity of a single layer is not enough to form a basis for all continuous functions.

braised_babbage 7y ago

One of the things about continuous functions is that they aren't necessarily differentiable, cf. https://en.wikipedia.org/wiki/Weierstrass_function https://en.wikipedia.org/wiki/Weierstrass_function

throwawaymath 7y ago
True, but that was kind of an afterthought of my point. All continuous functions still comprise a vector space.
- soVeryTired 7y ago
  I don't see how that helps? > A neural network computes linear transformations They compute a nonlinear function of an affine transform. Neural networks are nonlinear functions.

tnecniv 7y ago

Continuous != differentiable. The standard proof uses some functional analysis techniques but nothing too complicated to show you can get arbitrarily close to any continuous function with an NN. That includes things like step functions whose derivatives are not defined everywhere.

throwawaymath 7y ago
You could also just cite the Weierstrass function, which is continuous everywhere and differentiable nowhere. But that's separate from the meat of my point, which is that C(R) is a vector space.

contravariant 7y ago

A vector space with a countable basis to be precise. The set of all (real) functions is also linear, but is a lot trickier to work with.