4 ms·
If we're talking about approximation, continuous functions can converge to step functions just fine. Take a regular sigmoid and keep raising the weight to see o
by whatshisface 4y ago
If we're talking about approximation, continuous functions can converge to step functions just fine. Take a regular sigmoid and keep raising the weight to see one example. That's a good point about training though, that theorem doesn't fully explain why NNs work, although it somewhat sounds like it does.
- mjburgess 4y agoTheir derivatives, f^n(x) for arbitrary n, do not converge fine. Indeed, even n=1 is usually terrible in practice. Fit a pleasant day-trip velocity curve with a NN and the acceleration would kill you.
- whatshisface 4y agoWouldn't that be mostly a consequence of the loss function used to compute the distance between the curve and the model output?
- tsimionescu 4y agoThere are discontinuous functions which are not steps - one of the more useful ones being tan(x) for any real x. Of course, since tan(x) is piece wise continuous and periodic, it is probably easy to work around in practice.