3 ms·
I reckon piecewise continuity is enough.
by contravariant 8y ago
I reckon piecewise continuity is enough.
- whatshisface 8y agoNeural networks can't be discontinuous, even piecewise, because you wouldn't be able to differentiate at the discontinuities.
- dual_basis 8y agoYou're talking about training. That's a totally different question than approximation and representability.
- edflsafoiewq 8y agoIf that was the reason, the argument would demand the function be differentiable, not just continuous, wouldn't it?
- contravariant 8y agoPiecewise continuous functions can be approximated by differentiable ones. You lose some properties though. As you pointed out the derivatives might not converge, or even exist, and the limit converges pointwise at best, not uniformly.
- tprice7 8y ago"Piecewise continuous functions can be approximated by differentiable ones." It depends on how you measure the distance between functions. If you are using the uniform norm (as is used in the universal approximation theorem) then this is false.
- deleted 8y ago[deleted]
- analog31 8y agoDo neural nets need to be continuous functions of the inputs, or of the parameters? My recollection is that it only has to be the latter.
- alexchamberlain 8y agoMember of a Sobolev space enough? I think we're just reinventing finite difference and element methods here.