Given that they can represent continuous functions you can lightly modify them to produce all sorts of interesting behavior like piecewise continuity as well.

Given that they can represent continuous functions you can lightly modify them to produce all sorts of interesting behavior like piecewise continuity as well. The training thing is important though. There are problems which are arbitrarily likely to have all training algorithms fail, not just in a practical sense, but as a fundamental limitation.