This statement doesn't make sense to me. A neural network literally can't produce anything besides a continuous function, and the universality theorem says that

This statement doesn't make sense to me. A neural network literally can't produce anything besides a continuous function, and the universality theorem says that there is no continuous function they can't (approximately) produce. So what could you possibly mean when you say neural networks have "more" than they need to do something which characterizes exactly what they can and can't do?