I think these sorts of arguments are not great because they confuse "limiting behavior" with "behavior at the limit". Yes if you are able to construct an infini

I think these sorts of arguments are not great because they confuse "limiting behavior" with "behavior at the limit". Yes if you are able to construct an infinite-sized MLP it can exactly replicate a given function, and you can construct a sequence of MLPs that in some sense converge to this infinite behavior. But in other measures the approximation might be infinitely bad and never get better unless the net is truly infinite. For an example, consider approximating the identity function [f(x) = x] with a sigmoid-activation MLP. For any finite size of the net, the output will have a minimum and maximum value. One can change the parameters of the net to increase the range of the output, but at no point is the output range infinite. So even though you can construct a sequence of MLPs that in the limit in some sense converges to the identity function, in some sense it never does. The same kind of thinking that leads to the conclusion "neural nets are universal approximators" would support the existence of perpetual motion machines; check out the "ellipsoid paradox" for more info.