This [D]NN are "universal approximator" claim that gets trotted around really gets my goat. Sorry I have to vent now, and its no fault of my parent post or gra

This [D]NN are "universal approximator" claim that gets trotted around really gets my goat. Sorry I have to vent now, and its no fault of my parent post or grand parent post. Its not that we were short of universal approximators. As far as learning is concerned, whether the class is universal or not has little or no import. For the sake of argument lets even assume that there is no noise in the training set [in other words I having a white horse on wings with a horn on its forehead that shoots laser beams with its eyes and farts indigo rainbows]. There will be infinitely many functions that passes through your training points and not all of them will have the same error on an unseen point. Learning is a very different ball game compared to approximation. "Universal", but in what class of functions ? All differentiable functions ? (the typical NN is restricted to this class), all continuous functions ? all measurable functions ? all computable functions ? Its surely not _all_ functions Here is a function: +1 on rational and -1 on irrational, can it approximate that ? here's another that is harder, +1 on a non-measurable set and -1 on a measurable set. You say these functions don't matter, indeed, I would be scared if they did matter. But it shows the real deal is characterizing that class of functions that "matter".